2024-03-04T12:03:05.785 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-04T12:03:05.785 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} --verbose --archive /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054 --name vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi --owner scheduled_vshankar@teuthology -- /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054/orig.config.yaml 2024-03-04T12:03:05.845 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T12:03:05.939 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054 branch: wip-vshankar-testing-20240304.042522 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} email: vshankar@redhat.com first_in_suite: false job_id: '7581054' kernel: client: sha1: distro flavor: default kdb: true sha1: 786939134d7eee4dc7a805245d3f595929af37d7 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240304.042522 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240304.042522 sha1: 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 owner: scheduled_vshankar@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3119 sha1: 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 sleep_before_teardown: 0 subset: 270/1024 suite: fs suite_branch: wip-vshankar-testing-20240304.042522 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH2C7eOZ0wrUqj5MViV6ezJadMYDbaK5FmjU4W+QOUL367quJYKBuHlEaC5U6HQZJ+fZ/Gj0GUd4exvLCC9qG7E= smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBATIsvh1BLe9YhwLFtXJ2Sv7ZNO0g8aFNI679y+Jdt0fEx0C/mOgcroWE3IVVkjd42gfla3/dEWnOFCu0RX/bhM= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMYWj+J7HylnDPtk4An/coA6jD0uZFBqa9szme/tpHgu4LbUo5z/0AdWU8KJ3J+4bDaZWZCT4b+egmwW6qApGt4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-03-04_08:26:39 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-04T12:03:05.939 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa; will attempt to use it 2024-03-04T12:03:05.940 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks 2024-03-04T12:03:05.940 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-04T12:03:05.942 INFO:teuthology.task.internal:Checking packages... 2024-03-04T12:03:05.971 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '717ce3cce8d0166e4e577faa8004e6f9cb4128c0' 2024-03-04T12:03:05.972 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-04T12:03:05.972 INFO:teuthology.packaging:ref: None 2024-03-04T12:03:05.972 INFO:teuthology.packaging:tag: None 2024-03-04T12:03:05.972 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240304.042522 2024-03-04T12:03:05.972 INFO:teuthology.packaging:sha1: 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:03:05.972 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240304.042522 2024-03-04T12:03:06.120 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1845.g717ce3cc 2024-03-04T12:03:06.122 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-04T12:03:06.131 INFO:teuthology.task.internal:no buildpackages task found 2024-03-04T12:03:06.131 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-04T12:03:06.167 INFO:teuthology.task.internal:Saving configuration 2024-03-04T12:03:06.192 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-04T12:03:06.203 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T12:03:06.230 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 11:57:57.402365', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH2C7eOZ0wrUqj5MViV6ezJadMYDbaK5FmjU4W+QOUL367quJYKBuHlEaC5U6HQZJ+fZ/Gj0GUd4exvLCC9qG7E='} 2024-03-04T12:03:06.252 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 11:57:57.398791', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBATIsvh1BLe9YhwLFtXJ2Sv7ZNO0g8aFNI679y+Jdt0fEx0C/mOgcroWE3IVVkjd42gfla3/dEWnOFCu0RX/bhM='} 2024-03-04T12:03:06.276 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 11:57:57.400798', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMYWj+J7HylnDPtk4An/coA6jD0uZFBqa9szme/tpHgu4LbUo5z/0AdWU8KJ3J+4bDaZWZCT4b+egmwW6qApGt4='} 2024-03-04T12:03:06.276 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-04T12:03:06.285 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-04T12:03:06.285 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-04T12:03:06.285 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-04T12:03:06.285 INFO:teuthology.run_tasks:Running task console_log... 2024-03-04T12:03:06.466 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7e8ed9b160>, signals=[15]) 2024-03-04T12:03:06.466 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-04T12:03:06.476 INFO:teuthology.task.internal:Opening connections... 2024-03-04T12:03:06.476 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-03-04T12:03:06.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T12:03:06.585 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-03-04T12:03:06.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T12:03:06.691 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-03-04T12:03:06.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T12:03:06.778 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-04T12:03:06.914 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-03-04T12:03:06.932 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-03-04T12:03:06.933 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-03-04T12:03:06.990 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T12:03:06.991 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T12:03:06.995 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-03-04T12:03:07.020 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-03-04T12:03:07.038 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-03-04T12:03:07.039 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-03-04T12:03:07.096 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream" 2024-03-04T12:03:07.096 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="9" 2024-03-04T12:03:07.096 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos" 2024-03-04T12:03:07.096 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora" 2024-03-04T12:03:07.096 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="9" 2024-03-04T12:03:07.096 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el9" 2024-03-04T12:03:07.097 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T12:03:07.097 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-03-04T12:03:07.097 INFO:teuthology.orchestra.run.smithi063.stdout:LOGO="fedora-logo-icon" 2024-03-04T12:03:07.097 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T12:03:07.097 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/" 2024-03-04T12:03:07.097 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T12:03:07.097 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T12:03:07.097 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T12:03:07.097 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-03-04T12:03:07.120 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-03-04T12:03:07.137 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-03-04T12:03:07.137 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-03-04T12:03:07.194 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-03-04T12:03:07.194 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-03-04T12:03:07.194 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-03-04T12:03:07.194 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-03-04T12:03:07.194 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-03-04T12:03:07.194 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-03-04T12:03:07.194 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T12:03:07.194 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-03-04T12:03:07.195 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-03-04T12:03:07.195 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T12:03:07.195 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-03-04T12:03:07.195 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T12:03:07.195 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T12:03:07.195 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T12:03:07.195 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-03-04T12:03:07.220 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-04T12:03:07.232 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-04T12:03:07.239 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-04T12:03:07.239 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-03-04T12:03:07.242 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-03-04T12:03:07.244 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-03-04T12:03:07.260 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-04T12:03:07.267 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-04T12:03:07.268 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-03-04T12:03:07.300 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-03-04T12:03:07.302 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-03-04T12:03:07.428 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-04T12:03:07.505 INFO:teuthology.run_tasks:Running task kernel... 2024-03-04T12:03:07.521 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '786939134d7eee4dc7a805245d3f595929af37d7'} 2024-03-04T12:03:07.521 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-04T12:03:07.522 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-04T12:03:07.522 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T12:03:07.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:03:07.539 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-03-04T12:03:07.596 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-361.el9.x86_64 2024-03-04T12:03:07.597 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.14.0-361.el9.x86_64 2024-03-04T12:03:07.597 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum install -y kernel 2024-03-04T12:03:09.952 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-04T12:03:11.402 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-04T12:03:15.680 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:03 2024-03-04T12:03:30.020 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 19 MB 00:13 2024-03-04T12:03:35.525 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-03-04T12:03:37.098 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2024-03-04T12:03:41.702 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 4.6 MB/s | 20 MB 00:04 2024-03-04T12:03:46.750 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-04T12:03:49.098 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T12:03:49.099 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T12:03:49.196 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-04T12:03:49.216 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-04T12:03:49.216 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-03-04T12:03:49.216 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-04T12:03:49.216 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-04T12:03:49.216 INFO:teuthology.orchestra.run.smithi060.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T12:03:49.217 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-04T12:03:49.217 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T12:03:49.217 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T12:03:49.217 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T12:03:49.217 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:03:49.217 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-04T12:03:49.217 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-04T12:03:49.217 INFO:teuthology.orchestra.run.smithi060.stdout:Install 4 Packages 2024-03-04T12:03:49.217 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:03:49.218 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 100 M 2024-03-04T12:03:49.218 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 123 M 2024-03-04T12:03:49.218 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-04T12:03:49.626 INFO:teuthology.orchestra.run.smithi060.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 18 MB/s | 6.0 MB 00:00 2024-03-04T12:03:50.060 INFO:teuthology.orchestra.run.smithi060.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 27 MB/s | 21 MB 00:00 2024-03-04T12:03:51.377 INFO:teuthology.orchestra.run.smithi060.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 19 MB/s | 33 MB 00:01 2024-03-04T12:03:52.127 INFO:teuthology.orchestra.run.smithi060.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 14 MB/s | 39 MB 00:02 2024-03-04T12:03:52.127 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-04T12:03:52.127 INFO:teuthology.orchestra.run.smithi060.stdout:Total 34 MB/s | 100 MB 00:02 2024-03-04T12:03:53.047 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-04T12:03:53.510 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-04T12:03:53.510 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-04T12:03:54.889 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-04T12:03:54.889 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-04T12:03:57.756 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-04T12:03:58.444 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T12:03:58.571 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T12:04:01.495 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T12:04:01.675 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T12:04:06.774 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T12:04:06.896 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T12:04:12.026 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T12:04:42.880 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T12:04:42.881 INFO:teuthology.orchestra.run.smithi060.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T12:04:42.881 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:04:42.967 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T12:04:43.387 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T12:04:43.387 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T12:04:43.387 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T12:04:43.387 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T12:04:43.865 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T12:04:43.865 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:04:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-04T12:04:43.865 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T12:04:43.865 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T12:04:43.866 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T12:04:43.866 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T12:04:43.866 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:04:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-04T12:04:44.392 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T12:04:44.595 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T12:04:44.596 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T12:04:44.596 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T12:04:44.596 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T12:04:44.596 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T12:04:44.596 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-04T12:04:44.596 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T12:04:44.597 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T12:04:44.723 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T12:04:44.723 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T12:04:44.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -qi grub2-tools 2024-03-04T12:04:44.769 INFO:teuthology.orchestra.run.smithi060.stdout:Name : grub2-tools 2024-03-04T12:04:44.769 INFO:teuthology.orchestra.run.smithi060.stdout:Epoch : 1 2024-03-04T12:04:44.769 INFO:teuthology.orchestra.run.smithi060.stdout:Version : 2.06 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:Release : 68.el9 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: x86_64 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:Group : Unspecified 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:Size : 8264254 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:License : GPLv3+ 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:Packager : builder@centos.org 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:Vendor : CentOS 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T12:04:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:Summary : Support tools for GRUB. 2024-03-04T12:04:44.771 INFO:teuthology.orchestra.run.smithi060.stdout:Description : 2024-03-04T12:04:44.771 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:04:44.771 INFO:teuthology.orchestra.run.smithi060.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T12:04:44.771 INFO:teuthology.orchestra.run.smithi060.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T12:04:44.771 INFO:teuthology.orchestra.run.smithi060.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T12:04:44.771 INFO:teuthology.orchestra.run.smithi060.stdout:hardware devices. 2024-03-04T12:04:44.771 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:04:44.771 INFO:teuthology.orchestra.run.smithi060.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T12:04:44.772 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T12:04:44.772 INFO:teuthology.task.kernel:Updating grub on smithi060 to boot 5.14.0-427.el9.x86_64 2024-03-04T12:04:44.772 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T12:04:45.382 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2024-03-04T12:04:46.787 INFO:teuthology.orchestra.run.smithi060.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T12:04:46.834 INFO:teuthology.orchestra.run.smithi060.stderr:done 2024-03-04T12:04:46.837 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-03-04T12:04:46.852 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.0k7ntFIHlH 2024-03-04T12:04:46.852 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0k7ntFIHlH 2024-03-04T12:04:46.925 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.0k7ntFIHlH 2024-03-04T12:04:47.065 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.0k7ntFIHlH is 6KB 2024-03-04T12:04:47.113 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.0k7ntFIHlH 2024-03-04T12:04:47.128 DEBUG:teuthology.orchestra.run.smithi060:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T12:04:47.196 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T12:04:47.197 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T12:04:47.197 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T12:04:47.197 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T12:04:47.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T12:04:47.352 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2024-03-04T12:05:17.355 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-04T12:05:17.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-04T12:05:17.357 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T12:06:17.359 DEBUG:teuthology.orchestra.remote:timed out 2024-03-04T12:06:26.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-04T12:06:26.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T12:06:26.803 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-03-04T12:06:27.333 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-04T12:06:27.333 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-04T12:06:27.334 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-03-04T12:06:27.347 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-427.el9.x86_64 2024-03-04T12:06:27.347 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T12:06:27.348 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T12:06:27.348 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T12:06:28.348 DEBUG:teuthology.parallel:result is None 2024-03-04T12:06:28.349 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-04T12:06:28.356 INFO:teuthology.task.internal:Creating test directory... 2024-03-04T12:06:28.356 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T12:06:28.359 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T12:06:28.361 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T12:06:28.381 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-04T12:06:28.520 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-04T12:06:28.530 INFO:teuthology.task.internal:Creating archive directory... 2024-03-04T12:06:28.530 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T12:06:28.533 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T12:06:28.535 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T12:06:28.660 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-04T12:06:28.669 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-04T12:06:28.670 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T12:06:28.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:06:28.688 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T12:06:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:06:28.706 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T12:06:28.712 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T12:06:28.732 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T12:06:28.772 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T12:06:28.792 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T12:06:28.801 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T12:06:28.809 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T12:06:28.833 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T12:06:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T12:06:28.870 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-04T12:06:28.877 INFO:teuthology.task.internal:Configuring sudo... 2024-03-04T12:06:28.877 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T12:06:28.879 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T12:06:28.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T12:06:28.923 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-04T12:06:28.934 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-04T12:06:28.934 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T12:06:28.936 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T12:06:28.956 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T12:06:28.973 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T12:06:29.052 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T12:06:29.123 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T12:06:29.171 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T12:06:29.213 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:06:29.214 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T12:06:29.281 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T12:06:29.326 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T12:06:29.384 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T12:06:29.468 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T12:06:29.517 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-04T12:06:29.517 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T12:06:29.590 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T12:06:29.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T12:06:29.697 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T12:06:29.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T12:06:29.786 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T12:06:29.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T12:06:29.857 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-03-04T12:06:29.860 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-03-04T12:06:29.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-03-04T12:06:29.910 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T12:06:29.911 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T12:06:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T12:06:30.369 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-04T12:06:30.378 INFO:teuthology.task.internal:Starting timer... 2024-03-04T12:06:30.378 INFO:teuthology.run_tasks:Running task pcp... 2024-03-04T12:06:30.389 INFO:teuthology.run_tasks:Running task selinux... 2024-03-04T12:06:30.398 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-04T12:06:30.399 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2024-03-04T12:06:30.464 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2024-03-04T12:06:30.465 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-03-04T12:06:30.522 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-03-04T12:06:30.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-03-04T12:06:30.572 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-03-04T12:06:30.574 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-04T12:06:30.574 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2024-03-04T12:06:30.593 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2024-03-04T12:06:30.593 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-03-04T12:06:30.619 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-03-04T12:06:30.619 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-03-04T12:06:30.646 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-03-04T12:06:30.646 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi060.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi063.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2024-03-04T12:06:30.647 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-04T12:06:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:06:30.674 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-04T12:06:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:06:30.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-04T12:06:30.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:06:30.736 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-04T12:06:30.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2024-03-04T12:06:30.768 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-03-04T12:06:30.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-03-04T12:06:30.856 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-04T12:06:30.866 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-04T12:06:30.875 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-04T12:06:31.007 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-04T12:06:31.025 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-04T12:06:31.026 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com,smithi063.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-04T12:11:31.369 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com'), Remote(name='ubuntu@smithi063.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2024-03-04T12:11:31.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-04T12:11:31.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T12:11:31.462 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-03-04T12:11:31.543 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-04T12:11:31.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-04T12:11:31.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T12:11:31.647 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-03-04T12:11:31.715 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-03-04T12:11:31.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-04T12:11:31.716 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T12:11:31.805 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-03-04T12:11:31.876 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-04T12:11:31.877 INFO:teuthology.run_tasks:Running task clock... 2024-03-04T12:11:31.887 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-04T12:11:31.887 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T12:11:31.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T12:11:31.890 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T12:11:31.890 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T12:11:31.893 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T12:11:31.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T12:11:31.922 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T12:11:31.928 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T12:11:31.940 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T12:11:31.948 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T12:11:31.949 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T12:11:31.965 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T12:11:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2024-03-04T12:11:31.984 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2024-03-04T12:11:31.986 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-03-04T12:11:31.996 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-03-04T12:11:32.002 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T12:11:32.004 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-03-04T12:11:32.009 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-03-04T12:11:32.018 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T12:11:32.023 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T12:11:32.026 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T12:11:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T12:11:32.043 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T12:11:32.080 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-03-04T12:11:32.084 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T12:11:32.084 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-03-04T12:11:32.084 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:11:32.084 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:11:32.084 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:11:32.084 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:11:32.085 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-03-04T12:11:32.089 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T12:11:32.089 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-03-04T12:11:32.089 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:11:32.089 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:11:32.089 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:11:32.089 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:11:32.091 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-03-04T12:11:32.095 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T12:11:32.095 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-03-04T12:11:32.095 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:11:32.095 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:11:32.096 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:11:32.096 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:11:32.097 INFO:teuthology.run_tasks:Running task install... 2024-03-04T12:11:32.105 DEBUG:teuthology.task.install:project ceph 2024-03-04T12:11:32.105 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0'}} 2024-03-04T12:11:32.105 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0'} 2024-03-04T12:11:32.105 INFO:teuthology.task.install:Using flavor: default 2024-03-04T12:11:32.112 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-04T12:11:32.112 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-04T12:11:32.112 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0', 'tag': None, 'wait_for_package': False} 2024-03-04T12:11:32.113 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:11:32.114 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0', 'tag': None, 'wait_for_package': False} 2024-03-04T12:11:32.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:11:32.116 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0', 'tag': None, 'wait_for_package': False} 2024-03-04T12:11:32.116 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:11:32.255 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/ 2024-03-04T12:11:32.255 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1845.g717ce3cc 2024-03-04T12:11:32.267 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/ 2024-03-04T12:11:32.267 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1845.g717ce3cc 2024-03-04T12:11:32.340 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/ 2024-03-04T12:11:32.340 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1845.g717ce3cc 2024-03-04T12:11:32.375 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T12:11:32.375 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:11:32.375 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T12:11:32.404 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-04T12:11:32.404 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T12:11:32.438 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T12:11:32.438 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T12:11:32.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T12:11:32.447 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T12:11:32.447 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-04T12:11:32.447 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T12:11:32.465 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-04T12:11:32.466 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T12:11:32.475 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T12:11:32.480 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-04T12:11:32.480 DEBUG:teuthology.orchestra.run.smithi063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T12:11:32.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T12:11:32.552 DEBUG:teuthology.orchestra.run.smithi063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T12:11:32.555 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T12:11:32.614 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T12:11:32.621 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2024-03-04T12:11:32.622 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2024-03-04T12:11:32.638 DEBUG:teuthology.orchestra.run.smithi063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T12:11:32.667 INFO:teuthology.orchestra.run.smithi063.stdout:check_obsoletes = 1 2024-03-04T12:11:32.669 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum clean all 2024-03-04T12:11:32.680 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-03-04T12:11:32.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-03-04T12:11:32.873 INFO:teuthology.orchestra.run.smithi060.stdout:74 files removed 2024-03-04T12:11:32.901 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-04T12:11:32.917 INFO:teuthology.orchestra.run.smithi063.stdout:74 files removed 2024-03-04T12:11:32.918 INFO:teuthology.orchestra.run.smithi086.stdout:74 files removed 2024-03-04T12:11:32.944 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-04T12:11:32.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-04T12:11:33.650 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 193 kB/s | 82 kB 00:00 2024-03-04T12:11:33.710 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 171 kB/s | 82 kB 00:00 2024-03-04T12:11:33.713 INFO:teuthology.orchestra.run.smithi063.stdout:ceph packages for x86_64 178 kB/s | 82 kB 00:00 2024-03-04T12:11:34.030 INFO:teuthology.orchestra.run.smithi063.stdout:ceph noarch packages 36 kB/s | 10 kB 00:00 2024-03-04T12:11:34.101 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-04T12:11:34.152 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-04T12:11:34.544 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-03-04T12:11:34.547 INFO:teuthology.orchestra.run.smithi063.stdout:ceph source packages 3.9 kB/s | 1.9 kB 00:00 2024-03-04T12:11:34.569 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-04T12:11:35.488 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 7.5 MB/s | 6.7 MB 00:00 2024-03-04T12:11:36.413 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 3.7 MB/s | 6.7 MB 00:01 2024-03-04T12:11:36.981 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-04T12:11:37.417 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - AppStream 2.4 MB/s | 6.7 MB 00:02 2024-03-04T12:11:37.946 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-04T12:11:39.067 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-04T12:11:40.586 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-03-04T12:11:41.560 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-03-04T12:11:42.413 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:03 2024-03-04T12:11:47.719 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 19 MB 00:04 2024-03-04T12:11:47.753 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 3.2 MB/s | 19 MB 00:06 2024-03-04T12:11:53.656 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.1 MB 00:02 2024-03-04T12:11:53.859 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.1 MB 00:02 2024-03-04T12:11:55.153 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 15 kB 00:00 2024-03-04T12:11:55.788 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 15 kB 00:00 2024-03-04T12:11:57.040 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-03-04T12:11:57.398 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2024-03-04T12:12:02.013 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 7.6 kB/s | 1.7 kB 00:00 2024-03-04T12:12:02.479 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 993 kB/s | 19 MB 00:19 2024-03-04T12:12:02.549 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-04T12:12:04.518 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T12:12:04.519 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T12:12:04.533 INFO:teuthology.orchestra.run.smithi086.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-04T12:12:04.534 INFO:teuthology.orchestra.run.smithi086.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-04T12:12:04.535 INFO:teuthology.orchestra.run.smithi086.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-04T12:12:04.536 INFO:teuthology.orchestra.run.smithi086.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-04T12:12:04.538 INFO:teuthology.orchestra.run.smithi086.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-04T12:12:04.541 INFO:teuthology.orchestra.run.smithi086.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-04T12:12:04.542 INFO:teuthology.orchestra.run.smithi086.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-04T12:12:04.624 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-04T12:12:04.642 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-04T12:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-04T12:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-04T12:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-04T12:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-04T12:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 6.6 k 2024-03-04T12:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 5.3 M 2024-03-04T12:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 888 k 2024-03-04T12:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 145 k 2024-03-04T12:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 1.5 M 2024-03-04T12:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 138 k 2024-03-04T12:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 1.8 M 2024-03-04T12:12:04.643 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 7.4 M 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 49 k 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 9.8 M 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 47 M 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 264 k 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 749 k 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-mirror x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 220 k 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-shell x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 27 k 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-top noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 18 k 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 31 k 2024-03-04T12:12:04.644 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 745 k 2024-03-04T12:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 127 k 2024-03-04T12:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-04T12:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-04T12:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-04T12:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-04T12:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-04T12:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-04T12:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 162 k 2024-03-04T12:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 320 k 2024-03-04T12:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 298 k 2024-03-04T12:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 99 k 2024-03-04T12:12:04.645 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 85 k 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.1 M 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 170 k 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-04T12:12:04.646 INFO:teuthology.orchestra.run.smithi086.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.4 M 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.1 M 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-04T12:12:04.647 INFO:teuthology.orchestra.run.smithi086.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 20 M 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 31 k 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 2.1 M 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 248 k 2024-03-04T12:12:04.648 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 4.7 M 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 17 M 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 16 k 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 25 k 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T12:12:04.649 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 163 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 502 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 5.1 M 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T12:12:04.650 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T12:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T12:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T12:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T12:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T12:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-04T12:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T12:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T12:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T12:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-04T12:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T12:12:04.651 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T12:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T12:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 45 k 2024-03-04T12:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 131 k 2024-03-04T12:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T12:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T12:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T12:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T12:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-04T12:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-04T12:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T12:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T12:12:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T12:12:04.653 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T12:12:04.654 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T12:12:04.655 INFO:teuthology.orchestra.run.smithi086.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-04T12:12:04.656 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2024-03-04T12:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T12:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-04T12:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-04T12:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout:Install 150 Packages 2024-03-04T12:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 26 Packages 2024-03-04T12:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:12:04.657 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 223 M 2024-03-04T12:12:04.658 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-04T12:12:05.204 INFO:teuthology.orchestra.run.smithi063.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T12:12:05.205 INFO:teuthology.orchestra.run.smithi063.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T12:12:05.218 INFO:teuthology.orchestra.run.smithi063.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-04T12:12:05.219 INFO:teuthology.orchestra.run.smithi063.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-04T12:12:05.220 INFO:teuthology.orchestra.run.smithi063.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-04T12:12:05.221 INFO:teuthology.orchestra.run.smithi063.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-04T12:12:05.222 INFO:teuthology.orchestra.run.smithi063.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-04T12:12:05.225 INFO:teuthology.orchestra.run.smithi063.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-04T12:12:05.226 INFO:teuthology.orchestra.run.smithi063.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-04T12:12:05.267 INFO:teuthology.orchestra.run.smithi086.stdout:(1/176): ceph-19.0.0-1845.g717ce3cc.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-03-04T12:12:05.315 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-03-04T12:12:05.331 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-04T12:12:05.331 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-03-04T12:12:05.331 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-04T12:12:05.331 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-03-04T12:12:05.331 INFO:teuthology.orchestra.run.smithi063.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-04T12:12:05.331 INFO:teuthology.orchestra.run.smithi063.stdout: ceph x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 6.6 k 2024-03-04T12:12:05.331 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 5.3 M 2024-03-04T12:12:05.331 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 888 k 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 145 k 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 1.5 M 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 138 k 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 1.8 M 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 7.4 M 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 49 k 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 9.8 M 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 47 M 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 264 k 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 749 k 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-mirror x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 220 k 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-shell x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 27 k 2024-03-04T12:12:05.332 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-top noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 18 k 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 31 k 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 745 k 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 127 k 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 162 k 2024-03-04T12:12:05.333 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 320 k 2024-03-04T12:12:05.334 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 298 k 2024-03-04T12:12:05.334 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 99 k 2024-03-04T12:12:05.334 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 85 k 2024-03-04T12:12:05.334 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.1 M 2024-03-04T12:12:05.334 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 170 k 2024-03-04T12:12:05.334 INFO:teuthology.orchestra.run.smithi063.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-04T12:12:05.334 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-03-04T12:12:05.334 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-04T12:12:05.334 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-04T12:12:05.334 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-04T12:12:05.334 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-04T12:12:05.334 INFO:teuthology.orchestra.run.smithi063.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.4 M 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.1 M 2024-03-04T12:12:05.335 INFO:teuthology.orchestra.run.smithi063.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 20 M 2024-03-04T12:12:05.336 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 31 k 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 2.1 M 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 248 k 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 4.7 M 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 17 M 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 16 k 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 25 k 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T12:12:05.337 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T12:12:05.338 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T12:12:05.338 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 163 k 2024-03-04T12:12:05.338 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T12:12:05.338 INFO:teuthology.orchestra.run.smithi063.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T12:12:05.338 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T12:12:05.338 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T12:12:05.338 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T12:12:05.338 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 502 k 2024-03-04T12:12:05.338 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T12:12:05.338 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 5.1 M 2024-03-04T12:12:05.338 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T12:12:05.338 INFO:teuthology.orchestra.run.smithi063.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-04T12:12:05.339 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 45 k 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 131 k 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T12:12:05.340 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T12:12:05.341 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T12:12:05.342 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T12:12:05.343 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T12:12:05.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T12:12:05.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T12:12:05.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-04T12:12:05.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T12:12:05.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T12:12:05.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T12:12:05.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T12:12:05.344 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T12:12:05.344 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T12:12:05.344 INFO:teuthology.orchestra.run.smithi063.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T12:12:05.344 INFO:teuthology.orchestra.run.smithi063.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T12:12:05.344 INFO:teuthology.orchestra.run.smithi063.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T12:12:05.345 INFO:teuthology.orchestra.run.smithi063.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-04T12:12:05.345 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T12:12:05.345 INFO:teuthology.orchestra.run.smithi063.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-04T12:12:05.345 INFO:teuthology.orchestra.run.smithi063.stdout:Installing weak dependencies: 2024-03-04T12:12:05.345 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T12:12:05.345 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:12:05.345 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-03-04T12:12:05.345 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================================ 2024-03-04T12:12:05.345 INFO:teuthology.orchestra.run.smithi063.stdout:Install 150 Packages 2024-03-04T12:12:05.345 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 26 Packages 2024-03-04T12:12:05.345 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:12:05.346 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 223 M 2024-03-04T12:12:05.346 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-03-04T12:12:05.956 INFO:teuthology.orchestra.run.smithi086.stdout:(2/176): ceph-base-19.0.0-1845.g717ce3cc.el9.x8 5.4 MB/s | 5.3 MB 00:00 2024-03-04T12:12:05.989 INFO:teuthology.orchestra.run.smithi086.stdout:(3/176): ceph-fuse-19.0.0-1845.g717ce3cc.el9.x8 1.2 MB/s | 888 kB 00:00 2024-03-04T12:12:06.006 INFO:teuthology.orchestra.run.smithi086.stdout:(4/176): ceph-immutable-object-cache-19.0.0-184 2.8 MB/s | 145 kB 00:00 2024-03-04T12:12:06.165 INFO:teuthology.orchestra.run.smithi086.stdout:(5/176): ceph-mgr-19.0.0-1845.g717ce3cc.el9.x86 9.4 MB/s | 1.5 MB 00:00 2024-03-04T12:12:06.275 INFO:teuthology.orchestra.run.smithi063.stdout:(1/176): ceph-19.0.0-1845.g717ce3cc.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-03-04T12:12:06.699 INFO:teuthology.orchestra.run.smithi086.stdout:(6/176): ceph-mon-19.0.0-1845.g717ce3cc.el9.x86 8.8 MB/s | 4.7 MB 00:00 2024-03-04T12:12:07.124 INFO:teuthology.orchestra.run.smithi086.stdout:(7/176): ceph-common-19.0.0-1845.g717ce3cc.el9. 9.2 MB/s | 20 MB 00:02 2024-03-04T12:12:07.174 INFO:teuthology.orchestra.run.smithi086.stdout:(8/176): ceph-mds-19.0.0-1845.g717ce3cc.el9.x86 1.8 MB/s | 2.1 MB 00:01 2024-03-04T12:12:07.175 INFO:teuthology.orchestra.run.smithi063.stdout:(2/176): ceph-fuse-19.0.0-1845.g717ce3cc.el9.x8 986 kB/s | 888 kB 00:00 2024-03-04T12:12:07.283 INFO:teuthology.orchestra.run.smithi086.stdout:(9/176): ceph-selinux-19.0.0-1845.g717ce3cc.el9 232 kB/s | 25 kB 00:00 2024-03-04T12:12:07.376 INFO:teuthology.orchestra.run.smithi063.stdout:(3/176): ceph-immutable-object-cache-19.0.0-184 725 kB/s | 145 kB 00:00 2024-03-04T12:12:07.985 INFO:teuthology.orchestra.run.smithi063.stdout:(4/176): ceph-base-19.0.0-1845.g717ce3cc.el9.x8 2.7 MB/s | 5.3 MB 00:01 2024-03-04T12:12:07.991 INFO:teuthology.orchestra.run.smithi086.stdout:(10/176): ceph-radosgw-19.0.0-1845.g717ce3cc.el 11 MB/s | 9.8 MB 00:00 2024-03-04T12:12:08.075 INFO:teuthology.orchestra.run.smithi086.stdout:(11/176): cephfs-mirror-19.0.0-1845.g717ce3cc.e 2.6 MB/s | 220 kB 00:00 2024-03-04T12:12:08.142 INFO:teuthology.orchestra.run.smithi086.stdout:(12/176): cephfs-shell-19.0.0-1845.g717ce3cc.el 405 kB/s | 27 kB 00:00 2024-03-04T12:12:08.208 INFO:teuthology.orchestra.run.smithi086.stdout:(13/176): libcephfs-devel-19.0.0-1845.g717ce3cc 469 kB/s | 31 kB 00:00 2024-03-04T12:12:08.235 INFO:teuthology.orchestra.run.smithi063.stdout:(5/176): ceph-mgr-19.0.0-1845.g717ce3cc.el9.x86 6.0 MB/s | 1.5 MB 00:00 2024-03-04T12:12:08.393 INFO:teuthology.orchestra.run.smithi086.stdout:(14/176): ceph-osd-19.0.0-1845.g717ce3cc.el9.x8 9.9 MB/s | 17 MB 00:01 2024-03-04T12:12:08.427 INFO:teuthology.orchestra.run.smithi086.stdout:(15/176): libcephfs2-19.0.0-1845.g717ce3cc.el9. 3.3 MB/s | 745 kB 00:00 2024-03-04T12:12:08.444 INFO:teuthology.orchestra.run.smithi063.stdout:(6/176): ceph-mds-19.0.0-1845.g717ce3cc.el9.x86 2.0 MB/s | 2.1 MB 00:01 2024-03-04T12:12:08.451 INFO:teuthology.orchestra.run.smithi086.stdout:(16/176): libcephsqlite-19.0.0-1845.g717ce3cc.e 2.7 MB/s | 163 kB 00:00 2024-03-04T12:12:08.468 INFO:teuthology.orchestra.run.smithi086.stdout:(17/176): librados-devel-19.0.0-1845.g717ce3cc. 3.0 MB/s | 127 kB 00:00 2024-03-04T12:12:08.535 INFO:teuthology.orchestra.run.smithi086.stdout:(18/176): libradosstriper1-19.0.0-1845.g717ce3c 5.9 MB/s | 502 kB 00:00 2024-03-04T12:12:08.644 INFO:teuthology.orchestra.run.smithi086.stdout:(19/176): python3-ceph-argparse-19.0.0-1845.g71 417 kB/s | 45 kB 00:00 2024-03-04T12:12:08.719 INFO:teuthology.orchestra.run.smithi086.stdout:(20/176): python3-ceph-common-19.0.0-1845.g717c 1.7 MB/s | 131 kB 00:00 2024-03-04T12:12:08.749 INFO:teuthology.orchestra.run.smithi063.stdout:(7/176): ceph-mon-19.0.0-1845.g717ce3cc.el9.x86 9.1 MB/s | 4.7 MB 00:00 2024-03-04T12:12:08.795 INFO:teuthology.orchestra.run.smithi086.stdout:(21/176): python3-cephfs-19.0.0-1845.g717ce3cc. 2.1 MB/s | 162 kB 00:00 2024-03-04T12:12:08.895 INFO:teuthology.orchestra.run.smithi086.stdout:(22/176): librgw2-19.0.0-1845.g717ce3cc.el9.x86 12 MB/s | 5.1 MB 00:00 2024-03-04T12:12:08.920 INFO:teuthology.orchestra.run.smithi086.stdout:(23/176): python3-rados-19.0.0-1845.g717ce3cc.e 2.5 MB/s | 320 kB 00:00 2024-03-04T12:12:08.970 INFO:teuthology.orchestra.run.smithi086.stdout:(24/176): python3-rbd-19.0.0-1845.g717ce3cc.el9 3.9 MB/s | 298 kB 00:00 2024-03-04T12:12:08.987 INFO:teuthology.orchestra.run.smithi086.stdout:(25/176): python3-rgw-19.0.0-1845.g717ce3cc.el9 1.5 MB/s | 99 kB 00:00 2024-03-04T12:12:09.038 INFO:teuthology.orchestra.run.smithi086.stdout:(26/176): rbd-fuse-19.0.0-1845.g717ce3cc.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-04T12:12:09.129 INFO:teuthology.orchestra.run.smithi086.stdout:(27/176): rbd-nbd-19.0.0-1845.g717ce3cc.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-04T12:12:09.213 INFO:teuthology.orchestra.run.smithi086.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1845.g 378 kB/s | 31 kB 00:00 2024-03-04T12:12:09.313 INFO:teuthology.orchestra.run.smithi086.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1845.g717ce3c 1.4 MB/s | 138 kB 00:00 2024-03-04T12:12:09.380 INFO:teuthology.orchestra.run.smithi086.stdout:(30/176): rbd-mirror-19.0.0-1845.g717ce3cc.el9. 7.9 MB/s | 3.1 MB 00:00 2024-03-04T12:12:09.564 INFO:teuthology.orchestra.run.smithi086.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1845.g717ce 7.1 MB/s | 1.8 MB 00:00 2024-03-04T12:12:09.614 INFO:teuthology.orchestra.run.smithi086.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1845.g71 4.8 MB/s | 248 kB 00:00 2024-03-04T12:12:09.644 INFO:teuthology.orchestra.run.smithi063.stdout:(8/176): ceph-radosgw-19.0.0-1845.g717ce3cc.el9 11 MB/s | 9.8 MB 00:00 2024-03-04T12:12:09.656 INFO:teuthology.orchestra.run.smithi086.stdout:(33/176): ceph-mgr-rook-19.0.0-1845.g717ce3cc.e 1.2 MB/s | 49 kB 00:00 2024-03-04T12:12:09.686 INFO:teuthology.orchestra.run.smithi063.stdout:(9/176): ceph-selinux-19.0.0-1845.g717ce3cc.el9 606 kB/s | 25 kB 00:00 2024-03-04T12:12:09.706 INFO:teuthology.orchestra.run.smithi086.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1845.g7 329 kB/s | 16 kB 00:00 2024-03-04T12:12:09.773 INFO:teuthology.orchestra.run.smithi086.stdout:(35/176): ceph-volume-19.0.0-1845.g717ce3cc.el9 3.9 MB/s | 264 kB 00:00 2024-03-04T12:12:09.957 INFO:teuthology.orchestra.run.smithi086.stdout:(36/176): cephadm-19.0.0-1845.g717ce3cc.el9.noa 4.0 MB/s | 749 kB 00:00 2024-03-04T12:12:10.024 INFO:teuthology.orchestra.run.smithi086.stdout:(37/176): cephfs-top-19.0.0-1845.g717ce3cc.el9. 274 kB/s | 18 kB 00:00 2024-03-04T12:12:10.111 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-03-04T12:12:10.157 INFO:teuthology.orchestra.run.smithi086.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 4.9 MB/s | 662 kB 00:00 2024-03-04T12:12:10.216 INFO:teuthology.orchestra.run.smithi086.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-04T12:12:10.258 INFO:teuthology.orchestra.run.smithi086.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-04T12:12:10.283 INFO:teuthology.orchestra.run.smithi086.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 821 kB/s | 20 kB 00:00 2024-03-04T12:12:10.325 INFO:teuthology.orchestra.run.smithi086.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-04T12:12:10.351 INFO:teuthology.orchestra.run.smithi086.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 1.2 MB/s | 32 kB 00:00 2024-03-04T12:12:10.459 INFO:teuthology.orchestra.run.smithi086.stdout:(44/176): ceph-mgr-diskprediction-local-19.0.0- 6.9 MB/s | 7.4 MB 00:01 2024-03-04T12:12:10.509 INFO:teuthology.orchestra.run.smithi086.stdout:(45/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-03-04T12:12:10.535 INFO:teuthology.orchestra.run.smithi086.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 237 kB/s | 17 kB 00:00 2024-03-04T12:12:10.560 INFO:teuthology.orchestra.run.smithi086.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-04T12:12:10.585 INFO:teuthology.orchestra.run.smithi086.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-04T12:12:10.611 INFO:teuthology.orchestra.run.smithi086.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-04T12:12:10.636 INFO:teuthology.orchestra.run.smithi086.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-04T12:12:10.670 INFO:teuthology.orchestra.run.smithi086.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-04T12:12:10.695 INFO:teuthology.orchestra.run.smithi086.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 424 kB/s | 24 kB 00:00 2024-03-04T12:12:10.720 INFO:teuthology.orchestra.run.smithi086.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 928 kB/s | 46 kB 00:00 2024-03-04T12:12:10.745 INFO:teuthology.orchestra.run.smithi086.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-04T12:12:10.771 INFO:teuthology.orchestra.run.smithi086.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-04T12:12:10.791 INFO:teuthology.orchestra.run.smithi086.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 757 kB/s | 34 kB 00:00 2024-03-04T12:12:10.925 INFO:teuthology.orchestra.run.smithi086.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 31 MB/s | 4.7 MB 00:00 2024-03-04T12:12:10.992 INFO:teuthology.orchestra.run.smithi086.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 15 MB/s | 3.0 MB 00:00 2024-03-04T12:12:11.034 INFO:teuthology.orchestra.run.smithi086.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 14 MB/s | 1.5 MB 00:00 2024-03-04T12:12:11.068 INFO:teuthology.orchestra.run.smithi086.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-04T12:12:11.093 INFO:teuthology.orchestra.run.smithi086.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-04T12:12:11.260 INFO:teuthology.orchestra.run.smithi086.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-04T12:12:11.294 INFO:teuthology.orchestra.run.smithi086.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 5.1 MB/s | 1.0 MB 00:00 2024-03-04T12:12:11.319 INFO:teuthology.orchestra.run.smithi086.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-04T12:12:11.494 INFO:teuthology.orchestra.run.smithi086.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-04T12:12:11.519 INFO:teuthology.orchestra.run.smithi086.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.1 MB/s | 227 kB 00:00 2024-03-04T12:12:11.545 INFO:teuthology.orchestra.run.smithi086.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 491 kB/s | 24 kB 00:00 2024-03-04T12:12:11.570 INFO:teuthology.orchestra.run.smithi086.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 360 kB/s | 18 kB 00:00 2024-03-04T12:12:11.595 INFO:teuthology.orchestra.run.smithi086.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-04T12:12:11.621 INFO:teuthology.orchestra.run.smithi086.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 626 kB/s | 32 kB 00:00 2024-03-04T12:12:11.788 INFO:teuthology.orchestra.run.smithi086.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-04T12:12:11.813 INFO:teuthology.orchestra.run.smithi086.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-04T12:12:11.838 INFO:teuthology.orchestra.run.smithi086.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-04T12:12:11.863 INFO:teuthology.orchestra.run.smithi086.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-04T12:12:11.883 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2024-03-04T12:12:11.889 INFO:teuthology.orchestra.run.smithi086.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 821 kB/s | 41 kB 00:00 2024-03-04T12:12:11.985 INFO:teuthology.orchestra.run.smithi086.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 1.9 MB/s | 238 kB 00:00 2024-03-04T12:12:12.015 INFO:teuthology.orchestra.run.smithi086.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-03-04T12:12:12.040 INFO:teuthology.orchestra.run.smithi086.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 3.8 MB/s | 211 kB 00:00 2024-03-04T12:12:12.065 INFO:teuthology.orchestra.run.smithi086.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 953 kB/s | 47 kB 00:00 2024-03-04T12:12:12.090 INFO:teuthology.orchestra.run.smithi086.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 855 kB/s | 43 kB 00:00 2024-03-04T12:12:12.249 INFO:teuthology.orchestra.run.smithi086.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 245 kB/s | 39 kB 00:00 2024-03-04T12:12:12.486 INFO:teuthology.orchestra.run.smithi063.stdout:(10/176): ceph-test-19.0.0-1845.g717ce3cc.el9.x 17 MB/s | 47 MB 00:02 2024-03-04T12:12:12.525 INFO:teuthology.orchestra.run.smithi086.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-03-04T12:12:12.545 INFO:teuthology.orchestra.run.smithi063.stdout:(11/176): cephfs-mirror-19.0.0-1845.g717ce3cc.e 3.7 MB/s | 220 kB 00:00 2024-03-04T12:12:12.550 INFO:teuthology.orchestra.run.smithi086.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 138 kB/s | 41 kB 00:00 2024-03-04T12:12:12.575 INFO:teuthology.orchestra.run.smithi086.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 215 kB 00:00 2024-03-04T12:12:12.595 INFO:teuthology.orchestra.run.smithi063.stdout:(12/176): cephfs-shell-19.0.0-1845.g717ce3cc.el 542 kB/s | 27 kB 00:00 2024-03-04T12:12:12.600 INFO:teuthology.orchestra.run.smithi086.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 679 kB/s | 34 kB 00:00 2024-03-04T12:12:12.625 INFO:teuthology.orchestra.run.smithi086.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 866 kB/s | 43 kB 00:00 2024-03-04T12:12:12.645 INFO:teuthology.orchestra.run.smithi063.stdout:(13/176): libcephfs-devel-19.0.0-1845.g717ce3cc 626 kB/s | 31 kB 00:00 2024-03-04T12:12:12.651 INFO:teuthology.orchestra.run.smithi086.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-04T12:12:12.676 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 31 MB/s | 20 MB 00:00 2024-03-04T12:12:12.676 INFO:teuthology.orchestra.run.smithi086.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-04T12:12:12.693 INFO:teuthology.orchestra.run.smithi086.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2024-03-04T12:12:12.718 INFO:teuthology.orchestra.run.smithi086.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-04T12:12:12.721 INFO:teuthology.orchestra.run.smithi063.stdout:(14/176): libcephfs2-19.0.0-1845.g717ce3cc.el9. 9.7 MB/s | 745 kB 00:00 2024-03-04T12:12:12.744 INFO:teuthology.orchestra.run.smithi086.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-04T12:12:12.769 INFO:teuthology.orchestra.run.smithi086.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-04T12:12:12.779 INFO:teuthology.orchestra.run.smithi063.stdout:(15/176): libcephsqlite-19.0.0-1845.g717ce3cc.e 2.7 MB/s | 163 kB 00:00 2024-03-04T12:12:12.794 INFO:teuthology.orchestra.run.smithi086.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-03-04T12:12:12.828 INFO:teuthology.orchestra.run.smithi086.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-04T12:12:12.830 INFO:teuthology.orchestra.run.smithi063.stdout:(16/176): librados-devel-19.0.0-1845.g717ce3cc. 2.5 MB/s | 127 kB 00:00 2024-03-04T12:12:12.844 INFO:teuthology.orchestra.run.smithi086.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 194 kB 00:00 2024-03-04T12:12:12.870 INFO:teuthology.orchestra.run.smithi086.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-04T12:12:12.895 INFO:teuthology.orchestra.run.smithi086.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-04T12:12:12.897 INFO:teuthology.orchestra.run.smithi063.stdout:(17/176): libradosstriper1-19.0.0-1845.g717ce3c 7.4 MB/s | 502 kB 00:00 2024-03-04T12:12:12.937 INFO:teuthology.orchestra.run.smithi086.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-04T12:12:12.971 INFO:teuthology.orchestra.run.smithi086.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-04T12:12:12.997 INFO:teuthology.orchestra.run.smithi086.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.1 MB/s | 124 kB 00:00 2024-03-04T12:12:13.022 INFO:teuthology.orchestra.run.smithi086.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 2.2 MB/s | 113 kB 00:00 2024-03-04T12:12:13.055 INFO:teuthology.orchestra.run.smithi086.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 3.1 MB/s | 187 kB 00:00 2024-03-04T12:12:13.214 INFO:teuthology.orchestra.run.smithi086.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-04T12:12:13.230 INFO:teuthology.orchestra.run.smithi063.stdout:(18/176): librgw2-19.0.0-1845.g717ce3cc.el9.x86 15 MB/s | 5.1 MB 00:00 2024-03-04T12:12:13.280 INFO:teuthology.orchestra.run.smithi063.stdout:(19/176): python3-ceph-argparse-19.0.0-1845.g71 910 kB/s | 45 kB 00:00 2024-03-04T12:12:13.331 INFO:teuthology.orchestra.run.smithi063.stdout:(20/176): python3-ceph-common-19.0.0-1845.g717c 2.6 MB/s | 131 kB 00:00 2024-03-04T12:12:13.339 INFO:teuthology.orchestra.run.smithi086.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 775 kB/s | 245 kB 00:00 2024-03-04T12:12:13.381 INFO:teuthology.orchestra.run.smithi063.stdout:(21/176): python3-cephfs-19.0.0-1845.g717ce3cc. 3.2 MB/s | 162 kB 00:00 2024-03-04T12:12:13.431 INFO:teuthology.orchestra.run.smithi063.stdout:(22/176): python3-rados-19.0.0-1845.g717ce3cc.e 6.2 MB/s | 320 kB 00:00 2024-03-04T12:12:13.482 INFO:teuthology.orchestra.run.smithi063.stdout:(23/176): python3-rbd-19.0.0-1845.g717ce3cc.el9 5.8 MB/s | 298 kB 00:00 2024-03-04T12:12:13.523 INFO:teuthology.orchestra.run.smithi086.stdout:(105/176): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-03-04T12:12:13.532 INFO:teuthology.orchestra.run.smithi063.stdout:(24/176): python3-rgw-19.0.0-1845.g717ce3cc.el9 1.9 MB/s | 99 kB 00:00 2024-03-04T12:12:13.582 INFO:teuthology.orchestra.run.smithi063.stdout:(25/176): rbd-fuse-19.0.0-1845.g717ce3cc.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-04T12:12:13.783 INFO:teuthology.orchestra.run.smithi063.stdout:(26/176): ceph-osd-19.0.0-1845.g717ce3cc.el9.x8 3.1 MB/s | 17 MB 00:05 2024-03-04T12:12:13.925 INFO:teuthology.orchestra.run.smithi063.stdout:(27/176): rbd-mirror-19.0.0-1845.g717ce3cc.el9. 9.0 MB/s | 3.1 MB 00:00 2024-03-04T12:12:13.951 INFO:teuthology.orchestra.run.smithi063.stdout:(28/176): rbd-nbd-19.0.0-1845.g717ce3cc.el9.x86 1.0 MB/s | 170 kB 00:00 2024-03-04T12:12:13.967 INFO:teuthology.orchestra.run.smithi063.stdout:(29/176): ceph-grafana-dashboards-19.0.0-1845.g 758 kB/s | 31 kB 00:00 2024-03-04T12:12:14.009 INFO:teuthology.orchestra.run.smithi063.stdout:(30/176): ceph-mgr-cephadm-19.0.0-1845.g717ce3c 2.3 MB/s | 138 kB 00:00 2024-03-04T12:12:14.101 INFO:teuthology.orchestra.run.smithi063.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1845.g717ce 13 MB/s | 1.8 MB 00:00 2024-03-04T12:12:14.160 INFO:teuthology.orchestra.run.smithi063.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1845.g71 4.2 MB/s | 248 kB 00:00 2024-03-04T12:12:14.210 INFO:teuthology.orchestra.run.smithi063.stdout:(33/176): ceph-mgr-rook-19.0.0-1845.g717ce3cc.e 993 kB/s | 49 kB 00:00 2024-03-04T12:12:14.260 INFO:teuthology.orchestra.run.smithi063.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1845.g7 329 kB/s | 16 kB 00:00 2024-03-04T12:12:14.319 INFO:teuthology.orchestra.run.smithi063.stdout:(35/176): ceph-volume-19.0.0-1845.g717ce3cc.el9 4.4 MB/s | 264 kB 00:00 2024-03-04T12:12:14.381 INFO:teuthology.orchestra.run.smithi086.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 360 kB/s | 309 kB 00:00 2024-03-04T12:12:14.394 INFO:teuthology.orchestra.run.smithi063.stdout:(36/176): cephadm-19.0.0-1845.g717ce3cc.el9.noa 9.8 MB/s | 749 kB 00:00 2024-03-04T12:12:14.436 INFO:teuthology.orchestra.run.smithi063.stdout:(37/176): cephfs-top-19.0.0-1845.g717ce3cc.el9. 438 kB/s | 18 kB 00:00 2024-03-04T12:12:14.504 INFO:teuthology.orchestra.run.smithi086.stdout:(107/176): abseil-cpp-20211102.0-3.el9.x86_64.r 471 kB/s | 548 kB 00:01 2024-03-04T12:12:14.532 INFO:teuthology.orchestra.run.smithi086.stdout:(108/176): grpc-data-1.46.7-10.el9.noarch.rpm 129 kB/s | 19 kB 00:00 2024-03-04T12:12:14.641 INFO:teuthology.orchestra.run.smithi086.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-03-04T12:12:14.703 INFO:teuthology.orchestra.run.smithi063.stdout:(38/176): ceph-common-19.0.0-1845.g717ce3cc.el9 2.3 MB/s | 20 MB 00:08 2024-03-04T12:12:14.737 INFO:teuthology.orchestra.run.smithi063.stdout:(39/176): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 662 kB 00:00 2024-03-04T12:12:14.741 INFO:teuthology.orchestra.run.smithi086.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-04T12:12:14.779 INFO:teuthology.orchestra.run.smithi063.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 920 kB 00:00 2024-03-04T12:12:14.804 INFO:teuthology.orchestra.run.smithi063.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-04T12:12:14.829 INFO:teuthology.orchestra.run.smithi063.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 409 kB/s | 20 kB 00:00 2024-03-04T12:12:14.850 INFO:teuthology.orchestra.run.smithi086.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 621 kB/s | 67 kB 00:00 2024-03-04T12:12:14.863 INFO:teuthology.orchestra.run.smithi063.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 5.1 MB/s | 307 kB 00:00 2024-03-04T12:12:14.888 INFO:teuthology.orchestra.run.smithi063.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 546 kB/s | 32 kB 00:00 2024-03-04T12:12:14.964 INFO:teuthology.orchestra.run.smithi063.stdout:(45/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-04T12:12:14.989 INFO:teuthology.orchestra.run.smithi063.stdout:(46/176): flexiblas-openblas-openmp-3.0.4-8.el9 174 kB/s | 17 kB 00:00 2024-03-04T12:12:15.014 INFO:teuthology.orchestra.run.smithi063.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-04T12:12:15.048 INFO:teuthology.orchestra.run.smithi063.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-04T12:12:15.069 INFO:teuthology.orchestra.run.smithi063.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 823 kB/s | 44 kB 00:00 2024-03-04T12:12:15.094 INFO:teuthology.orchestra.run.smithi063.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 14 MB/s | 661 kB 00:00 2024-03-04T12:12:15.117 INFO:teuthology.orchestra.run.smithi086.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 567 kB/s | 151 kB 00:00 2024-03-04T12:12:15.127 INFO:teuthology.orchestra.run.smithi063.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-04T12:12:15.144 INFO:teuthology.orchestra.run.smithi063.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 496 kB/s | 24 kB 00:00 2024-03-04T12:12:15.162 INFO:teuthology.orchestra.run.smithi063.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-04T12:12:15.187 INFO:teuthology.orchestra.run.smithi063.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-04T12:12:15.212 INFO:teuthology.orchestra.run.smithi063.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-04T12:12:15.246 INFO:teuthology.orchestra.run.smithi063.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 586 kB/s | 34 kB 00:00 2024-03-04T12:12:15.355 INFO:teuthology.orchestra.run.smithi063.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 33 MB/s | 4.7 MB 00:00 2024-03-04T12:12:15.413 INFO:teuthology.orchestra.run.smithi063.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 18 MB/s | 3.0 MB 00:00 2024-03-04T12:12:15.455 INFO:teuthology.orchestra.run.smithi063.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 16 MB/s | 1.5 MB 00:00 2024-03-04T12:12:15.514 INFO:teuthology.orchestra.run.smithi063.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 8.2 MB/s | 811 kB 00:00 2024-03-04T12:12:15.539 INFO:teuthology.orchestra.run.smithi063.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 1.6 MB/s | 134 kB 00:00 2024-03-04T12:12:15.681 INFO:teuthology.orchestra.run.smithi063.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-04T12:12:15.723 INFO:teuthology.orchestra.run.smithi063.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 5.6 MB/s | 1.0 MB 00:00 2024-03-04T12:12:15.748 INFO:teuthology.orchestra.run.smithi063.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-03-04T12:12:15.899 INFO:teuthology.orchestra.run.smithi063.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 5.8 MB 00:00 2024-03-04T12:12:15.924 INFO:teuthology.orchestra.run.smithi063.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.3 MB/s | 227 kB 00:00 2024-03-04T12:12:15.941 INFO:teuthology.orchestra.run.smithi063.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 590 kB/s | 24 kB 00:00 2024-03-04T12:12:15.957 INFO:teuthology.orchestra.run.smithi063.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 543 kB/s | 18 kB 00:00 2024-03-04T12:12:15.983 INFO:teuthology.orchestra.run.smithi063.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-04T12:12:16.000 INFO:teuthology.orchestra.run.smithi063.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 757 kB/s | 32 kB 00:00 2024-03-04T12:12:16.092 INFO:teuthology.orchestra.run.smithi063.stdout:(71/176): ceph-mgr-diskprediction-local-19.0.0- 3.6 MB/s | 7.4 MB 00:02 2024-03-04T12:12:16.175 INFO:teuthology.orchestra.run.smithi063.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-04T12:12:16.209 INFO:teuthology.orchestra.run.smithi063.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2024-03-04T12:12:16.242 INFO:teuthology.orchestra.run.smithi063.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-04T12:12:16.243 INFO:teuthology.orchestra.run.smithi086.stdout:(113/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 745 kB/s | 838 kB 00:01 2024-03-04T12:12:16.268 INFO:teuthology.orchestra.run.smithi063.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 751 kB/s | 69 kB 00:00 2024-03-04T12:12:16.284 INFO:teuthology.orchestra.run.smithi063.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 989 kB/s | 41 kB 00:00 2024-03-04T12:12:16.310 INFO:teuthology.orchestra.run.smithi063.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-04T12:12:16.335 INFO:teuthology.orchestra.run.smithi063.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-04T12:12:16.352 INFO:teuthology.orchestra.run.smithi063.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-04T12:12:16.386 INFO:teuthology.orchestra.run.smithi063.stdout:(80/176): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-04T12:12:16.402 INFO:teuthology.orchestra.run.smithi063.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 639 kB/s | 43 kB 00:00 2024-03-04T12:12:16.718 INFO:teuthology.orchestra.run.smithi063.stdout:(82/176): python3-toml-0.10.2-6.el9.noarch.rpm 123 kB/s | 39 kB 00:00 2024-03-04T12:12:16.736 INFO:teuthology.orchestra.run.smithi063.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 124 kB/s | 41 kB 00:00 2024-03-04T12:12:16.887 INFO:teuthology.orchestra.run.smithi063.stdout:(84/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-03-04T12:12:16.928 INFO:teuthology.orchestra.run.smithi063.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 1.0 MB/s | 215 kB 00:00 2024-03-04T12:12:16.942 INFO:teuthology.orchestra.run.smithi086.stdout:(114/176): python3-asyncssh-2.13.2-4.el9.noarch 783 kB/s | 548 kB 00:00 2024-03-04T12:12:16.945 INFO:teuthology.orchestra.run.smithi063.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 162 kB/s | 34 kB 00:00 2024-03-04T12:12:16.962 INFO:teuthology.orchestra.run.smithi063.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-04T12:12:16.987 INFO:teuthology.orchestra.run.smithi063.stdout:(88/176): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-04T12:12:17.005 INFO:teuthology.orchestra.run.smithi063.stdout:(89/176): uuid-1.6.2-55.el9.x86_64.rpm 956 kB/s | 56 kB 00:00 2024-03-04T12:12:17.038 INFO:teuthology.orchestra.run.smithi063.stdout:(90/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 87 kB 00:00 2024-03-04T12:12:17.055 INFO:teuthology.orchestra.run.smithi063.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 934 kB/s | 63 kB 00:00 2024-03-04T12:12:17.072 INFO:teuthology.orchestra.run.smithi063.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 659 kB/s | 44 kB 00:00 2024-03-04T12:12:17.084 INFO:teuthology.orchestra.run.smithi086.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 307 kB/s | 43 kB 00:00 2024-03-04T12:12:17.097 INFO:teuthology.orchestra.run.smithi063.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-04T12:12:17.156 INFO:teuthology.orchestra.run.smithi063.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 9.4 MB/s | 802 kB 00:00 2024-03-04T12:12:17.181 INFO:teuthology.orchestra.run.smithi063.stdout:(95/176): jq-1.6-16.el9.x86_64.rpm 1.7 MB/s | 186 kB 00:00 2024-03-04T12:12:17.206 INFO:teuthology.orchestra.run.smithi063.stdout:(96/176): libquadmath-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 194 kB 00:00 2024-03-04T12:12:17.210 INFO:teuthology.orchestra.run.smithi086.stdout:(116/176): python3-cachetools-4.2.4-1.el9.noarc 258 kB/s | 32 kB 00:00 2024-03-04T12:12:17.232 INFO:teuthology.orchestra.run.smithi063.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-04T12:12:17.257 INFO:teuthology.orchestra.run.smithi063.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-04T12:12:17.290 INFO:teuthology.orchestra.run.smithi063.stdout:(99/176): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-04T12:12:17.315 INFO:teuthology.orchestra.run.smithi063.stdout:(100/176): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-04T12:12:17.318 INFO:teuthology.orchestra.run.smithi086.stdout:(117/176): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-04T12:12:17.341 INFO:teuthology.orchestra.run.smithi063.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 1.5 MB/s | 124 kB 00:00 2024-03-04T12:12:17.366 INFO:teuthology.orchestra.run.smithi063.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 2.2 MB/s | 113 kB 00:00 2024-03-04T12:12:17.391 INFO:teuthology.orchestra.run.smithi063.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 2.4 MB/s | 187 kB 00:00 2024-03-04T12:12:17.450 INFO:teuthology.orchestra.run.smithi063.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 269 kB/s | 22 kB 00:00 2024-03-04T12:12:17.492 INFO:teuthology.orchestra.run.smithi063.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.6 MB/s | 245 kB 00:00 2024-03-04T12:12:17.685 INFO:teuthology.orchestra.run.smithi086.stdout:(118/176): python3-cheroot-8.6.0-4.el9.noarch.r 469 kB/s | 172 kB 00:00 2024-03-04T12:12:17.717 INFO:teuthology.orchestra.run.smithi063.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-03-04T12:12:17.742 INFO:teuthology.orchestra.run.smithi063.stdout:(107/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-03-04T12:12:17.784 INFO:teuthology.orchestra.run.smithi063.stdout:(108/176): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-03-04T12:12:17.793 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-04T12:12:17.843 INFO:teuthology.orchestra.run.smithi063.stdout:(109/176): protobuf-compiler-3.14.0-13.el9.x86_ 1.9 MB/s | 863 kB 00:00 2024-03-04T12:12:17.860 INFO:teuthology.orchestra.run.smithi063.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-03-04T12:12:17.918 INFO:teuthology.orchestra.run.smithi063.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-04T12:12:17.994 INFO:teuthology.orchestra.run.smithi063.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-04T12:12:18.011 INFO:teuthology.orchestra.run.smithi063.stdout:(113/176): liboath-2.6.7-2.el9.x86_64.rpm 290 kB/s | 49 kB 00:00 2024-03-04T12:12:18.170 INFO:teuthology.orchestra.run.smithi063.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-04T12:12:18.395 INFO:teuthology.orchestra.run.smithi086.stdout:(119/176): python3-cherrypy-18.6.1-2.el9.noarch 662 kB/s | 358 kB 00:00 2024-03-04T12:12:18.395 INFO:teuthology.orchestra.run.smithi063.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 579 kB/s | 43 kB 00:00 2024-03-04T12:12:18.395 INFO:teuthology.orchestra.run.smithi063.stdout:(116/176): libarrow-9.0.0-10.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-03-04T12:12:18.395 INFO:teuthology.orchestra.run.smithi063.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 385 kB/s | 32 kB 00:00 2024-03-04T12:12:18.396 INFO:teuthology.orchestra.run.smithi063.stdout:(118/176): python3-asyncssh-2.13.2-4.el9.noarch 1.5 MB/s | 548 kB 00:00 2024-03-04T12:12:18.396 INFO:teuthology.orchestra.run.smithi063.stdout:(119/176): python3-certifi-2021.10.8-2.el9.noar 223 kB/s | 15 kB 00:00 2024-03-04T12:12:18.405 INFO:teuthology.orchestra.run.smithi063.stdout:(120/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-04T12:12:18.438 INFO:teuthology.orchestra.run.smithi063.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-03-04T12:12:18.463 INFO:teuthology.orchestra.run.smithi063.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 604 kB/s | 35 kB 00:00 2024-03-04T12:12:18.505 INFO:teuthology.orchestra.run.smithi063.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-04T12:12:18.531 INFO:teuthology.orchestra.run.smithi063.stdout:(124/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2024-03-04T12:12:18.572 INFO:teuthology.orchestra.run.smithi063.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-04T12:12:18.589 INFO:teuthology.orchestra.run.smithi063.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 182 kB/s | 11 kB 00:00 2024-03-04T12:12:18.623 INFO:teuthology.orchestra.run.smithi063.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-03-04T12:12:18.640 INFO:teuthology.orchestra.run.smithi063.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-03-04T12:12:18.661 INFO:teuthology.orchestra.run.smithi086.stdout:(120/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 700 kB/s | 303 kB 00:00 2024-03-04T12:12:18.673 INFO:teuthology.orchestra.run.smithi063.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-03-04T12:12:18.691 INFO:teuthology.orchestra.run.smithi063.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-03-04T12:12:18.766 INFO:teuthology.orchestra.run.smithi063.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 620 kB/s | 46 kB 00:00 2024-03-04T12:12:18.769 INFO:teuthology.orchestra.run.smithi086.stdout:(121/176): python3-colorama-0.4.4-12.el9.noarch 330 kB/s | 35 kB 00:00 2024-03-04T12:12:18.808 INFO:teuthology.orchestra.run.smithi063.stdout:(132/176): python3-grpcio-1.46.7-10.el9.x86_64. 5.9 MB/s | 2.0 MB 00:00 2024-03-04T12:12:18.833 INFO:teuthology.orchestra.run.smithi063.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 1.1 MB/s | 79 kB 00:00 2024-03-04T12:12:18.867 INFO:teuthology.orchestra.run.smithi063.stdout:(134/176): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-03-04T12:12:18.892 INFO:teuthology.orchestra.run.smithi063.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 690 kB/s | 58 kB 00:00 2024-03-04T12:12:18.925 INFO:teuthology.orchestra.run.smithi063.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 281 kB/s | 16 kB 00:00 2024-03-04T12:12:18.984 INFO:teuthology.orchestra.run.smithi063.stdout:(137/176): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-03-04T12:12:19.009 INFO:teuthology.orchestra.run.smithi063.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-03-04T12:12:19.035 INFO:teuthology.orchestra.run.smithi063.stdout:(139/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 631 kB/s | 90 kB 00:00 2024-03-04T12:12:19.060 INFO:teuthology.orchestra.run.smithi063.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-03-04T12:12:19.085 INFO:teuthology.orchestra.run.smithi063.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-04T12:12:19.110 INFO:teuthology.orchestra.run.smithi063.stdout:(142/176): python3-rsa-4.9-2.el9.noarch.rpm 786 kB/s | 59 kB 00:00 2024-03-04T12:12:19.136 INFO:teuthology.orchestra.run.smithi063.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-03-04T12:12:19.136 INFO:teuthology.orchestra.run.smithi086.stdout:(122/176): python3-google-auth-2.28.0-1.el9.noa 553 kB/s | 203 kB 00:00 2024-03-04T12:12:19.161 INFO:teuthology.orchestra.run.smithi063.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-03-04T12:12:19.186 INFO:teuthology.orchestra.run.smithi063.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-04T12:12:19.211 INFO:teuthology.orchestra.run.smithi063.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-04T12:12:19.237 INFO:teuthology.orchestra.run.smithi063.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-03-04T12:12:19.262 INFO:teuthology.orchestra.run.smithi063.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-04T12:12:19.321 INFO:teuthology.orchestra.run.smithi063.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-04T12:12:19.413 INFO:teuthology.orchestra.run.smithi063.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-04T12:12:19.471 INFO:teuthology.orchestra.run.smithi063.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.1 MB/s | 68 kB 00:00 2024-03-04T12:12:19.580 INFO:teuthology.orchestra.run.smithi063.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 21 MB/s | 2.3 MB 00:00 2024-03-04T12:12:19.639 INFO:teuthology.orchestra.run.smithi063.stdout:(153/176): librbd1-19.0.0-1845.g717ce3cc.el9.x8 9.9 MB/s | 3.1 MB 00:00 2024-03-04T12:12:19.697 INFO:teuthology.orchestra.run.smithi063.stdout:(154/176): NetworkManager-libnm-1.46.0-1.el9.x8 15 MB/s | 1.8 MB 00:00 2024-03-04T12:12:19.714 INFO:teuthology.orchestra.run.smithi063.stdout:(155/176): NetworkManager-team-1.46.0-1.el9.x86 572 kB/s | 41 kB 00:00 2024-03-04T12:12:19.756 INFO:teuthology.orchestra.run.smithi063.stdout:(156/176): NetworkManager-tui-1.46.0-1.el9.x86_ 4.1 MB/s | 247 kB 00:00 2024-03-04T12:12:19.781 INFO:teuthology.orchestra.run.smithi063.stdout:(157/176): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 69 kB 00:00 2024-03-04T12:12:19.798 INFO:teuthology.orchestra.run.smithi063.stdout:(158/176): elfutils-debuginfod-client-0.190-2.e 847 kB/s | 35 kB 00:00 2024-03-04T12:12:19.832 INFO:teuthology.orchestra.run.smithi063.stdout:(159/176): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 191 kB 00:00 2024-03-04T12:12:19.865 INFO:teuthology.orchestra.run.smithi063.stdout:(160/176): elfutils-libs-0.190-2.el9.x86_64.rpm 3.7 MB/s | 254 kB 00:00 2024-03-04T12:12:19.899 INFO:teuthology.orchestra.run.smithi063.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 12 MB/s | 802 kB 00:00 2024-03-04T12:12:19.924 INFO:teuthology.orchestra.run.smithi063.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 7.4 MB/s | 433 kB 00:00 2024-03-04T12:12:19.941 INFO:teuthology.orchestra.run.smithi063.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 536 kB/s | 22 kB 00:00 2024-03-04T12:12:19.966 INFO:teuthology.orchestra.run.smithi063.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 107 kB 00:00 2024-03-04T12:12:19.992 INFO:teuthology.orchestra.run.smithi063.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 154 kB 00:00 2024-03-04T12:12:20.017 INFO:teuthology.orchestra.run.smithi063.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 135 kB 00:00 2024-03-04T12:12:20.034 INFO:teuthology.orchestra.run.smithi063.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 63 kB 00:00 2024-03-04T12:12:20.051 INFO:teuthology.orchestra.run.smithi063.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 829 kB/s | 28 kB 00:00 2024-03-04T12:12:20.126 INFO:teuthology.orchestra.run.smithi063.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 13 MB/s | 1.2 MB 00:00 2024-03-04T12:12:20.235 INFO:teuthology.orchestra.run.smithi063.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-03-04T12:12:20.319 INFO:teuthology.orchestra.run.smithi063.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.2 MB 00:00 2024-03-04T12:12:20.344 INFO:teuthology.orchestra.run.smithi063.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 4.0 MB/s | 438 kB 00:00 2024-03-04T12:12:20.386 INFO:teuthology.orchestra.run.smithi063.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 455 kB/s | 30 kB 00:00 2024-03-04T12:12:20.440 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T12:12:20.441 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-04T12:12:20.444 INFO:teuthology.orchestra.run.smithi063.stdout:(174/176): librados2-19.0.0-1845.g717ce3cc.el9. 2.9 MB/s | 3.4 MB 00:01 2024-03-04T12:12:20.456 INFO:teuthology.orchestra.run.smithi060.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-04T12:12:20.457 INFO:teuthology.orchestra.run.smithi060.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-04T12:12:20.458 INFO:teuthology.orchestra.run.smithi060.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-04T12:12:20.459 INFO:teuthology.orchestra.run.smithi060.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-04T12:12:20.461 INFO:teuthology.orchestra.run.smithi060.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-04T12:12:20.464 INFO:teuthology.orchestra.run.smithi060.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-04T12:12:20.465 INFO:teuthology.orchestra.run.smithi060.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-04T12:12:20.469 INFO:teuthology.orchestra.run.smithi063.stdout:(175/176): xfsdump-3.1.12-4.el9.x86_64.rpm 2.5 MB/s | 321 kB 00:00 2024-03-04T12:12:20.503 INFO:teuthology.orchestra.run.smithi063.stdout:(176/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.0 MB/s | 1.1 MB 00:00 2024-03-04T12:12:20.508 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-03-04T12:12:20.508 INFO:teuthology.orchestra.run.smithi063.stdout:Total 15 MB/s | 223 MB 00:15 2024-03-04T12:12:20.552 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-04T12:12:20.559 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-03-04T12:12:20.579 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2024-03-04T12:12:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-04T12:12:20.579 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2024-03-04T12:12:20.579 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-04T12:12:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-04T12:12:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 6.6 k 2024-03-04T12:12:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 5.3 M 2024-03-04T12:12:20.579 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 888 k 2024-03-04T12:12:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 145 k 2024-03-04T12:12:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 1.5 M 2024-03-04T12:12:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 138 k 2024-03-04T12:12:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 1.8 M 2024-03-04T12:12:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 7.4 M 2024-03-04T12:12:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 49 k 2024-03-04T12:12:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 9.8 M 2024-03-04T12:12:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 47 M 2024-03-04T12:12:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 264 k 2024-03-04T12:12:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 749 k 2024-03-04T12:12:20.580 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-mirror x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 220 k 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-shell x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 27 k 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-top noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 18 k 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 31 k 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 745 k 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 127 k 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-04T12:12:20.581 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-04T12:12:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-04T12:12:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 162 k 2024-03-04T12:12:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 320 k 2024-03-04T12:12:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 298 k 2024-03-04T12:12:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 99 k 2024-03-04T12:12:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 85 k 2024-03-04T12:12:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.1 M 2024-03-04T12:12:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 170 k 2024-03-04T12:12:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-04T12:12:20.582 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2024-03-04T12:12:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-04T12:12:20.582 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-04T12:12:20.583 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.4 M 2024-03-04T12:12:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 3.1 M 2024-03-04T12:12:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-04T12:12:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-04T12:12:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-04T12:12:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-04T12:12:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-04T12:12:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-04T12:12:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-04T12:12:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-04T12:12:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-04T12:12:20.584 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-04T12:12:20.584 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 20 M 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 31 k 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 2.1 M 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 248 k 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 4.7 M 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 17 M 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1845.g717ce3cc.el9 ceph-noarch 16 k 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 25 k 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T12:12:20.585 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T12:12:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T12:12:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T12:12:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T12:12:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T12:12:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T12:12:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T12:12:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 163 k 2024-03-04T12:12:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T12:12:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T12:12:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T12:12:20.586 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 502 k 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 5.1 M 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T12:12:20.587 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T12:12:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-04T12:12:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T12:12:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T12:12:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T12:12:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-04T12:12:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T12:12:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T12:12:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T12:12:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 45 k 2024-03-04T12:12:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:19.0.0-1845.g717ce3cc.el9 ceph 131 k 2024-03-04T12:12:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T12:12:20.588 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T12:12:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T12:12:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T12:12:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-04T12:12:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-04T12:12:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T12:12:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T12:12:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T12:12:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T12:12:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T12:12:20.589 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T12:12:20.590 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T12:12:20.591 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T12:12:20.592 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T12:12:20.592 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T12:12:20.592 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T12:12:20.592 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T12:12:20.592 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T12:12:20.592 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T12:12:20.592 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T12:12:20.592 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T12:12:20.592 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T12:12:20.592 INFO:teuthology.orchestra.run.smithi060.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-04T12:12:20.592 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T12:12:20.592 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout:Installing weak dependencies: 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:12:20.593 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-04T12:12:20.594 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2024-03-04T12:12:20.594 INFO:teuthology.orchestra.run.smithi060.stdout:Install 150 Packages 2024-03-04T12:12:20.594 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 26 Packages 2024-03-04T12:12:20.594 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:12:20.594 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 223 M 2024-03-04T12:12:20.594 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-04T12:12:20.860 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-03-04T12:12:20.860 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-03-04T12:12:21.244 INFO:teuthology.orchestra.run.smithi060.stdout:(1/176): ceph-19.0.0-1845.g717ce3cc.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-03-04T12:12:21.470 INFO:teuthology.orchestra.run.smithi086.stdout:(123/176): libarrow-9.0.0-10.el9.x86_64.rpm 649 kB/s | 4.4 MB 00:06 2024-03-04T12:12:21.711 INFO:teuthology.orchestra.run.smithi060.stdout:(2/176): ceph-base-19.0.0-1845.g717ce3cc.el9.x8 6.6 MB/s | 5.3 MB 00:00 2024-03-04T12:12:21.762 INFO:teuthology.orchestra.run.smithi086.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 496 kB/s | 144 kB 00:00 2024-03-04T12:12:21.769 INFO:teuthology.orchestra.run.smithi060.stdout:(3/176): ceph-immutable-object-cache-19.0.0-184 2.4 MB/s | 145 kB 00:00 2024-03-04T12:12:21.895 INFO:teuthology.orchestra.run.smithi086.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-03-04T12:12:21.987 INFO:teuthology.orchestra.run.smithi060.stdout:(4/176): ceph-mds-19.0.0-1845.g717ce3cc.el9.x86 9.8 MB/s | 2.1 MB 00:00 2024-03-04T12:12:22.012 INFO:teuthology.orchestra.run.smithi086.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 152 kB/s | 18 kB 00:00 2024-03-04T12:12:22.129 INFO:teuthology.orchestra.run.smithi086.stdout:(127/176): python3-jaraco-collections-3.0.0-8.e 198 kB/s | 23 kB 00:00 2024-03-04T12:12:22.170 INFO:teuthology.orchestra.run.smithi060.stdout:(5/176): ceph-mgr-19.0.0-1845.g717ce3cc.el9.x86 8.1 MB/s | 1.5 MB 00:00 2024-03-04T12:12:22.212 INFO:teuthology.orchestra.run.smithi060.stdout:(6/176): ceph-fuse-19.0.0-1845.g717ce3cc.el9.x8 917 kB/s | 888 kB 00:00 2024-03-04T12:12:22.297 INFO:teuthology.orchestra.run.smithi086.stdout:(128/176): python3-jaraco-functools-3.5.0-2.el9 116 kB/s | 19 kB 00:00 2024-03-04T12:12:22.347 INFO:teuthology.orchestra.run.smithi086.stdout:(129/176): python3-grpcio-1.46.7-10.el9.x86_64. 650 kB/s | 2.0 MB 00:03 2024-03-04T12:12:22.405 INFO:teuthology.orchestra.run.smithi086.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-03-04T12:12:22.514 INFO:teuthology.orchestra.run.smithi086.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 427 kB/s | 46 kB 00:00 2024-03-04T12:12:22.620 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-03-04T12:12:22.620 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-03-04T12:12:22.656 INFO:teuthology.orchestra.run.smithi086.stdout:(132/176): python3-more-itertools-8.12.0-2.el9. 557 kB/s | 79 kB 00:00 2024-03-04T12:12:22.756 INFO:teuthology.orchestra.run.smithi086.stdout:(133/176): python3-natsort-7.1.1-5.el9.noarch.r 579 kB/s | 58 kB 00:00 2024-03-04T12:12:22.788 INFO:teuthology.orchestra.run.smithi060.stdout:(7/176): ceph-common-19.0.0-1845.g717ce3cc.el9. 11 MB/s | 20 MB 00:01 2024-03-04T12:12:22.854 INFO:teuthology.orchestra.run.smithi060.stdout:(8/176): ceph-mon-19.0.0-1845.g717ce3cc.el9.x86 6.9 MB/s | 4.7 MB 00:00 2024-03-04T12:12:22.921 INFO:teuthology.orchestra.run.smithi060.stdout:(9/176): ceph-selinux-19.0.0-1845.g717ce3cc.el9 379 kB/s | 25 kB 00:00 2024-03-04T12:12:23.190 INFO:teuthology.orchestra.run.smithi086.stdout:(134/176): python3-pecan-1.4.2-3.el9.noarch.rpm 628 kB/s | 272 kB 00:00 2024-03-04T12:12:23.307 INFO:teuthology.orchestra.run.smithi086.stdout:(135/176): python3-portend-3.1.0-2.el9.noarch.r 141 kB/s | 16 kB 00:00 2024-03-04T12:12:23.371 INFO:teuthology.orchestra.run.smithi060.stdout:(10/176): ceph-radosgw-19.0.0-1845.g717ce3cc.el 17 MB/s | 9.8 MB 00:00 2024-03-04T12:12:23.438 INFO:teuthology.orchestra.run.smithi060.stdout:(11/176): cephfs-mirror-19.0.0-1845.g717ce3cc.e 3.2 MB/s | 220 kB 00:00 2024-03-04T12:12:23.480 INFO:teuthology.orchestra.run.smithi060.stdout:(12/176): cephfs-shell-19.0.0-1845.g717ce3cc.el 654 kB/s | 27 kB 00:00 2024-03-04T12:12:23.490 INFO:teuthology.orchestra.run.smithi086.stdout:(136/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 491 kB/s | 90 kB 00:00 2024-03-04T12:12:23.530 INFO:teuthology.orchestra.run.smithi060.stdout:(13/176): libcephfs-devel-19.0.0-1845.g717ce3cc 624 kB/s | 31 kB 00:00 2024-03-04T12:12:23.591 INFO:teuthology.orchestra.run.smithi086.stdout:(137/176): python3-pyperclip-1.8.2-2.el9.noarch 283 kB/s | 28 kB 00:00 2024-03-04T12:12:23.643 INFO:teuthology.orchestra.run.smithi060.stdout:(14/176): libcephfs2-19.0.0-1845.g717ce3cc.el9. 6.5 MB/s | 745 kB 00:00 2024-03-04T12:12:23.716 INFO:teuthology.orchestra.run.smithi086.stdout:(138/176): python3-kubernetes-26.1.0-2.el9.noar 764 kB/s | 1.0 MB 00:01 2024-03-04T12:12:23.723 INFO:teuthology.orchestra.run.smithi060.stdout:(15/176): libcephsqlite-19.0.0-1845.g717ce3cc.e 2.0 MB/s | 163 kB 00:00 2024-03-04T12:12:23.741 INFO:teuthology.orchestra.run.smithi086.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 205 kB/s | 31 kB 00:00 2024-03-04T12:12:23.790 INFO:teuthology.orchestra.run.smithi060.stdout:(16/176): librados-devel-19.0.0-1845.g717ce3cc. 1.9 MB/s | 127 kB 00:00 2024-03-04T12:12:23.841 INFO:teuthology.orchestra.run.smithi086.stdout:(140/176): python3-rsa-4.9-2.el9.noarch.rpm 593 kB/s | 59 kB 00:00 2024-03-04T12:12:23.873 INFO:teuthology.orchestra.run.smithi060.stdout:(17/176): libradosstriper1-19.0.0-1845.g717ce3c 5.9 MB/s | 502 kB 00:00 2024-03-04T12:12:23.942 INFO:teuthology.orchestra.run.smithi086.stdout:(141/176): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-03-04T12:12:24.042 INFO:teuthology.orchestra.run.smithi086.stdout:(142/176): python3-typing-extensions-4.4.0-2.el 510 kB/s | 51 kB 00:00 2024-03-04T12:12:24.067 INFO:teuthology.orchestra.run.smithi086.stdout:(143/176): python3-routes-2.5.1-5.el9.noarch.rp 579 kB/s | 188 kB 00:00 2024-03-04T12:12:24.251 INFO:teuthology.orchestra.run.smithi086.stdout:(144/176): python3-websocket-client-1.2.3-2.el9 488 kB/s | 90 kB 00:00 2024-03-04T12:12:24.299 INFO:teuthology.orchestra.run.smithi060.stdout:(18/176): librgw2-19.0.0-1845.g717ce3cc.el9.x86 12 MB/s | 5.1 MB 00:00 2024-03-04T12:12:24.351 INFO:teuthology.orchestra.run.smithi060.stdout:(19/176): python3-ceph-argparse-19.0.0-1845.g71 867 kB/s | 45 kB 00:00 2024-03-04T12:12:24.401 INFO:teuthology.orchestra.run.smithi086.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 640 kB/s | 230 kB 00:00 2024-03-04T12:12:24.416 INFO:teuthology.orchestra.run.smithi060.stdout:(20/176): python3-ceph-common-19.0.0-1845.g717c 2.0 MB/s | 131 kB 00:00 2024-03-04T12:12:24.483 INFO:teuthology.orchestra.run.smithi060.stdout:(21/176): python3-cephfs-19.0.0-1845.g717ce3cc. 2.4 MB/s | 162 kB 00:00 2024-03-04T12:12:24.518 INFO:teuthology.orchestra.run.smithi086.stdout:(146/176): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2024-03-04T12:12:24.558 INFO:teuthology.orchestra.run.smithi060.stdout:(22/176): python3-rados-19.0.0-1845.g717ce3cc.e 4.2 MB/s | 320 kB 00:00 2024-03-04T12:12:24.642 INFO:teuthology.orchestra.run.smithi060.stdout:(23/176): python3-rbd-19.0.0-1845.g717ce3cc.el9 3.5 MB/s | 298 kB 00:00 2024-03-04T12:12:24.643 INFO:teuthology.orchestra.run.smithi086.stdout:(147/176): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-04T12:12:24.709 INFO:teuthology.orchestra.run.smithi060.stdout:(24/176): python3-rgw-19.0.0-1845.g717ce3cc.el9 1.5 MB/s | 99 kB 00:00 2024-03-04T12:12:24.776 INFO:teuthology.orchestra.run.smithi060.stdout:(25/176): rbd-fuse-19.0.0-1845.g717ce3cc.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-04T12:12:24.852 INFO:teuthology.orchestra.run.smithi086.stdout:(148/176): python3-werkzeug-2.0.3-3.el9.1.noarc 711 kB/s | 427 kB 00:00 2024-03-04T12:12:24.927 INFO:teuthology.orchestra.run.smithi086.stdout:(149/176): thrift-0.15.0-2.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-03-04T12:12:25.043 INFO:teuthology.orchestra.run.smithi060.stdout:(26/176): rbd-mirror-19.0.0-1845.g717ce3cc.el9. 12 MB/s | 3.1 MB 00:00 2024-03-04T12:12:25.110 INFO:teuthology.orchestra.run.smithi060.stdout:(27/176): rbd-nbd-19.0.0-1845.g717ce3cc.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-04T12:12:25.185 INFO:teuthology.orchestra.run.smithi060.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1845.g 419 kB/s | 31 kB 00:00 2024-03-04T12:12:25.277 INFO:teuthology.orchestra.run.smithi060.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1845.g717ce3c 1.5 MB/s | 138 kB 00:00 2024-03-04T12:12:25.602 INFO:teuthology.orchestra.run.smithi060.stdout:(30/176): ceph-mgr-dashboard-19.0.0-1845.g717ce 5.5 MB/s | 1.8 MB 00:00 2024-03-04T12:12:25.744 INFO:teuthology.orchestra.run.smithi086.stdout:(150/176): librados2-19.0.0-1845.g717ce3cc.el9. 3.8 MB/s | 3.4 MB 00:00 2024-03-04T12:12:25.786 INFO:teuthology.orchestra.run.smithi086.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-03-04T12:12:25.886 INFO:teuthology.orchestra.run.smithi086.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 23 MB/s | 2.3 MB 00:00 2024-03-04T12:12:25.937 INFO:teuthology.orchestra.run.smithi086.stdout:(153/176): librbd1-19.0.0-1845.g717ce3cc.el9.x8 3.1 MB/s | 3.1 MB 00:01 2024-03-04T12:12:25.987 INFO:teuthology.orchestra.run.smithi086.stdout:(154/176): NetworkManager-libnm-1.46.0-1.el9.x8 18 MB/s | 1.8 MB 00:00 2024-03-04T12:12:26.004 INFO:teuthology.orchestra.run.smithi086.stdout:(155/176): NetworkManager-team-1.46.0-1.el9.x86 642 kB/s | 41 kB 00:00 2024-03-04T12:12:26.029 INFO:teuthology.orchestra.run.smithi086.stdout:(156/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-04T12:12:26.054 INFO:teuthology.orchestra.run.smithi086.stdout:(157/176): acl-2.3.1-4.el9.x86_64.rpm 1.3 MB/s | 69 kB 00:00 2024-03-04T12:12:26.080 INFO:teuthology.orchestra.run.smithi086.stdout:(158/176): elfutils-debuginfod-client-0.190-2.e 704 kB/s | 35 kB 00:00 2024-03-04T12:12:26.105 INFO:teuthology.orchestra.run.smithi086.stdout:(159/176): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 191 kB 00:00 2024-03-04T12:12:26.130 INFO:teuthology.orchestra.run.smithi086.stdout:(160/176): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 254 kB 00:00 2024-03-04T12:12:26.164 INFO:teuthology.orchestra.run.smithi086.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 14 MB/s | 802 kB 00:00 2024-03-04T12:12:26.189 INFO:teuthology.orchestra.run.smithi086.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 7.4 MB/s | 433 kB 00:00 2024-03-04T12:12:26.215 INFO:teuthology.orchestra.run.smithi086.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 444 kB/s | 22 kB 00:00 2024-03-04T12:12:26.240 INFO:teuthology.orchestra.run.smithi086.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-03-04T12:12:26.265 INFO:teuthology.orchestra.run.smithi086.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 154 kB 00:00 2024-03-04T12:12:26.290 INFO:teuthology.orchestra.run.smithi086.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 135 kB 00:00 2024-03-04T12:12:26.316 INFO:teuthology.orchestra.run.smithi086.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 63 kB 00:00 2024-03-04T12:12:26.341 INFO:teuthology.orchestra.run.smithi086.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 553 kB/s | 28 kB 00:00 2024-03-04T12:12:26.378 INFO:teuthology.orchestra.run.smithi060.stdout:(31/176): ceph-test-19.0.0-1845.g717ce3cc.el9.x 14 MB/s | 47 MB 00:03 2024-03-04T12:12:26.383 INFO:teuthology.orchestra.run.smithi086.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 17 MB/s | 1.2 MB 00:00 2024-03-04T12:12:26.433 INFO:teuthology.orchestra.run.smithi086.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 24 MB/s | 2.2 MB 00:00 2024-03-04T12:12:26.478 INFO:teuthology.orchestra.run.smithi060.stdout:(32/176): ceph-mgr-diskprediction-local-19.0.0- 8.5 MB/s | 7.4 MB 00:00 2024-03-04T12:12:26.483 INFO:teuthology.orchestra.run.smithi086.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 23 MB/s | 2.2 MB 00:00 2024-03-04T12:12:26.503 INFO:teuthology.orchestra.run.smithi060.stdout:(33/176): ceph-mgr-modules-core-19.0.0-1845.g71 1.9 MB/s | 248 kB 00:00 2024-03-04T12:12:26.509 INFO:teuthology.orchestra.run.smithi086.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 5.9 MB/s | 438 kB 00:00 2024-03-04T12:12:26.528 INFO:teuthology.orchestra.run.smithi060.stdout:(34/176): ceph-mgr-rook-19.0.0-1845.g717ce3cc.e 990 kB/s | 49 kB 00:00 2024-03-04T12:12:26.534 INFO:teuthology.orchestra.run.smithi086.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 609 kB/s | 30 kB 00:00 2024-03-04T12:12:26.554 INFO:teuthology.orchestra.run.smithi060.stdout:(35/176): ceph-prometheus-alerts-19.0.0-1845.g7 327 kB/s | 16 kB 00:00 2024-03-04T12:12:26.559 INFO:teuthology.orchestra.run.smithi086.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-04T12:12:26.579 INFO:teuthology.orchestra.run.smithi060.stdout:(36/176): ceph-volume-19.0.0-1845.g717ce3cc.el9 5.1 MB/s | 264 kB 00:00 2024-03-04T12:12:26.593 INFO:teuthology.orchestra.run.smithi086.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-04T12:12:26.621 INFO:teuthology.orchestra.run.smithi060.stdout:(37/176): cephfs-top-19.0.0-1845.g717ce3cc.el9. 439 kB/s | 18 kB 00:00 2024-03-04T12:12:26.654 INFO:teuthology.orchestra.run.smithi060.stdout:(38/176): cephadm-19.0.0-1845.g717ce3cc.el9.noa 7.3 MB/s | 749 kB 00:00 2024-03-04T12:12:26.838 INFO:teuthology.orchestra.run.smithi060.stdout:(39/176): automake-1.16.2-8.el9.noarch.rpm 3.0 MB/s | 662 kB 00:00 2024-03-04T12:12:26.880 INFO:teuthology.orchestra.run.smithi060.stdout:(40/176): bison-3.7.4-5.el9.x86_64.rpm 4.0 MB/s | 920 kB 00:00 2024-03-04T12:12:26.905 INFO:teuthology.orchestra.run.smithi060.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-04T12:12:26.930 INFO:teuthology.orchestra.run.smithi060.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 407 kB/s | 20 kB 00:00 2024-03-04T12:12:26.965 INFO:teuthology.orchestra.run.smithi060.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 5.1 MB/s | 307 kB 00:00 2024-03-04T12:12:26.990 INFO:teuthology.orchestra.run.smithi060.stdout:(44/176): flexiblas-3.0.4-8.el9.x86_64.rpm 539 kB/s | 32 kB 00:00 2024-03-04T12:12:27.073 INFO:teuthology.orchestra.run.smithi060.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 208 kB/s | 17 kB 00:00 2024-03-04T12:12:27.124 INFO:teuthology.orchestra.run.smithi060.stdout:(46/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-03-04T12:12:27.149 INFO:teuthology.orchestra.run.smithi060.stdout:(47/176): libaio-devel-0.3.111-13.el9.x86_64.rp 134 kB/s | 10 kB 00:00 2024-03-04T12:12:27.174 INFO:teuthology.orchestra.run.smithi060.stdout:(48/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-04T12:12:27.199 INFO:teuthology.orchestra.run.smithi060.stdout:(49/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-04T12:12:27.225 INFO:teuthology.orchestra.run.smithi060.stdout:(50/176): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-04T12:12:27.250 INFO:teuthology.orchestra.run.smithi060.stdout:(51/176): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2024-03-04T12:12:27.267 INFO:teuthology.orchestra.run.smithi060.stdout:(52/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 597 kB/s | 24 kB 00:00 2024-03-04T12:12:27.284 INFO:teuthology.orchestra.run.smithi060.stdout:(53/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-04T12:12:27.318 INFO:teuthology.orchestra.run.smithi060.stdout:(54/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-04T12:12:27.343 INFO:teuthology.orchestra.run.smithi060.stdout:(55/176): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-04T12:12:27.368 INFO:teuthology.orchestra.run.smithi060.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-03-04T12:12:27.502 INFO:teuthology.orchestra.run.smithi060.stdout:(57/176): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-04T12:12:27.560 INFO:teuthology.orchestra.run.smithi060.stdout:(58/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-03-04T12:12:27.602 INFO:teuthology.orchestra.run.smithi060.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 16 MB/s | 1.5 MB 00:00 2024-03-04T12:12:27.644 INFO:teuthology.orchestra.run.smithi060.stdout:(60/176): postgresql-contrib-13.11-1.el9.x86_64 9.8 MB/s | 811 kB 00:00 2024-03-04T12:12:27.669 INFO:teuthology.orchestra.run.smithi060.stdout:(61/176): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-04T12:12:27.895 INFO:teuthology.orchestra.run.smithi060.stdout:(62/176): postgresql-server-13.11-1.el9.x86_64. 23 MB/s | 5.7 MB 00:00 2024-03-04T12:12:27.928 INFO:teuthology.orchestra.run.smithi060.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2024-03-04T12:12:27.954 INFO:teuthology.orchestra.run.smithi060.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-04T12:12:28.171 INFO:teuthology.orchestra.run.smithi060.stdout:(65/176): python3-babel-2.9.1-2.el9.noarch.rpm 24 MB/s | 5.8 MB 00:00 2024-03-04T12:12:28.196 INFO:teuthology.orchestra.run.smithi060.stdout:(66/176): python3-jinja2-2.11.3-5.el9.noarch.rp 944 kB/s | 227 kB 00:00 2024-03-04T12:12:28.221 INFO:teuthology.orchestra.run.smithi060.stdout:(67/176): python3-jsonpatch-1.21-16.el9.noarch. 493 kB/s | 24 kB 00:00 2024-03-04T12:12:28.246 INFO:teuthology.orchestra.run.smithi060.stdout:(68/176): python3-jsonpointer-2.0-4.el9.noarch. 360 kB/s | 18 kB 00:00 2024-03-04T12:12:28.272 INFO:teuthology.orchestra.run.smithi060.stdout:(69/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-04T12:12:28.297 INFO:teuthology.orchestra.run.smithi060.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 634 kB/s | 32 kB 00:00 2024-03-04T12:12:28.472 INFO:teuthology.orchestra.run.smithi060.stdout:(71/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2024-03-04T12:12:28.498 INFO:teuthology.orchestra.run.smithi060.stdout:(72/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-04T12:12:28.523 INFO:teuthology.orchestra.run.smithi060.stdout:(73/176): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-04T12:12:28.548 INFO:teuthology.orchestra.run.smithi060.stdout:(74/176): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-04T12:12:28.574 INFO:teuthology.orchestra.run.smithi060.stdout:(75/176): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-04T12:12:28.607 INFO:teuthology.orchestra.run.smithi060.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-04T12:12:28.641 INFO:teuthology.orchestra.run.smithi060.stdout:(77/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-04T12:12:28.666 INFO:teuthology.orchestra.run.smithi060.stdout:(78/176): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-04T12:12:28.692 INFO:teuthology.orchestra.run.smithi060.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 952 kB/s | 47 kB 00:00 2024-03-04T12:12:28.717 INFO:teuthology.orchestra.run.smithi060.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 853 kB/s | 43 kB 00:00 2024-03-04T12:12:29.209 INFO:teuthology.orchestra.run.smithi060.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 79 kB/s | 39 kB 00:00 2024-03-04T12:12:29.459 INFO:teuthology.orchestra.run.smithi060.stdout:(82/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-03-04T12:12:29.484 INFO:teuthology.orchestra.run.smithi060.stdout:(83/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 150 kB/s | 41 kB 00:00 2024-03-04T12:12:29.509 INFO:teuthology.orchestra.run.smithi060.stdout:(84/176): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 215 kB 00:00 2024-03-04T12:12:29.535 INFO:teuthology.orchestra.run.smithi060.stdout:(85/176): qatlib-service-23.11.0-1.el9.x86_64.r 677 kB/s | 34 kB 00:00 2024-03-04T12:12:29.560 INFO:teuthology.orchestra.run.smithi060.stdout:(86/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 870 kB/s | 43 kB 00:00 2024-03-04T12:12:29.594 INFO:teuthology.orchestra.run.smithi060.stdout:(87/176): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-04T12:12:29.611 INFO:teuthology.orchestra.run.smithi060.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-04T12:12:29.644 INFO:teuthology.orchestra.run.smithi060.stdout:(89/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-03-04T12:12:29.661 INFO:teuthology.orchestra.run.smithi060.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-04T12:12:29.687 INFO:teuthology.orchestra.run.smithi060.stdout:(91/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-04T12:12:29.712 INFO:teuthology.orchestra.run.smithi060.stdout:(92/176): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-04T12:12:29.746 INFO:teuthology.orchestra.run.smithi060.stdout:(93/176): jq-1.6-16.el9.x86_64.rpm 3.1 MB/s | 186 kB 00:00 2024-03-04T12:12:29.796 INFO:teuthology.orchestra.run.smithi060.stdout:(94/176): libgfortran-11.4.1-3.el9.x86_64.rpm 9.4 MB/s | 802 kB 00:00 2024-03-04T12:12:29.821 INFO:teuthology.orchestra.run.smithi060.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 194 kB 00:00 2024-03-04T12:12:29.846 INFO:teuthology.orchestra.run.smithi060.stdout:(96/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-04T12:12:29.888 INFO:teuthology.orchestra.run.smithi060.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-04T12:12:29.903 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-03-04T12:12:29.955 INFO:teuthology.orchestra.run.smithi060.stdout:(98/176): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-04T12:12:29.980 INFO:teuthology.orchestra.run.smithi060.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-04T12:12:30.006 INFO:teuthology.orchestra.run.smithi060.stdout:(100/176): python3-pycparser-2.20-6.el9.noarch. 2.4 MB/s | 124 kB 00:00 2024-03-04T12:12:30.016 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-04T12:12:30.048 INFO:teuthology.orchestra.run.smithi060.stdout:(101/176): python3-requests-2.25.1-8.el9.noarch 1.7 MB/s | 113 kB 00:00 2024-03-04T12:12:30.050 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-04T12:12:30.073 INFO:teuthology.orchestra.run.smithi060.stdout:(102/176): python3-urllib3-1.26.5-5.el9.noarch. 2.7 MB/s | 187 kB 00:00 2024-03-04T12:12:30.080 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T12:12:30.106 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T12:12:30.159 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-04T12:12:30.198 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-04T12:12:30.215 INFO:teuthology.orchestra.run.smithi060.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-03-04T12:12:30.252 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-04T12:12:30.294 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-04T12:12:30.319 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-04T12:12:30.361 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-04T12:12:30.432 INFO:teuthology.orchestra.run.smithi060.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 640 kB/s | 245 kB 00:00 2024-03-04T12:12:30.698 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-04T12:12:30.745 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T12:12:30.860 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T12:12:30.874 INFO:teuthology.orchestra.run.smithi060.stdout:(105/176): protobuf-compiler-3.14.0-13.el9.x86_ 1.3 MB/s | 863 kB 00:00 2024-03-04T12:12:30.904 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T12:12:30.924 INFO:teuthology.orchestra.run.smithi060.stdout:(106/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-04T12:12:30.979 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T12:12:31.008 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 13/202 2024-03-04T12:12:31.025 INFO:teuthology.orchestra.run.smithi060.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 266 kB/s | 19 kB 00:00 2024-03-04T12:12:31.164 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-04T12:12:31.222 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-04T12:12:31.258 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T12:12:31.300 INFO:teuthology.orchestra.run.smithi060.stdout:(108/176): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-03-04T12:12:31.325 INFO:teuthology.orchestra.run.smithi060.stdout:(109/176): gperftools-libs-2.9.1-2.el9.x86_64.r 685 kB/s | 309 kB 00:00 2024-03-04T12:12:31.355 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T12:12:31.376 INFO:teuthology.orchestra.run.smithi060.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-03-04T12:12:31.393 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 17/202 2024-03-04T12:12:31.401 INFO:teuthology.orchestra.run.smithi060.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 645 kB/s | 49 kB 00:00 2024-03-04T12:12:31.443 INFO:teuthology.orchestra.run.smithi060.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-04T12:12:31.456 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 17/202 2024-03-04T12:12:31.493 INFO:teuthology.orchestra.run.smithi060.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-04T12:12:31.505 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-04T12:12:31.527 INFO:teuthology.orchestra.run.smithi060.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-04T12:12:31.527 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T12:12:31.544 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T12:12:31.602 INFO:teuthology.orchestra.run.smithi060.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 578 kB/s | 43 kB 00:00 2024-03-04T12:12:31.635 INFO:teuthology.orchestra.run.smithi060.stdout:(116/176): python3-asyncssh-2.13.2-4.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-03-04T12:12:31.678 INFO:teuthology.orchestra.run.smithi060.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-03-04T12:12:31.711 INFO:teuthology.orchestra.run.smithi060.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-03-04T12:12:31.745 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T12:12:31.755 INFO:teuthology.orchestra.run.smithi060.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-04T12:12:31.775 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T12:12:31.795 INFO:teuthology.orchestra.run.smithi060.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-04T12:12:31.823 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-04T12:12:31.853 INFO:teuthology.orchestra.run.smithi060.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-03-04T12:12:31.871 INFO:teuthology.orchestra.run.smithi060.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 469 kB/s | 35 kB 00:00 2024-03-04T12:12:31.929 INFO:teuthology.orchestra.run.smithi060.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-04T12:12:31.932 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-04T12:12:32.005 INFO:teuthology.orchestra.run.smithi060.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-04T12:12:32.034 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T12:12:32.047 INFO:teuthology.orchestra.run.smithi060.stdout:(125/176): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-03-04T12:12:32.080 INFO:teuthology.orchestra.run.smithi060.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-04T12:12:32.094 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T12:12:32.122 INFO:teuthology.orchestra.run.smithi060.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-03-04T12:12:32.147 INFO:teuthology.orchestra.run.smithi060.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 347 kB/s | 23 kB 00:00 2024-03-04T12:12:32.172 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T12:12:32.189 INFO:teuthology.orchestra.run.smithi060.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 289 kB/s | 19 kB 00:00 2024-03-04T12:12:32.215 INFO:teuthology.orchestra.run.smithi060.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 292 kB/s | 20 kB 00:00 2024-03-04T12:12:32.245 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T12:12:32.282 INFO:teuthology.orchestra.run.smithi060.stdout:(131/176): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-03-04T12:12:32.299 INFO:teuthology.orchestra.run.smithi060.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-03-04T12:12:32.358 INFO:teuthology.orchestra.run.smithi060.stdout:(133/176): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-04T12:12:32.383 INFO:teuthology.orchestra.run.smithi060.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 689 kB/s | 58 kB 00:00 2024-03-04T12:12:32.433 INFO:teuthology.orchestra.run.smithi060.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-04T12:12:32.458 INFO:teuthology.orchestra.run.smithi060.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-04T12:12:32.483 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T12:12:32.500 INFO:teuthology.orchestra.run.smithi060.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-04T12:12:32.534 INFO:teuthology.orchestra.run.smithi060.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-03-04T12:12:32.576 INFO:teuthology.orchestra.run.smithi060.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-03-04T12:12:32.597 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-04T12:12:32.609 INFO:teuthology.orchestra.run.smithi060.stdout:(140/176): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-03-04T12:12:32.651 INFO:teuthology.orchestra.run.smithi060.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2024-03-04T12:12:32.685 INFO:teuthology.orchestra.run.smithi060.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-03-04T12:12:32.700 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-04T12:12:32.727 INFO:teuthology.orchestra.run.smithi060.stdout:(143/176): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-03-04T12:12:32.733 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-04T12:12:32.761 INFO:teuthology.orchestra.run.smithi060.stdout:(144/176): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-04T12:12:32.802 INFO:teuthology.orchestra.run.smithi060.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-04T12:12:32.814 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-04T12:12:32.836 INFO:teuthology.orchestra.run.smithi060.stdout:(146/176): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-03-04T12:12:32.873 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-04T12:12:32.878 INFO:teuthology.orchestra.run.smithi060.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-03-04T12:12:32.903 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-04T12:12:32.912 INFO:teuthology.orchestra.run.smithi060.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-04T12:12:32.939 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-04T12:12:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-04T12:12:32.987 INFO:teuthology.orchestra.run.smithi060.stdout:(149/176): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-04T12:12:33.047 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-04T12:12:33.086 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-04T12:12:33.130 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-04T12:12:33.162 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-04T12:12:33.198 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-04T12:12:33.271 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-04T12:12:33.287 INFO:teuthology.orchestra.run.smithi060.stdout:(150/176): librados2-19.0.0-1845.g717ce3cc.el9. 9.0 MB/s | 3.4 MB 00:00 2024-03-04T12:12:33.304 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9. 39/202 2024-03-04T12:12:33.346 INFO:teuthology.orchestra.run.smithi060.stdout:(151/176): librbd1-19.0.0-1845.g717ce3cc.el9.x8 8.8 MB/s | 3.1 MB 00:00 2024-03-04T12:12:33.347 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el 40/202 2024-03-04T12:12:33.371 INFO:teuthology.orchestra.run.smithi060.stdout:(152/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 811 kB/s | 68 kB 00:00 2024-03-04T12:12:33.376 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_6 41/202 2024-03-04T12:12:33.447 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-04T12:12:33.491 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-04T12:12:33.505 INFO:teuthology.orchestra.run.smithi060.stdout:(153/176): NetworkManager-1.46.0-1.el9.x86_64.r 14 MB/s | 2.3 MB 00:00 2024-03-04T12:12:33.568 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-04T12:12:33.572 INFO:teuthology.orchestra.run.smithi060.stdout:(154/176): NetworkManager-libnm-1.46.0-1.el9.x8 9.1 MB/s | 1.8 MB 00:00 2024-03-04T12:12:33.597 INFO:teuthology.orchestra.run.smithi060.stdout:(155/176): NetworkManager-team-1.46.0-1.el9.x86 457 kB/s | 41 kB 00:00 2024-03-04T12:12:33.622 INFO:teuthology.orchestra.run.smithi060.stdout:(156/176): NetworkManager-tui-1.46.0-1.el9.x86_ 4.8 MB/s | 247 kB 00:00 2024-03-04T12:12:33.647 INFO:teuthology.orchestra.run.smithi060.stdout:(157/176): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 69 kB 00:00 2024-03-04T12:12:33.673 INFO:teuthology.orchestra.run.smithi060.stdout:(158/176): elfutils-debuginfod-client-0.190-2.e 705 kB/s | 35 kB 00:00 2024-03-04T12:12:33.698 INFO:teuthology.orchestra.run.smithi060.stdout:(159/176): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 191 kB 00:00 2024-03-04T12:12:33.706 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-04T12:12:33.724 INFO:teuthology.orchestra.run.smithi060.stdout:(160/176): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 254 kB 00:00 2024-03-04T12:12:33.760 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-04T12:12:33.772 INFO:teuthology.orchestra.run.smithi060.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 11 MB/s | 802 kB 00:00 2024-03-04T12:12:33.797 INFO:teuthology.orchestra.run.smithi060.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 5.9 MB/s | 433 kB 00:00 2024-03-04T12:12:33.823 INFO:teuthology.orchestra.run.smithi060.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 445 kB/s | 22 kB 00:00 2024-03-04T12:12:33.843 INFO:teuthology.orchestra.run.smithi086.stdout:(176/176): ceph-test-19.0.0-1845.g717ce3cc.el9. 1.8 MB/s | 47 MB 00:26 2024-03-04T12:12:33.848 INFO:teuthology.orchestra.run.smithi060.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-03-04T12:12:33.849 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-04T12:12:33.850 INFO:teuthology.orchestra.run.smithi086.stdout:Total 7.6 MB/s | 223 MB 00:29 2024-03-04T12:12:33.896 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-04T12:12:33.907 INFO:teuthology.orchestra.run.smithi060.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 154 kB 00:00 2024-03-04T12:12:33.932 INFO:teuthology.orchestra.run.smithi060.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 135 kB 00:00 2024-03-04T12:12:33.958 INFO:teuthology.orchestra.run.smithi060.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 63 kB 00:00 2024-03-04T12:12:33.983 INFO:teuthology.orchestra.run.smithi060.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 552 kB/s | 28 kB 00:00 2024-03-04T12:12:34.150 INFO:teuthology.orchestra.run.smithi060.stdout:(169/176): ceph-osd-19.0.0-1845.g717ce3cc.el9.x 1.4 MB/s | 17 MB 00:11 2024-03-04T12:12:34.171 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-04T12:12:34.171 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-04T12:12:34.192 INFO:teuthology.orchestra.run.smithi060.stdout:(170/176): openssl-3.0.7-27.el9.x86_64.rpm 4.9 MB/s | 1.2 MB 00:00 2024-03-04T12:12:34.234 INFO:teuthology.orchestra.run.smithi060.stdout:(171/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.6 MB/s | 2.2 MB 00:00 2024-03-04T12:12:34.284 INFO:teuthology.orchestra.run.smithi060.stdout:(172/176): util-linux-2.37.4-18.el9.x86_64.rpm 17 MB/s | 2.2 MB 00:00 2024-03-04T12:12:34.318 INFO:teuthology.orchestra.run.smithi060.stdout:(173/176): util-linux-core-2.37.4-18.el9.x86_64 3.5 MB/s | 438 kB 00:00 2024-03-04T12:12:34.324 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-04T12:12:34.343 INFO:teuthology.orchestra.run.smithi060.stdout:(174/176): util-linux-user-2.37.4-18.el9.x86_64 283 kB/s | 30 kB 00:00 2024-03-04T12:12:34.377 INFO:teuthology.orchestra.run.smithi060.stdout:(175/176): xfsdump-3.1.12-4.el9.x86_64.rpm 3.4 MB/s | 321 kB 00:00 2024-03-04T12:12:34.408 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-04T12:12:34.418 INFO:teuthology.orchestra.run.smithi060.stdout:(176/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 14 MB/s | 1.1 MB 00:00 2024-03-04T12:12:34.425 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-04T12:12:34.425 INFO:teuthology.orchestra.run.smithi060.stdout:Total 16 MB/s | 223 MB 00:13 2024-03-04T12:12:34.469 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-04T12:12:34.502 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-04T12:12:34.546 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T12:12:34.555 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T12:12:34.594 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-04T12:12:34.632 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_6 52/202 2024-03-04T12:12:34.663 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-04T12:12:34.756 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-04T12:12:34.807 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-04T12:12:34.807 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-04T12:12:34.813 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-04T12:12:35.066 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-04T12:12:35.188 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-04T12:12:35.257 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-04T12:12:35.302 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-04T12:12:35.650 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-04T12:12:35.705 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-04T12:12:35.762 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-04T12:12:35.835 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-04T12:12:35.886 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-04T12:12:35.887 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-04T12:12:35.888 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-04T12:12:35.925 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-04T12:12:35.972 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-04T12:12:36.007 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-04T12:12:36.051 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-04T12:12:36.094 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-04T12:12:36.235 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-04T12:12:36.290 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-04T12:12:36.320 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-04T12:12:36.379 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-04T12:12:36.476 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-04T12:12:36.512 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-04T12:12:36.552 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-04T12:12:36.577 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-04T12:12:36.602 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-04T12:12:36.677 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-04T12:12:36.809 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-04T12:12:36.876 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-04T12:12:36.877 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-04T12:12:36.921 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-04T12:12:36.968 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-04T12:12:37.026 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-04T12:12:37.082 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-04T12:12:37.277 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-04T12:12:37.331 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-04T12:12:37.482 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-04T12:12:37.520 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-04T12:12:37.553 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-04T12:12:37.584 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-04T12:12:37.628 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-04T12:12:37.659 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-04T12:12:37.687 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-04T12:12:37.722 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-04T12:12:37.769 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-04T12:12:37.809 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-04T12:12:37.839 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-04T12:12:37.918 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-04T12:12:37.949 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-04T12:12:37.983 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-04T12:12:38.024 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-04T12:12:38.058 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-04T12:12:38.151 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-04T12:12:38.249 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-04T12:12:38.307 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-04T12:12:38.337 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-04T12:12:38.385 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-04T12:12:38.487 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-04T12:12:38.540 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-04T12:12:39.314 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-04T12:12:39.369 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-04T12:12:39.403 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-04T12:12:39.452 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-04T12:12:39.543 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-04T12:12:39.587 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-04T12:12:39.772 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-04T12:12:39.809 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-04T12:12:39.890 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-04T12:12:40.564 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-04T12:12:40.676 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-04T12:12:41.975 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-04T12:12:42.054 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-04T12:12:42.081 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-04T12:12:42.410 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-04T12:12:42.435 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 125/202 2024-03-04T12:12:42.527 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 125/202 2024-03-04T12:12:42.560 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 126/202 2024-03-04T12:12:42.594 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-04T12:12:42.805 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-04T12:12:42.922 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-04T12:12:42.948 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-04T12:12:42.987 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T12:12:43.010 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T12:12:43.065 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-04T12:12:43.073 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-04T12:12:43.103 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-04T12:12:43.114 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 129/202 2024-03-04T12:12:43.157 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-04T12:12:43.187 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 129/202 2024-03-04T12:12:43.191 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-04T12:12:43.217 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-04T12:12:43.223 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 130/202 2024-03-04T12:12:43.258 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-04T12:12:43.577 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-04T12:12:43.634 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T12:12:43.739 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T12:12:43.760 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T12:12:43.834 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T12:12:43.864 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 13/202 2024-03-04T12:12:44.011 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-04T12:12:44.061 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-04T12:12:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T12:12:44.195 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T12:12:44.224 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 17/202 2024-03-04T12:12:44.246 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-04T12:12:44.287 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 17/202 2024-03-04T12:12:44.336 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-04T12:12:44.357 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-04T12:12:44.358 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T12:12:44.374 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T12:12:44.383 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-04T12:12:44.421 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T12:12:44.445 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-04T12:12:44.500 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-04T12:12:44.548 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-04T12:12:44.558 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T12:12:44.589 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T12:12:44.601 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-04T12:12:44.623 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-04T12:12:44.643 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-04T12:12:44.668 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-04T12:12:44.716 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-04T12:12:44.727 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-04T12:12:44.825 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T12:12:44.885 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T12:12:44.944 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T12:12:45.011 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T12:12:45.070 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-04T12:12:45.119 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T12:12:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T12:12:45.243 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T12:12:45.314 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-04T12:12:45.395 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T12:12:45.402 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-04T12:12:45.441 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-04T12:12:45.495 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T12:12:45.498 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T12:12:45.508 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-04T12:12:45.528 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T12:12:45.540 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 13/202 2024-03-04T12:12:45.554 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-04T12:12:45.586 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-04T12:12:45.621 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-04T12:12:45.661 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-04T12:12:45.703 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-04T12:12:45.723 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-04T12:12:45.759 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-04T12:12:45.766 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-04T12:12:45.794 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T12:12:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-04T12:12:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-04T12:12:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-04T12:12:45.882 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T12:12:45.912 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 17/202 2024-03-04T12:12:45.932 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-04T12:12:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9. 39/202 2024-03-04T12:12:45.977 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 17/202 2024-03-04T12:12:46.012 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el 40/202 2024-03-04T12:12:46.016 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-04T12:12:46.039 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T12:12:46.040 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_6 41/202 2024-03-04T12:12:46.054 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-04T12:12:46.096 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T12:12:46.108 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-04T12:12:46.131 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 132/202 2024-03-04T12:12:46.155 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-04T12:12:46.214 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 132/202 2024-03-04T12:12:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T12:12:46.214 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T12:12:46.214 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:12:46.222 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T12:12:46.226 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-04T12:12:46.241 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 133/202 2024-03-04T12:12:46.253 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-04T12:12:46.285 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-04T12:12:46.358 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-04T12:12:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-04T12:12:46.468 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-04T12:12:46.629 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T12:12:46.689 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-04T12:12:46.810 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T12:12:46.857 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T12:12:46.923 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-04T12:12:47.002 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-04T12:12:47.071 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-04T12:12:47.081 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-04T12:12:47.121 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T12:12:47.129 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T12:12:47.168 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-04T12:12:47.182 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-04T12:12:47.206 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_6 52/202 2024-03-04T12:12:47.247 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-04T12:12:47.277 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-04T12:12:47.334 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-04T12:12:47.380 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-04T12:12:47.412 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-04T12:12:47.482 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-04T12:12:47.543 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-04T12:12:47.582 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-04T12:12:47.618 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-04T12:12:47.625 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-04T12:12:47.657 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-04T12:12:47.724 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-04T12:12:47.757 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-04T12:12:47.767 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-04T12:12:47.800 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-04T12:12:47.831 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-04T12:12:47.833 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-04T12:12:47.861 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-04T12:12:47.876 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-04T12:12:47.932 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-04T12:12:47.966 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9. 39/202 2024-03-04T12:12:48.018 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el 40/202 2024-03-04T12:12:48.023 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-04T12:12:48.054 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_6 41/202 2024-03-04T12:12:48.062 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-04T12:12:48.114 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-04T12:12:48.127 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-04T12:12:48.169 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-04T12:12:48.194 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-04T12:12:48.226 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-04T12:12:48.262 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-04T12:12:48.269 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-04T12:12:48.402 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-04T12:12:48.408 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-04T12:12:48.464 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-04T12:12:48.587 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-04T12:12:48.722 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-04T12:12:48.764 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-04T12:12:48.896 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-04T12:12:48.944 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-04T12:12:48.974 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-04T12:12:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-04T12:12:49.016 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-04T12:12:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-04T12:12:49.144 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-04T12:12:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-04T12:12:49.189 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-04T12:12:49.215 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-04T12:12:49.239 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-04T12:12:49.247 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-04T12:12:49.295 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T12:12:49.302 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-04T12:12:49.303 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-04T12:12:49.342 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-04T12:12:49.380 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_6 52/202 2024-03-04T12:12:49.416 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-04T12:12:49.430 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-04T12:12:49.523 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-04T12:12:49.529 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-04T12:12:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-04T12:12:49.588 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-04T12:12:49.620 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-04T12:12:49.675 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-04T12:12:49.866 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-04T12:12:49.917 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-04T12:12:50.044 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-04T12:12:50.082 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-04T12:12:50.093 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-04T12:12:50.114 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-04T12:12:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-04T12:12:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-04T12:12:50.213 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-04T12:12:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-04T12:12:50.267 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-04T12:12:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-04T12:12:50.314 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-04T12:12:50.345 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-04T12:12:50.347 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-04T12:12:50.375 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-04T12:12:50.392 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-04T12:12:50.450 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-04T12:12:50.485 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-04T12:12:50.510 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-04T12:12:50.523 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-04T12:12:50.551 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-04T12:12:50.570 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-04T12:12:50.586 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-04T12:12:50.626 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-04T12:12:50.673 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-04T12:12:50.699 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-04T12:12:50.745 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-04T12:12:50.756 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-04T12:12:50.782 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-04T12:12:50.800 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-04T12:12:50.830 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-04T12:12:50.831 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-04T12:12:50.865 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-04T12:12:50.870 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-04T12:12:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-04T12:12:50.951 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-04T12:12:50.966 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-04T12:12:51.024 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-04T12:12:51.089 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-04T12:12:51.147 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-04T12:12:51.186 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-04T12:12:51.228 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-04T12:12:51.324 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-04T12:12:51.361 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-04T12:12:51.402 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-04T12:12:51.426 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-04T12:12:51.451 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-04T12:12:51.535 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-04T12:12:51.663 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-04T12:12:51.737 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-04T12:12:51.780 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-04T12:12:51.788 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-04T12:12:51.822 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-04T12:12:51.826 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-04T12:12:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-04T12:12:51.876 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-04T12:12:51.932 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-04T12:12:51.948 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-04T12:12:51.981 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-04T12:12:52.121 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-04T12:12:52.157 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-04T12:12:52.172 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-04T12:12:52.187 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-04T12:12:52.264 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-04T12:12:52.314 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-04T12:12:52.357 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-04T12:12:52.394 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-04T12:12:52.426 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-04T12:12:52.461 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-04T12:12:52.501 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-04T12:12:52.537 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-04T12:12:52.580 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-04T12:12:52.619 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-04T12:12:52.669 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-04T12:12:52.698 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-04T12:12:52.777 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-04T12:12:52.816 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-04T12:12:52.850 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-04T12:12:52.887 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-04T12:12:52.891 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-04T12:12:52.926 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-04T12:12:52.999 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-04T12:12:53.017 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-04T12:12:53.124 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-04T12:12:53.165 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-04T12:12:53.196 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-04T12:12:53.244 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-04T12:12:53.344 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-04T12:12:53.390 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-04T12:12:54.159 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-04T12:12:54.194 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-04T12:12:54.203 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-04T12:12:54.237 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-04T12:12:54.264 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-04T12:12:54.286 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-04T12:12:54.291 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-04T12:12:54.375 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-04T12:12:54.405 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-04T12:12:54.588 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-04T12:12:54.603 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-04T12:12:54.619 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-04T12:12:54.629 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 125/202 2024-03-04T12:12:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-04T12:12:54.718 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 125/202 2024-03-04T12:12:54.750 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 126/202 2024-03-04T12:12:54.787 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-04T12:12:55.238 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-04T12:12:55.265 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 129/202 2024-03-04T12:12:55.335 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 129/202 2024-03-04T12:12:55.358 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 130/202 2024-03-04T12:12:55.404 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-04T12:12:55.519 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-04T12:12:56.795 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-04T12:12:56.865 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-04T12:12:56.900 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-04T12:12:57.229 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-04T12:12:57.255 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 125/202 2024-03-04T12:12:57.345 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 125/202 2024-03-04T12:12:57.376 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 126/202 2024-03-04T12:12:57.413 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-04T12:12:57.496 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T12:12:57.529 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T12:12:57.894 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-04T12:12:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 129/202 2024-03-04T12:12:57.996 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 129/202 2024-03-04T12:12:58.026 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 130/202 2024-03-04T12:12:58.067 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T12:12:58.077 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 133/202 2024-03-04T12:12:58.078 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /sys 2024-03-04T12:12:58.078 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /proc 2024-03-04T12:12:58.078 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /mnt 2024-03-04T12:12:58.078 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /var/tmp 2024-03-04T12:12:58.078 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /home 2024-03-04T12:12:58.078 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /root 2024-03-04T12:12:58.078 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /tmp 2024-03-04T12:12:58.078 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:12:58.099 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 132/202 2024-03-04T12:12:58.172 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 132/202 2024-03-04T12:12:58.173 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T12:12:58.173 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T12:12:58.173 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:12:58.200 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 133/202 2024-03-04T12:12:58.244 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 134/202 2024-03-04T12:12:58.260 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 134/202 2024-03-04T12:12:58.261 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:12:58.261 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T12:12:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:12:59.807 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 135/202 2024-03-04T12:12:59.835 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 135/202 2024-03-04T12:12:59.835 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:12:59.835 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T12:12:59.835 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T12:12:59.835 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T12:12:59.835 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:13:00.058 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 136/202 2024-03-04T12:13:00.082 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 136/202 2024-03-04T12:13:00.082 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:00.083 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T12:13:00.083 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T12:13:00.083 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T12:13:00.083 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:13:00.314 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T12:13:00.356 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T12:13:00.550 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 137/202 2024-03-04T12:13:00.574 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 137/202 2024-03-04T12:13:00.574 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:00.574 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T12:13:00.575 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T12:13:00.575 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T12:13:00.575 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:13:00.677 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-04T12:13:00.927 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 131/202 2024-03-04T12:13:00.968 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 132/202 2024-03-04T12:13:01.042 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 132/202 2024-03-04T12:13:01.042 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T12:13:01.042 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T12:13:01.042 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:01.085 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 133/202 2024-03-04T12:13:01.121 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 139/202 2024-03-04T12:13:01.150 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 139/202 2024-03-04T12:13:01.182 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.e 140/202 2024-03-04T12:13:01.244 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc. 141/202 2024-03-04T12:13:01.282 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 142/202 2024-03-04T12:13:01.522 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 142/202 2024-03-04T12:13:01.557 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 143/202 2024-03-04T12:13:02.578 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 143/202 2024-03-04T12:13:02.608 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 144/202 2024-03-04T12:13:02.713 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 144/202 2024-03-04T12:13:02.878 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el 145/202 2024-03-04T12:13:02.910 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 146/202 2024-03-04T12:13:02.936 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 146/202 2024-03-04T12:13:02.936 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:02.936 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T12:13:02.936 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T12:13:02.936 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T12:13:02.936 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:13:02.994 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 147/202 2024-03-04T12:13:03.006 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 147/202 2024-03-04T12:13:03.051 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 148/202 2024-03-04T12:13:03.953 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-04T12:13:03.988 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 150/202 2024-03-04T12:13:04.012 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 150/202 2024-03-04T12:13:04.012 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:04.012 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T12:13:04.012 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T12:13:04.012 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T12:13:04.012 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:13:04.055 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1845.g717ce 151/202 2024-03-04T12:13:04.079 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1845.g717ce 151/202 2024-03-04T12:13:04.079 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:04.079 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T12:13:04.079 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:13:04.122 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 152/202 2024-03-04T12:13:04.146 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 152/202 2024-03-04T12:13:04.146 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:04.146 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-04T12:13:04.147 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T12:13:04.147 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T12:13:04.147 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:13:04.464 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T12:13:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T12:13:04.492 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:04.492 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T12:13:04.492 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T12:13:04.492 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T12:13:04.492 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:13:09.169 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 154/202 2024-03-04T12:13:09.211 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 155/202 2024-03-04T12:13:09.240 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 156/202 2024-03-04T12:13:09.268 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-04T12:13:09.465 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 158/202 2024-03-04T12:13:09.587 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-04T12:13:09.633 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 160/202 2024-03-04T12:13:09.672 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-04T12:13:09.732 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-04T12:13:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 133/202 2024-03-04T12:13:09.739 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-03-04T12:13:09.739 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-03-04T12:13:09.740 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-03-04T12:13:09.740 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-03-04T12:13:09.740 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-03-04T12:13:09.740 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /root 2024-03-04T12:13:09.740 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-03-04T12:13:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:13:09.801 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-04T12:13:09.842 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-04T12:13:09.894 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-04T12:13:09.904 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 134/202 2024-03-04T12:13:09.919 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 134/202 2024-03-04T12:13:09.919 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:09.919 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T12:13:09.919 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:13:09.924 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_ 166/202 2024-03-04T12:13:09.954 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-04T12:13:10.024 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-04T12:13:10.084 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-04T12:13:10.128 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-04T12:13:10.668 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T12:13:10.709 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T12:13:10.729 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T12:13:11.382 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 135/202 2024-03-04T12:13:11.407 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 135/202 2024-03-04T12:13:11.407 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:11.407 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T12:13:11.407 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T12:13:11.407 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T12:13:11.407 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:13:11.608 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 172/202 2024-03-04T12:13:11.616 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 136/202 2024-03-04T12:13:11.638 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 136/202 2024-03-04T12:13:11.638 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:11.638 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T12:13:11.638 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T12:13:11.638 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T12:13:11.638 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:13:11.687 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-04T12:13:11.728 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-04T12:13:11.812 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-04T12:13:11.849 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-04T12:13:11.873 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-04T12:13:11.899 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-04T12:13:11.925 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-04T12:13:11.926 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T12:13:12.027 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T12:13:12.055 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-04T12:13:12.075 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 137/202 2024-03-04T12:13:12.093 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-04T12:13:12.093 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T12:13:12.097 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 137/202 2024-03-04T12:13:12.097 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:12.097 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T12:13:12.097 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T12:13:12.097 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T12:13:12.097 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:13:12.178 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T12:13:12.193 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-04T12:13:12.204 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-04T12:13:12.238 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-04T12:13:12.263 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-04T12:13:12.290 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-04T12:13:12.331 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-04T12:13:12.372 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-04T12:13:12.411 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-04T12:13:12.443 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-04T12:13:12.469 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-04T12:13:12.494 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-04T12:13:12.528 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-04T12:13:12.561 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-04T12:13:12.568 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 139/202 2024-03-04T12:13:12.568 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T12:13:12.569 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T12:13:12.599 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 139/202 2024-03-04T12:13:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.e 140/202 2024-03-04T12:13:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc. 141/202 2024-03-04T12:13:12.714 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 142/202 2024-03-04T12:13:12.756 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T12:13:12.756 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T12:13:12.847 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T12:13:12.879 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-04T12:13:12.905 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-04T12:13:12.931 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-04T12:13:12.942 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 142/202 2024-03-04T12:13:12.956 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-04T12:13:12.973 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 143/202 2024-03-04T12:13:13.023 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T12:13:13.191 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 133/202 2024-03-04T12:13:13.191 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2024-03-04T12:13:13.191 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2024-03-04T12:13:13.191 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2024-03-04T12:13:13.192 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2024-03-04T12:13:13.192 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2024-03-04T12:13:13.192 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /root 2024-03-04T12:13:13.192 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2024-03-04T12:13:13.192 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:13.374 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 134/202 2024-03-04T12:13:13.389 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 134/202 2024-03-04T12:13:13.389 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:13.389 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T12:13:13.389 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:13.934 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 143/202 2024-03-04T12:13:13.957 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 144/202 2024-03-04T12:13:14.055 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 144/202 2024-03-04T12:13:14.227 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el 145/202 2024-03-04T12:13:14.259 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 146/202 2024-03-04T12:13:14.282 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 146/202 2024-03-04T12:13:14.282 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:14.282 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T12:13:14.282 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T12:13:14.282 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T12:13:14.282 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:13:14.326 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 147/202 2024-03-04T12:13:14.337 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 147/202 2024-03-04T12:13:14.382 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 148/202 2024-03-04T12:13:14.919 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 135/202 2024-03-04T12:13:14.944 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 135/202 2024-03-04T12:13:14.944 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:14.945 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T12:13:14.945 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T12:13:14.945 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T12:13:14.945 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:15.169 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 136/202 2024-03-04T12:13:15.192 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 136/202 2024-03-04T12:13:15.192 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:15.192 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T12:13:15.192 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T12:13:15.192 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T12:13:15.192 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:15.242 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-04T12:13:15.270 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 150/202 2024-03-04T12:13:15.292 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 150/202 2024-03-04T12:13:15.292 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:15.292 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T12:13:15.292 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T12:13:15.292 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T12:13:15.292 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:13:15.337 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1845.g717ce 151/202 2024-03-04T12:13:15.359 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1845.g717ce 151/202 2024-03-04T12:13:15.359 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:15.359 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T12:13:15.359 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:13:15.413 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 152/202 2024-03-04T12:13:15.435 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 152/202 2024-03-04T12:13:15.435 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:15.435 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-04T12:13:15.435 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T12:13:15.435 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T12:13:15.435 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:13:15.662 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 137/202 2024-03-04T12:13:15.683 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 137/202 2024-03-04T12:13:15.683 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:15.683 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T12:13:15.683 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T12:13:15.683 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T12:13:15.683 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:15.738 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T12:13:15.760 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T12:13:15.761 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:15.761 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T12:13:15.761 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T12:13:15.761 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T12:13:15.761 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:13:15.780 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-04T12:13:16.149 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 139/202 2024-03-04T12:13:16.187 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 139/202 2024-03-04T12:13:16.219 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.e 140/202 2024-03-04T12:13:16.282 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc. 141/202 2024-03-04T12:13:16.310 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 142/202 2024-03-04T12:13:16.550 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 142/202 2024-03-04T12:13:16.577 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 143/202 2024-03-04T12:13:17.599 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 143/202 2024-03-04T12:13:17.619 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 144/202 2024-03-04T12:13:17.701 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T12:13:17.701 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 1/202 2024-03-04T12:13:17.701 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 2/202 2024-03-04T12:13:17.701 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 3/202 2024-03-04T12:13:17.701 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 4/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1845.g717ce 5/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 6/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 7/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 8/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 9/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 10/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 13/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 14/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_ 15/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 17/202 2024-03-04T12:13:17.702 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_6 18/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 19/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 20/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el 21/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9. 22/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_6 23/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 24/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 25/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 26/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 27/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 28/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 29/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc. 30/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 31/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 32/202 2024-03-04T12:13:17.703 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 33/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el 34/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 35/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.e 36/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 37/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 38/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 39/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-04T12:13:17.704 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-04T12:13:17.705 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-04T12:13:17.706 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-04T12:13:17.707 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-04T12:13:17.708 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-04T12:13:17.709 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-04T12:13:17.710 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 151/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-04T12:13:17.711 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-04T12:13:17.712 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-04T12:13:17.713 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-04T12:13:17.714 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-04T12:13:17.715 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-04T12:13:17.725 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 144/202 2024-03-04T12:13:17.905 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el 145/202 2024-03-04T12:13:17.930 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 146/202 2024-03-04T12:13:17.952 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 146/202 2024-03-04T12:13:17.952 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:17.952 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T12:13:17.952 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T12:13:17.952 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T12:13:17.952 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:17.997 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 147/202 2024-03-04T12:13:18.008 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 147/202 2024-03-04T12:13:18.062 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 148/202 2024-03-04T12:13:18.965 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-04T12:13:18.991 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 150/202 2024-03-04T12:13:19.013 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 150/202 2024-03-04T12:13:19.013 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:19.013 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T12:13:19.013 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T12:13:19.013 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T12:13:19.013 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:19.059 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1845.g717ce 151/202 2024-03-04T12:13:19.080 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1845.g717ce 151/202 2024-03-04T12:13:19.080 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:19.080 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T12:13:19.080 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:19.126 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 152/202 2024-03-04T12:13:19.147 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 152/202 2024-03-04T12:13:19.148 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:19.148 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-04T12:13:19.148 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T12:13:19.148 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-04T12:13:19.148 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:19.460 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T12:13:19.482 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T12:13:19.482 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T12:13:19.482 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T12:13:19.482 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T12:13:19.482 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T12:13:19.482 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:20.160 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 154/202 2024-03-04T12:13:20.200 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 155/202 2024-03-04T12:13:20.239 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 156/202 2024-03-04T12:13:20.266 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-04T12:13:20.444 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 158/202 2024-03-04T12:13:20.489 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-04T12:13:20.489 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:13:20.489 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-03-04T12:13:20.489 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-04T12:13:20.489 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: acl-2.3.1-4.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-04T12:13:20.490 INFO:teuthology.orchestra.run.smithi063.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T12:13:20.491 INFO:teuthology.orchestra.run.smithi063.stdout: automake-1.16.2-8.el9.noarch 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: bison-3.7.4-5.el9.x86_64 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:20.492 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-04T12:13:20.493 INFO:teuthology.orchestra.run.smithi063.stdout: flex-2.6.4-9.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: jq-1.6-16.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T12:13:20.494 INFO:teuthology.orchestra.run.smithi063.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T12:13:20.495 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T12:13:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T12:13:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T12:13:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T12:13:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-04T12:13:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T12:13:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-13.11-1.el9.x86_64 2024-03-04T12:13:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-04T12:13:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-04T12:13:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-04T12:13:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T12:13:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T12:13:20.496 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-04T12:13:20.497 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T12:13:20.498 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T12:13:20.499 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T12:13:20.500 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.501 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.502 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:20.502 INFO:teuthology.orchestra.run.smithi063.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T12:13:20.502 INFO:teuthology.orchestra.run.smithi063.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T12:13:20.502 INFO:teuthology.orchestra.run.smithi063.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T12:13:20.502 INFO:teuthology.orchestra.run.smithi063.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-04T12:13:20.502 INFO:teuthology.orchestra.run.smithi063.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-04T12:13:20.502 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T12:13:20.502 INFO:teuthology.orchestra.run.smithi063.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-04T12:13:20.502 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-03-04T12:13:20.503 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-03-04T12:13:20.556 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-04T12:13:20.597 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 160/202 2024-03-04T12:13:20.638 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-04T12:13:20.683 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-04T12:13:20.760 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-04T12:13:20.816 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-04T12:13:20.859 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-04T12:13:20.889 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_ 166/202 2024-03-04T12:13:20.919 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-04T12:13:20.983 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-04T12:13:21.024 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-04T12:13:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-04T12:13:21.079 DEBUG:teuthology.parallel:result is None 2024-03-04T12:13:21.688 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T12:13:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T12:13:21.743 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T12:13:22.543 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 172/202 2024-03-04T12:13:22.611 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-04T12:13:22.644 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-04T12:13:22.734 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-04T12:13:22.790 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-04T12:13:22.815 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-04T12:13:22.840 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-04T12:13:22.867 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-04T12:13:22.867 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T12:13:22.952 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T12:13:22.979 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-04T12:13:23.018 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-04T12:13:23.018 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T12:13:23.102 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T12:13:23.128 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-04T12:13:23.162 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-04T12:13:23.187 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-04T12:13:23.215 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-04T12:13:23.247 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-04T12:13:23.280 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-04T12:13:23.319 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-04T12:13:23.351 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-04T12:13:23.377 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-04T12:13:23.402 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-04T12:13:23.427 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-04T12:13:23.452 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-04T12:13:23.459 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T12:13:23.459 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T12:13:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T12:13:23.647 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T12:13:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T12:13:23.761 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-04T12:13:23.788 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-04T12:13:23.814 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-04T12:13:23.839 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-04T12:13:23.903 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T12:13:24.206 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 154/202 2024-03-04T12:13:24.256 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 155/202 2024-03-04T12:13:24.286 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 156/202 2024-03-04T12:13:24.314 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-04T12:13:24.525 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 158/202 2024-03-04T12:13:24.650 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-04T12:13:24.695 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 160/202 2024-03-04T12:13:24.748 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-04T12:13:24.805 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-04T12:13:24.898 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-04T12:13:24.940 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-04T12:13:24.984 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-04T12:13:25.014 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_ 166/202 2024-03-04T12:13:25.044 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-04T12:13:25.125 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-04T12:13:25.181 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-04T12:13:25.225 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-04T12:13:25.835 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T12:13:25.873 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T12:13:25.892 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-04T12:13:26.770 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 172/202 2024-03-04T12:13:26.852 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-04T12:13:26.894 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-04T12:13:26.996 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-04T12:13:27.031 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-04T12:13:27.056 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-04T12:13:27.081 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-04T12:13:27.108 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-04T12:13:27.108 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T12:13:27.192 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-04T12:13:27.220 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-04T12:13:27.259 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-04T12:13:27.259 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T12:13:27.335 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-04T12:13:27.362 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-04T12:13:27.395 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-04T12:13:27.420 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-04T12:13:27.448 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-04T12:13:27.480 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-04T12:13:27.524 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-04T12:13:27.561 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-04T12:13:27.601 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-04T12:13:27.634 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-04T12:13:27.660 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-04T12:13:27.685 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-04T12:13:27.711 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-04T12:13:27.717 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T12:13:27.717 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T12:13:27.905 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-04T12:13:27.906 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T12:13:27.971 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-04T12:13:28.003 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-04T12:13:28.038 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-04T12:13:28.072 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-04T12:13:28.098 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-04T12:13:28.166 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T12:13:28.595 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T12:13:28.595 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 1/202 2024-03-04T12:13:28.595 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 2/202 2024-03-04T12:13:28.595 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 3/202 2024-03-04T12:13:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 4/202 2024-03-04T12:13:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1845.g717ce 5/202 2024-03-04T12:13:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 6/202 2024-03-04T12:13:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 7/202 2024-03-04T12:13:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 8/202 2024-03-04T12:13:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 9/202 2024-03-04T12:13:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 10/202 2024-03-04T12:13:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T12:13:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T12:13:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 13/202 2024-03-04T12:13:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 14/202 2024-03-04T12:13:28.596 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_ 15/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 17/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_6 18/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 19/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 20/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el 21/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9. 22/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_6 23/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 24/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 25/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 26/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 27/202 2024-03-04T12:13:28.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 28/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 29/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc. 30/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 31/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 32/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 33/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el 34/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 35/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.e 36/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 37/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 38/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 39/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-04T12:13:28.598 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-04T12:13:28.599 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-04T12:13:28.600 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-04T12:13:28.601 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-04T12:13:28.602 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-04T12:13:28.603 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-04T12:13:28.604 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-04T12:13:28.604 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-04T12:13:28.604 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-04T12:13:28.604 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-04T12:13:28.604 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-04T12:13:28.604 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-04T12:13:28.604 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-04T12:13:28.604 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-04T12:13:28.604 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-04T12:13:28.604 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-04T12:13:28.604 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-04T12:13:28.604 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-04T12:13:28.605 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-04T12:13:28.606 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-04T12:13:28.607 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 151/202 2024-03-04T12:13:28.607 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-04T12:13:28.607 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T12:13:28.607 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-04T12:13:28.607 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-04T12:13:28.607 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-04T12:13:28.607 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-04T12:13:28.607 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-04T12:13:28.607 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-04T12:13:28.607 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-04T12:13:28.607 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-04T12:13:28.607 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-04T12:13:28.608 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-04T12:13:28.609 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-04T12:13:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-04T12:13:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-04T12:13:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-04T12:13:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-04T12:13:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-04T12:13:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-04T12:13:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-04T12:13:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-04T12:13:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-04T12:13:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-04T12:13:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-04T12:13:31.464 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-04T12:13:31.464 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:13:31.464 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-03-04T12:13:31.464 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-04T12:13:31.464 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-04T12:13:31.464 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-04T12:13:31.464 INFO:teuthology.orchestra.run.smithi086.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: acl-2.3.1-4.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.465 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout: automake-1.16.2-8.el9.noarch 2024-03-04T12:13:31.466 INFO:teuthology.orchestra.run.smithi086.stdout: bison-3.7.4-5.el9.x86_64 2024-03-04T12:13:31.467 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T12:13:31.467 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.467 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.467 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.467 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.467 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:31.467 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.467 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.467 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.467 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:31.467 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:31.467 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:31.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:31.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:31.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:31.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.468 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:31.468 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:31.468 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.468 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: flex-2.6.4-9.el9.x86_64 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: jq-1.6-16.el9.x86_64 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T12:13:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.470 INFO:teuthology.orchestra.run.smithi086.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-13.11-1.el9.x86_64 2024-03-04T12:13:31.471 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-04T12:13:31.472 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-04T12:13:31.472 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-04T12:13:31.472 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T12:13:31.472 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T12:13:31.472 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T12:13:31.472 INFO:teuthology.orchestra.run.smithi086.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-04T12:13:31.472 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T12:13:31.472 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T12:13:31.472 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T12:13:31.472 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.472 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.472 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T12:13:31.473 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T12:13:31.474 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T12:13:31.475 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-04T12:13:31.476 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T12:13:31.477 INFO:teuthology.orchestra.run.smithi086.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-04T12:13:31.478 INFO:teuthology.orchestra.run.smithi086.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-04T12:13:31.478 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T12:13:31.478 INFO:teuthology.orchestra.run.smithi086.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-04T12:13:31.478 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:13:31.478 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-04T12:13:32.119 DEBUG:teuthology.parallel:result is None 2024-03-04T12:13:32.950 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-04T12:13:32.950 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 1/202 2024-03-04T12:13:32.950 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 2/202 2024-03-04T12:13:32.950 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 3/202 2024-03-04T12:13:32.950 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 4/202 2024-03-04T12:13:32.950 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1845.g717ce 5/202 2024-03-04T12:13:32.950 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 6/202 2024-03-04T12:13:32.950 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 7/202 2024-03-04T12:13:32.950 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 8/202 2024-03-04T12:13:32.950 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 9/202 2024-03-04T12:13:32.950 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 10/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 11/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 12/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 13/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 14/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_ 15/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 16/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 17/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_6 18/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86 19/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 20/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el 21/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9. 22/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_6 23/202 2024-03-04T12:13:32.951 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 24/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 25/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 26/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 27/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 28/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 29/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc. 30/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noa 31/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.n 32/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1845.g717 33/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el 34/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 35/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.e 36/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 37/202 2024-03-04T12:13:32.952 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 38/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 39/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-04T12:13:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-04T12:13:32.954 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-04T12:13:32.955 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-04T12:13:32.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-04T12:13:32.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-04T12:13:32.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-04T12:13:32.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-04T12:13:32.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-04T12:13:32.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-04T12:13:32.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-04T12:13:32.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-04T12:13:32.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-04T12:13:32.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-04T12:13:32.956 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-04T12:13:32.957 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-04T12:13:32.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-04T12:13:32.959 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-04T12:13:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-04T12:13:32.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-04T12:13:32.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-04T12:13:32.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 151/202 2024-03-04T12:13:32.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-04T12:13:32.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 153/202 2024-03-04T12:13:32.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-04T12:13:32.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-04T12:13:32.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-04T12:13:32.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-04T12:13:32.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-04T12:13:32.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-04T12:13:32.961 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-04T12:13:32.962 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-04T12:13:32.963 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-04T12:13:32.964 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-04T12:13:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-04T12:13:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:35.941 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2024-03-04T12:13:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-04T12:13:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-04T12:13:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-04T12:13:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-04T12:13:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: acl-2.3.1-4.el9.x86_64 2024-03-04T12:13:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-04T12:13:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-04T12:13:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-04T12:13:35.941 INFO:teuthology.orchestra.run.smithi060.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-04T12:13:35.942 INFO:teuthology.orchestra.run.smithi060.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: automake-1.16.2-8.el9.noarch 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: bison-3.7.4-5.el9.x86_64 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.943 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.944 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-shell-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: cephfs-top-2:19.0.0-1845.g717ce3cc.el9.noarch 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: flex-2.6.4-9.el9.x86_64 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: jq-1.6-16.el9.x86_64 2024-03-04T12:13:35.945 INFO:teuthology.orchestra.run.smithi060.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T12:13:35.946 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-13.11-1.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-04T12:13:35.947 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T12:13:35.948 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T12:13:35.949 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T12:13:35.950 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T12:13:35.951 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T12:13:35.952 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T12:13:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T12:13:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T12:13:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:19.0.0-1845.g717ce3cc.el9.x86_64 2024-03-04T12:13:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T12:13:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T12:13:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T12:13:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-04T12:13:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-04T12:13:35.953 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T12:13:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-04T12:13:35.954 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:13:35.954 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-04T12:13:36.526 DEBUG:teuthology.parallel:result is None 2024-03-04T12:13:36.526 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:13:36.674 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T12:13:36.696 INFO:teuthology.orchestra.run.smithi060.stdout:19.0.0-1845.g717ce3cc.el9 2024-03-04T12:13:36.697 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1845.g717ce3cc.el9 2024-03-04T12:13:36.697 INFO:teuthology.task.install:The correct ceph version 19.0.0-1845.g717ce3cc is installed. 2024-03-04T12:13:36.699 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:13:36.942 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T12:13:36.968 INFO:teuthology.orchestra.run.smithi063.stdout:19.0.0-1845.g717ce3cc.el9 2024-03-04T12:13:36.968 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1845.g717ce3cc.el9 2024-03-04T12:13:36.969 INFO:teuthology.task.install:The correct ceph version 19.0.0-1845.g717ce3cc is installed. 2024-03-04T12:13:36.971 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:13:37.110 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T12:13:37.136 INFO:teuthology.orchestra.run.smithi086.stdout:19.0.0-1845.g717ce3cc.el9 2024-03-04T12:13:37.136 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1845.g717ce3cc.el9 2024-03-04T12:13:37.136 INFO:teuthology.task.install:The correct ceph version 19.0.0-1845.g717ce3cc is installed. 2024-03-04T12:13:37.138 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-04T12:13:37.138 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:13:37.138 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T12:13:37.172 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-04T12:13:37.173 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T12:13:37.211 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T12:13:37.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T12:13:37.245 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-04T12:13:37.245 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:13:37.245 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T12:13:37.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T12:13:37.349 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-04T12:13:37.349 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T12:13:37.385 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T12:13:37.460 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T12:13:37.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T12:13:37.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T12:13:37.563 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-04T12:13:37.563 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:13:37.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T12:13:37.594 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T12:13:37.665 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-04T12:13:37.665 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T12:13:37.701 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T12:13:37.775 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T12:13:37.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T12:13:37.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T12:13:37.876 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-04T12:13:37.877 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:13:37.877 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T12:13:37.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T12:13:37.979 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-04T12:13:37.979 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T12:13:38.014 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T12:13:38.089 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T12:13:38.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T12:13:38.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T12:13:38.190 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-04T12:13:38.288 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0', 'subvols': {'create': 2}} 2024-03-04T12:13:38.288 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:13:38.288 INFO:tasks.cephadm:Cluster fsid is a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:13:38.289 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-04T12:13:38.289 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.60', 'mon.b': '172.21.15.63', 'mon.c': '172.21.15.86'} 2024-03-04T12:13:38.289 INFO:tasks.cephadm:First mon is mon.a on smithi060 2024-03-04T12:13:38.289 INFO:tasks.cephadm:First mgr is x 2024-03-04T12:13:38.289 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-04T12:13:38.289 DEBUG:teuthology.orchestra.run.smithi060:> sudo hostname $(hostname -s) 2024-03-04T12:13:38.321 DEBUG:teuthology.orchestra.run.smithi063:> sudo hostname $(hostname -s) 2024-03-04T12:13:38.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2024-03-04T12:13:38.388 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-04T12:13:38.388 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:13:38.603 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '717ce3cce8d0166e4e577faa8004e6f9cb4128c0', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77282/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1845-g717ce3cc', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1845.g717ce3cc'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-04 05:43:48.583911', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240304.042522', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-04T12:13:38.744 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-vshankar-testing-20240304.042522, sha1 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:13:38.746 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/x86_64/flavors/default/cephadm 2024-03-04T12:13:38.746 DEBUG:teuthology.orchestra.run.smithi060:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T12:13:39.199 INFO:teuthology.orchestra.run.smithi060.stdout:-rw-r--r--. 1 ubuntu ubuntu 767452 Mar 4 12:13 /home/ubuntu/cephtest/cephadm 2024-03-04T12:13:39.200 DEBUG:teuthology.orchestra.run.smithi063:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T12:13:39.639 INFO:teuthology.orchestra.run.smithi063.stdout:-rw-r--r--. 1 ubuntu ubuntu 767452 Mar 4 12:13 /home/ubuntu/cephtest/cephadm 2024-03-04T12:13:39.639 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240304.042522/717ce3cce8d0166e4e577faa8004e6f9cb4128c0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T12:13:40.292 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r--. 1 ubuntu ubuntu 767452 Mar 4 12:13 /home/ubuntu/cephtest/cephadm 2024-03-04T12:13:40.292 DEBUG:teuthology.orchestra.run.smithi060:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T12:13:40.311 DEBUG:teuthology.orchestra.run.smithi063:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T12:13:40.330 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T12:13:40.377 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 on all hosts... 2024-03-04T12:13:40.377 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 pull 2024-03-04T12:13:40.380 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 pull 2024-03-04T12:13:40.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 pull 2024-03-04T12:13:41.264 INFO:teuthology.orchestra.run.smithi060.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0... 2024-03-04T12:13:41.269 INFO:teuthology.orchestra.run.smithi086.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0... 2024-03-04T12:13:41.285 INFO:teuthology.orchestra.run.smithi063.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0... 2024-03-04T12:14:41.332 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:14:41.333 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_version": "ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev)", 2024-03-04T12:14:41.333 INFO:teuthology.orchestra.run.smithi060.stdout: "image_id": "d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf", 2024-03-04T12:14:41.333 INFO:teuthology.orchestra.run.smithi060.stdout: "repo_digests": [ 2024-03-04T12:14:41.333 INFO:teuthology.orchestra.run.smithi060.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377" 2024-03-04T12:14:41.333 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-04T12:14:41.333 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:14:49.036 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-04T12:14:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev)", 2024-03-04T12:14:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: "image_id": "d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf", 2024-03-04T12:14:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: "repo_digests": [ 2024-03-04T12:14:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377" 2024-03-04T12:14:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-04T12:14:49.036 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-04T12:14:50.950 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-04T12:14:50.951 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_version": "ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev)", 2024-03-04T12:14:50.951 INFO:teuthology.orchestra.run.smithi063.stdout: "image_id": "d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf", 2024-03-04T12:14:50.951 INFO:teuthology.orchestra.run.smithi063.stdout: "repo_digests": [ 2024-03-04T12:14:50.951 INFO:teuthology.orchestra.run.smithi063.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377" 2024-03-04T12:14:50.951 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-03-04T12:14:50.951 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-04T12:14:50.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph 2024-03-04T12:14:51.010 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph 2024-03-04T12:14:51.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2024-03-04T12:14:51.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /etc/ceph 2024-03-04T12:14:51.109 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /etc/ceph 2024-03-04T12:14:51.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2024-03-04T12:14:51.171 INFO:tasks.cephadm:Writing seed config... 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-04T12:14:51.172 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-04T12:14:51.173 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-04T12:14:51.174 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-04T12:14:51.175 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-04T12:14:51.175 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-04T12:14:51.175 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-04T12:14:51.175 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-04T12:14:51.175 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-04T12:14:51.175 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-04T12:14:51.176 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:14:51.176 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-04T12:14:51.193 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a1c49686-da20-11ee-95c6-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-04T12:14:51.193 DEBUG:teuthology.orchestra.run.smithi060:mon.a> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.a.service 2024-03-04T12:14:51.236 DEBUG:teuthology.orchestra.run.smithi060:mgr.x> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.x.service 2024-03-04T12:14:51.278 INFO:tasks.cephadm:Bootstrapping... 2024-03-04T12:14:51.278 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 -v bootstrap --fsid a1c49686-da20-11ee-95c6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.60 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-04T12:14:51.475 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-04T12:14:51.475 INFO:teuthology.orchestra.run.smithi060.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0', '-v', 'bootstrap', '--fsid', 'a1c49686-da20-11ee-95c6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.60', '--skip-admin-label'] 2024-03-04T12:14:51.499 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T12:14:51.500 INFO:teuthology.orchestra.run.smithi060.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-04T12:14:51.500 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying podman|docker is present... 2024-03-04T12:14:51.523 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T12:14:51.523 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying lvm2 is present... 2024-03-04T12:14:51.524 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying time synchronization is in place... 2024-03-04T12:14:51.531 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-04T12:14:51.531 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T12:14:51.537 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-04T12:14:51.537 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout inactive 2024-03-04T12:14:51.543 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout enabled 2024-03-04T12:14:51.549 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout active 2024-03-04T12:14:51.549 INFO:teuthology.orchestra.run.smithi060.stdout:Unit chronyd.service is enabled and running 2024-03-04T12:14:51.549 INFO:teuthology.orchestra.run.smithi060.stdout:Repeating the final host check... 2024-03-04T12:14:51.572 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T12:14:51.572 INFO:teuthology.orchestra.run.smithi060.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-04T12:14:51.572 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl is present 2024-03-04T12:14:51.572 INFO:teuthology.orchestra.run.smithi060.stdout:lvcreate is present 2024-03-04T12:14:51.578 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-04T12:14:51.579 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T12:14:51.584 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-04T12:14:51.584 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout inactive 2024-03-04T12:14:51.590 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout enabled 2024-03-04T12:14:51.596 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout active 2024-03-04T12:14:51.596 INFO:teuthology.orchestra.run.smithi060.stdout:Unit chronyd.service is enabled and running 2024-03-04T12:14:51.596 INFO:teuthology.orchestra.run.smithi060.stdout:Host looks OK 2024-03-04T12:14:51.596 INFO:teuthology.orchestra.run.smithi060.stdout:Cluster fsid: a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:14:51.596 INFO:teuthology.orchestra.run.smithi060.stdout:Acquiring lock 140167383383488 on /run/cephadm/a1c49686-da20-11ee-95c6-87774f69a715.lock 2024-03-04T12:14:51.596 INFO:teuthology.orchestra.run.smithi060.stdout:Lock 140167383383488 acquired on /run/cephadm/a1c49686-da20-11ee-95c6-87774f69a715.lock 2024-03-04T12:14:51.597 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying IP 172.21.15.60 port 3300 ... 2024-03-04T12:14:51.597 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying IP 172.21.15.60 port 6789 ... 2024-03-04T12:14:51.597 INFO:teuthology.orchestra.run.smithi060.stdout:Base mon IP(s) is [172.21.15.60:3300, 172.21.15.60:6789], mon addrv is [v2:172.21.15.60:3300,v1:172.21.15.60:6789] 2024-03-04T12:14:51.600 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.60 metric 100 2024-03-04T12:14:51.600 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.60 metric 100 2024-03-04T12:14:51.603 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-04T12:14:51.603 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-04T12:14:51.603 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-04T12:14:51.605 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-04T12:14:51.605 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-04T12:14:51.606 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-04T12:14:51.606 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-04T12:14:51.606 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2024-03-04T12:14:51.606 INFO:teuthology.orchestra.run.smithi060.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-04T12:14:51.606 INFO:teuthology.orchestra.run.smithi060.stdout:Mon IP `172.21.15.60` is in CIDR network `172.21.0.0/20` 2024-03-04T12:14:51.606 INFO:teuthology.orchestra.run.smithi060.stdout:Mon IP `172.21.15.60` is in CIDR network `172.21.0.0/20` 2024-03-04T12:14:51.606 INFO:teuthology.orchestra.run.smithi060.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-04T12:14:51.607 INFO:teuthology.orchestra.run.smithi060.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-04T12:14:51.608 INFO:teuthology.orchestra.run.smithi060.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0... 2024-03-04T12:14:52.075 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stdout d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf 2024-03-04T12:14:52.075 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0... 2024-03-04T12:14:52.075 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Getting image source signatures 2024-03-04T12:14:52.075 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Copying blob sha256:4b74008183047a4bc3222fc340675fc7aae42c4ded2fd13ee78566aa74ead154 2024-03-04T12:14:52.075 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-03-04T12:14:52.075 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Copying config sha256:d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf 2024-03-04T12:14:52.075 INFO:teuthology.orchestra.run.smithi060.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-04T12:14:53.113 INFO:teuthology.orchestra.run.smithi060.stdout:ceph: stdout ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T12:14:53.113 INFO:teuthology.orchestra.run.smithi060.stdout:Ceph version: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T12:14:53.114 INFO:teuthology.orchestra.run.smithi060.stdout:Extracting ceph user uid/gid from container image... 2024-03-04T12:14:54.119 INFO:teuthology.orchestra.run.smithi060.stdout:stat: stdout 167 167 2024-03-04T12:14:54.120 INFO:teuthology.orchestra.run.smithi060.stdout:Creating initial keys... 2024-03-04T12:14:55.173 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph-authtool: stdout AQA+u+Vlv7uDGxAA277LSF6ov1q2rdE8hfMWfQ== 2024-03-04T12:15:04.305 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph-authtool: stdout AQBCu+VlAqx3HBAAUxWIZkkmsfHBukAERI71Gg== 2024-03-04T12:15:05.193 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph-authtool: stdout AQBIu+Vle5j3JBAAf2wYU9Qev1tK61yTJEQnbg== 2024-03-04T12:15:05.195 INFO:teuthology.orchestra.run.smithi060.stdout:Creating initial monmap... 2024-03-04T12:15:06.158 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T12:15:06.159 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-04T12:15:06.159 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:15:06.159 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T12:15:06.159 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool for a [v2:172.21.15.60:3300,v1:172.21.15.60:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T12:15:06.159 INFO:teuthology.orchestra.run.smithi060.stdout:setting min_mon_release = quincy 2024-03-04T12:15:06.159 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/monmaptool: set fsid to a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:15:06.159 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T12:15:06.160 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:15:06.160 INFO:teuthology.orchestra.run.smithi060.stdout:Creating mon... 2024-03-04T12:15:07.023 INFO:teuthology.orchestra.run.smithi060.stdout:create mon.a on 2024-03-04T12:15:07.227 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-04T12:15:07.369 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-04T12:15:07.594 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a1c49686-da20-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-a1c49686-da20-11ee-95c6-87774f69a715.target. 2024-03-04T12:15:07.594 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a1c49686-da20-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-a1c49686-da20-11ee-95c6-87774f69a715.target. 2024-03-04T12:15:07.823 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.a 2024-03-04T12:15:07.823 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.a.service: Unit ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.a.service not loaded. 2024-03-04T12:15:07.977 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a1c49686-da20-11ee-95c6-87774f69a715.target.wants/ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.a.service → /etc/systemd/system/ceph-a1c49686-da20-11ee-95c6-87774f69a715@.service. 2024-03-04T12:15:08.437 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T12:15:08.438 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout disabled 2024-03-04T12:15:08.444 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T12:15:08.444 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout inactive 2024-03-04T12:15:08.444 INFO:teuthology.orchestra.run.smithi060.stdout:firewalld.service is not enabled 2024-03-04T12:15:08.444 INFO:teuthology.orchestra.run.smithi060.stdout:Not possible to enable service . firewalld.service is not available 2024-03-04T12:15:08.445 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for mon to start... 2024-03-04T12:15:08.445 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for mon... 2024-03-04T12:15:08.550 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:08 smithi060 systemd[1]: Started Ceph mon.a for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:15:09.456 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout cluster: 2024-03-04T12:15:09.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout id: a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:15:09.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-04T12:15:09.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:09.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout services: 2024-03-04T12:15:09.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.497551s) 2024-03-04T12:15:09.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-04T12:15:09.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-04T12:15:09.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:09.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout data: 2024-03-04T12:15:09.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-04T12:15:09.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-04T12:15:09.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout pgs: 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.833+0000 7f961a588700 1 Processor -- start 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.833+0000 7f961a588700 1 -- start start 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.833+0000 7f961a588700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 0x7f9614105910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.833+0000 7f961a588700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9614105e50 con 0x7f96141054f0 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.833+0000 7f9613fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 0x7f9614105910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.833+0000 7f9613fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 0x7f9614105910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43602/0 (socket says 172.21.15.60:43602) 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.833+0000 7f9613fff700 1 -- 172.21.15.60:0/1441725772 learned_addr learned my addr 172.21.15.60:0/1441725772 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.834+0000 7f9613fff700 1 -- 172.21.15.60:0/1441725772 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614105fd0 con 0x7f96141054f0 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.834+0000 7f9613fff700 1 --2- 172.21.15.60:0/1441725772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 0x7f9614105910 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f96040098a0 tx=0x7f9604009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f01949925c1fc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.834+0000 7f9612ffd700 1 -- 172.21.15.60:0/1441725772 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9604004160 con 0x7f96141054f0 2024-03-04T12:15:09.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.834+0000 7f9612ffd700 1 -- 172.21.15.60:0/1441725772 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9604004300 con 0x7f96141054f0 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.834+0000 7f9612ffd700 1 -- 172.21.15.60:0/1441725772 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96040036a0 con 0x7f96141054f0 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.834+0000 7f961a588700 1 -- 172.21.15.60:0/1441725772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 msgr2=0x7f9614105910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.834+0000 7f961a588700 1 --2- 172.21.15.60:0/1441725772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 0x7f9614105910 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f96040098a0 tx=0x7f9604009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.835+0000 7f961a588700 1 -- 172.21.15.60:0/1441725772 shutdown_connections 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.835+0000 7f961a588700 1 --2- 172.21.15.60:0/1441725772 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 0x7f9614105910 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.835+0000 7f961a588700 1 -- 172.21.15.60:0/1441725772 >> 172.21.15.60:0/1441725772 conn(0x7f9614100ff0 msgr2=0x7f9614103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.835+0000 7f961a588700 1 -- 172.21.15.60:0/1441725772 shutdown_connections 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.835+0000 7f961a588700 1 -- 172.21.15.60:0/1441725772 wait complete. 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.835+0000 7f961a588700 1 Processor -- start 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.835+0000 7f961a588700 1 -- start start 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.835+0000 7f961a588700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 0x7f96141086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.835+0000 7f961a588700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961410afd0 con 0x7f96141054f0 2024-03-04T12:15:09.459 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.836+0000 7f9613fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 0x7f96141086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.836+0000 7f9613fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 0x7f96141086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43614/0 (socket says 172.21.15.60:43614) 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.836+0000 7f9613fff700 1 -- 172.21.15.60:0/1912266559 learned_addr learned my addr 172.21.15.60:0/1912266559 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.836+0000 7f9613fff700 1 -- 172.21.15.60:0/1912266559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614108c00 con 0x7f96141054f0 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.836+0000 7f9613fff700 1 --2- 172.21.15.60:0/1912266559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 0x7f96141086c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9604000c00 tx=0x7f9604004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.836+0000 7f96117fa700 1 -- 172.21.15.60:0/1912266559 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9604003db0 con 0x7f96141054f0 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.836+0000 7f96117fa700 1 -- 172.21.15.60:0/1912266559 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9604011630 con 0x7f96141054f0 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.836+0000 7f961a588700 1 -- 172.21.15.60:0/1912266559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9614106df0 con 0x7f96141054f0 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.836+0000 7f96117fa700 1 -- 172.21.15.60:0/1912266559 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9604011950 con 0x7f96141054f0 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.836+0000 7f961a588700 1 -- 172.21.15.60:0/1912266559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9614107330 con 0x7f96141054f0 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.836+0000 7f96117fa700 1 -- 172.21.15.60:0/1912266559 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9604011af0 con 0x7f96141054f0 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.837+0000 7f96117fa700 1 -- 172.21.15.60:0/1912266559 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f960401c450 con 0x7f96141054f0 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.837+0000 7f961a588700 1 -- 172.21.15.60:0/1912266559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9614068f00 con 0x7f96141054f0 2024-03-04T12:15:09.460 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.840+0000 7f96117fa700 1 -- 172.21.15.60:0/1912266559 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9604032020 con 0x7f96141054f0 2024-03-04T12:15:09.461 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.914+0000 7f961a588700 1 -- 172.21.15.60:0/1912266559 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f9614194820 con 0x7f96141054f0 2024-03-04T12:15:09.461 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.914+0000 7f96117fa700 1 -- 172.21.15.60:0/1912266559 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f9604004260 con 0x7f96141054f0 2024-03-04T12:15:09.461 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.915+0000 7f961a588700 1 -- 172.21.15.60:0/1912266559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 msgr2=0x7f96141086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:09.461 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.915+0000 7f961a588700 1 --2- 172.21.15.60:0/1912266559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 0x7f96141086c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9604000c00 tx=0x7f9604004750 comp rx=0 tx=0).stop 2024-03-04T12:15:09.461 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.915+0000 7f961a588700 1 -- 172.21.15.60:0/1912266559 shutdown_connections 2024-03-04T12:15:09.461 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.915+0000 7f961a588700 1 --2- 172.21.15.60:0/1912266559 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f96141054f0 0x7f96141086c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:09.461 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.915+0000 7f961a588700 1 -- 172.21.15.60:0/1912266559 >> 172.21.15.60:0/1912266559 conn(0x7f9614100ff0 msgr2=0x7f9614103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:09.461 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.915+0000 7f961a588700 1 -- 172.21.15.60:0/1912266559 shutdown_connections 2024-03-04T12:15:09.461 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:08.916+0000 7f961a588700 1 -- 172.21.15.60:0/1912266559 wait complete. 2024-03-04T12:15:09.461 INFO:teuthology.orchestra.run.smithi060.stdout:mon is available 2024-03-04T12:15:09.461 INFO:teuthology.orchestra.run.smithi060.stdout:Assimilating anything we can from ceph.conf... 2024-03-04T12:15:09.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:09 smithi060 ceph-mon[32360]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T12:15:09.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:09 smithi060 ceph-mon[32360]: monmap e1: 1 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T12:15:09.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:09 smithi060 ceph-mon[32360]: fsmap 2024-03-04T12:15:09.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:09 smithi060 ceph-mon[32360]: osdmap e1: 0 total, 0 up, 0 in 2024-03-04T12:15:09.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:09 smithi060 ceph-mon[32360]: mgrmap e1: no daemons active 2024-03-04T12:15:09.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:09 smithi060 ceph-mon[32360]: from='client.? 172.21.15.60:0/1912266559' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-04T12:15:11.764 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:11 smithi060 ceph-mon[32360]: from='client.? 172.21.15.60:0/930422464' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-04T12:15:11.764 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:11 smithi060 ceph-mon[32360]: from='client.? 172.21.15.60:0/930422464' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-04T12:15:12.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout [global] 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout fsid = a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.60:3300,v1:172.21.15.60:6789] 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout [osd] 2024-03-04T12:15:12.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.369+0000 7fd42c93a700 1 Processor -- start 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.369+0000 7fd42c93a700 1 -- start start 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.369+0000 7fd42c93a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 0x7fd42810a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.370+0000 7fd42c93a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42807ae50 con 0x7fd428108290 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.370+0000 7fd42659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 0x7fd42810a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.370+0000 7fd42659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 0x7fd42810a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43616/0 (socket says 172.21.15.60:43616) 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.370+0000 7fd42659c700 1 -- 172.21.15.60:0/201009945 learned_addr learned my addr 172.21.15.60:0/201009945 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.370+0000 7fd42659c700 1 -- 172.21.15.60:0/201009945 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42810abc0 con 0x7fd428108290 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.370+0000 7fd42659c700 1 --2- 172.21.15.60:0/201009945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 0x7fd42810a680 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd4100098a0 tx=0x7fd410009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5178e7886ba32fd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:12.042 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.370+0000 7fd42559a700 1 -- 172.21.15.60:0/201009945 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd410004160 con 0x7fd428108290 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.370+0000 7fd42559a700 1 -- 172.21.15.60:0/201009945 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd410004300 con 0x7fd428108290 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.370+0000 7fd42559a700 1 -- 172.21.15.60:0/201009945 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4100036a0 con 0x7fd428108290 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.371+0000 7fd42c93a700 1 -- 172.21.15.60:0/201009945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 msgr2=0x7fd42810a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.371+0000 7fd42c93a700 1 --2- 172.21.15.60:0/201009945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 0x7fd42810a680 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd4100098a0 tx=0x7fd410009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.371+0000 7fd42c93a700 1 -- 172.21.15.60:0/201009945 shutdown_connections 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.371+0000 7fd42c93a700 1 --2- 172.21.15.60:0/201009945 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 0x7fd42810a680 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.371+0000 7fd42c93a700 1 -- 172.21.15.60:0/201009945 >> 172.21.15.60:0/201009945 conn(0x7fd428101d40 msgr2=0x7fd428104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.371+0000 7fd42c93a700 1 -- 172.21.15.60:0/201009945 shutdown_connections 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.371+0000 7fd42c93a700 1 -- 172.21.15.60:0/201009945 wait complete. 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.372+0000 7fd42c93a700 1 Processor -- start 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.372+0000 7fd42c93a700 1 -- start start 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.372+0000 7fd42c93a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 0x7fd428107ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.372+0000 7fd42c93a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42810b030 con 0x7fd428108290 2024-03-04T12:15:12.043 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.372+0000 7fd42659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 0x7fd428107ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.372+0000 7fd42659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 0x7fd428107ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43622/0 (socket says 172.21.15.60:43622) 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.372+0000 7fd42659c700 1 -- 172.21.15.60:0/930422464 learned_addr learned my addr 172.21.15.60:0/930422464 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.372+0000 7fd42659c700 1 -- 172.21.15.60:0/930422464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd428108000 con 0x7fd428108290 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.373+0000 7fd42659c700 1 --2- 172.21.15.60:0/930422464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 0x7fd428107ac0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd410000c00 tx=0x7fd410004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.373+0000 7fd41f7fe700 1 -- 172.21.15.60:0/930422464 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd410003db0 con 0x7fd428108290 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.373+0000 7fd41f7fe700 1 -- 172.21.15.60:0/930422464 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd410011630 con 0x7fd428108290 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.373+0000 7fd42c93a700 1 -- 172.21.15.60:0/930422464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd428106240 con 0x7fd428108290 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.373+0000 7fd41f7fe700 1 -- 172.21.15.60:0/930422464 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd410011950 con 0x7fd428108290 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.373+0000 7fd42c93a700 1 -- 172.21.15.60:0/930422464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd428106720 con 0x7fd428108290 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.373+0000 7fd41f7fe700 1 -- 172.21.15.60:0/930422464 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd410011af0 con 0x7fd428108290 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.373+0000 7fd41f7fe700 1 -- 172.21.15.60:0/930422464 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd41001c450 con 0x7fd428108290 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.374+0000 7fd42c93a700 1 -- 172.21.15.60:0/930422464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd428068f00 con 0x7fd428108290 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.376+0000 7fd41f7fe700 1 -- 172.21.15.60:0/930422464 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd410032020 con 0x7fd428108290 2024-03-04T12:15:12.044 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.447+0000 7fd42c93a700 1 -- 172.21.15.60:0/930422464 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fd428194810 con 0x7fd428108290 2024-03-04T12:15:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.451+0000 7fd41f7fe700 1 -- 172.21.15.60:0/930422464 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fd410004260 con 0x7fd428108290 2024-03-04T12:15:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.451+0000 7fd41f7fe700 1 -- 172.21.15.60:0/930422464 <== mon.0 v2:172.21.15.60:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd41001c650 con 0x7fd428108290 2024-03-04T12:15:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.452+0000 7fd42c93a700 1 -- 172.21.15.60:0/930422464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 msgr2=0x7fd428107ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.452+0000 7fd42c93a700 1 --2- 172.21.15.60:0/930422464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 0x7fd428107ac0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fd410000c00 tx=0x7fd410004750 comp rx=0 tx=0).stop 2024-03-04T12:15:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.452+0000 7fd42c93a700 1 -- 172.21.15.60:0/930422464 shutdown_connections 2024-03-04T12:15:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.452+0000 7fd42c93a700 1 --2- 172.21.15.60:0/930422464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd428108290 0x7fd428107ac0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.452+0000 7fd42c93a700 1 -- 172.21.15.60:0/930422464 >> 172.21.15.60:0/930422464 conn(0x7fd428101d40 msgr2=0x7fd428104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.452+0000 7fd42c93a700 1 -- 172.21.15.60:0/930422464 shutdown_connections 2024-03-04T12:15:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:11.452+0000 7fd42c93a700 1 -- 172.21.15.60:0/930422464 wait complete. 2024-03-04T12:15:12.045 INFO:teuthology.orchestra.run.smithi060.stdout:Generating new minimal ceph.conf... 2024-03-04T12:15:12.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:12 smithi060 ceph-mon[32360]: from='client.? 172.21.15.60:0/2974536654' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:15:13.129 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.440+0000 7fb190cb8700 1 Processor -- start 2024-03-04T12:15:13.130 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.440+0000 7fb190cb8700 1 -- start start 2024-03-04T12:15:13.130 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.440+0000 7fb190cb8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 0x7fb18c107cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:13.130 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.440+0000 7fb190cb8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c108290 con 0x7fb18c1078a0 2024-03-04T12:15:13.130 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.440+0000 7fb18a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 0x7fb18c107cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:13.130 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.441+0000 7fb18a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 0x7fb18c107cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43630/0 (socket says 172.21.15.60:43630) 2024-03-04T12:15:13.130 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.441+0000 7fb18a59c700 1 -- 172.21.15.60:0/3528313759 learned_addr learned my addr 172.21.15.60:0/3528313759 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:13.130 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.441+0000 7fb18a59c700 1 -- 172.21.15.60:0/3528313759 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c108a50 con 0x7fb18c1078a0 2024-03-04T12:15:13.130 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.441+0000 7fb18a59c700 1 --2- 172.21.15.60:0/3528313759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 0x7fb18c107cc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0098a0 tx=0x7fb17c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cca980f7e624f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:13.130 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.441+0000 7fb18959a700 1 -- 172.21.15.60:0/3528313759 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb17c004160 con 0x7fb18c1078a0 2024-03-04T12:15:13.130 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.441+0000 7fb18959a700 1 -- 172.21.15.60:0/3528313759 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb17c004300 con 0x7fb18c1078a0 2024-03-04T12:15:13.131 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.441+0000 7fb18959a700 1 -- 172.21.15.60:0/3528313759 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb17c011b50 con 0x7fb18c1078a0 2024-03-04T12:15:13.131 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.442+0000 7fb190cb8700 1 -- 172.21.15.60:0/3528313759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 msgr2=0x7fb18c107cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:13.131 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.442+0000 7fb190cb8700 1 --2- 172.21.15.60:0/3528313759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 0x7fb18c107cc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb17c0098a0 tx=0x7fb17c009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:13.131 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.442+0000 7fb190cb8700 1 -- 172.21.15.60:0/3528313759 shutdown_connections 2024-03-04T12:15:13.131 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.442+0000 7fb190cb8700 1 --2- 172.21.15.60:0/3528313759 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 0x7fb18c107cc0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:13.131 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.442+0000 7fb190cb8700 1 -- 172.21.15.60:0/3528313759 >> 172.21.15.60:0/3528313759 conn(0x7fb18c102ec0 msgr2=0x7fb18c1052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:13.131 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.442+0000 7fb190cb8700 1 -- 172.21.15.60:0/3528313759 shutdown_connections 2024-03-04T12:15:13.131 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.442+0000 7fb190cb8700 1 -- 172.21.15.60:0/3528313759 wait complete. 2024-03-04T12:15:13.131 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.443+0000 7fb190cb8700 1 Processor -- start 2024-03-04T12:15:13.131 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.443+0000 7fb190cb8700 1 -- start start 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.443+0000 7fb190cb8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 0x7fb18c07cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.443+0000 7fb190cb8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb18c10be00 con 0x7fb18c1078a0 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.443+0000 7fb18a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 0x7fb18c07cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.443+0000 7fb18a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 0x7fb18c07cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43632/0 (socket says 172.21.15.60:43632) 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.443+0000 7fb18a59c700 1 -- 172.21.15.60:0/2974536654 learned_addr learned my addr 172.21.15.60:0/2974536654 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.443+0000 7fb18a59c700 1 -- 172.21.15.60:0/2974536654 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb18c07d090 con 0x7fb18c1078a0 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.443+0000 7fb18a59c700 1 --2- 172.21.15.60:0/2974536654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 0x7fb18c07cb50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009fb0 tx=0x7fb17c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.443+0000 7fb17b7fe700 1 -- 172.21.15.60:0/2974536654 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb17c0038b0 con 0x7fb18c1078a0 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.444+0000 7fb17b7fe700 1 -- 172.21.15.60:0/2974536654 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb17c011e50 con 0x7fb18c1078a0 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.444+0000 7fb190cb8700 1 -- 172.21.15.60:0/2974536654 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18c07b280 con 0x7fb18c1078a0 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.444+0000 7fb17b7fe700 1 -- 172.21.15.60:0/2974536654 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb17c019550 con 0x7fb18c1078a0 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.444+0000 7fb190cb8700 1 -- 172.21.15.60:0/2974536654 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb18c07b7c0 con 0x7fb18c1078a0 2024-03-04T12:15:13.132 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.444+0000 7fb17b7fe700 1 -- 172.21.15.60:0/2974536654 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb17c019be0 con 0x7fb18c1078a0 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.444+0000 7fb17b7fe700 1 -- 172.21.15.60:0/2974536654 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb17c024730 con 0x7fb18c1078a0 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.445+0000 7fb190cb8700 1 -- 172.21.15.60:0/2974536654 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb18c068f00 con 0x7fb18c1078a0 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.447+0000 7fb17b7fe700 1 -- 172.21.15.60:0/2974536654 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb17c043dd0 con 0x7fb18c1078a0 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.517+0000 7fb190cb8700 1 -- 172.21.15.60:0/2974536654 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fb18c07bcd0 con 0x7fb18c1078a0 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.518+0000 7fb17b7fe700 1 -- 172.21.15.60:0/2974536654 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fb17c032030 con 0x7fb18c1078a0 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.518+0000 7fb190cb8700 1 -- 172.21.15.60:0/2974536654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 msgr2=0x7fb18c07cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.518+0000 7fb190cb8700 1 --2- 172.21.15.60:0/2974536654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 0x7fb18c07cb50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009fb0 tx=0x7fb17c011bf0 comp rx=0 tx=0).stop 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.518+0000 7fb190cb8700 1 -- 172.21.15.60:0/2974536654 shutdown_connections 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.518+0000 7fb190cb8700 1 --2- 172.21.15.60:0/2974536654 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb18c1078a0 0x7fb18c07cb50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.518+0000 7fb190cb8700 1 -- 172.21.15.60:0/2974536654 >> 172.21.15.60:0/2974536654 conn(0x7fb18c102ec0 msgr2=0x7fb18c103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.518+0000 7fb190cb8700 1 -- 172.21.15.60:0/2974536654 shutdown_connections 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:12.519+0000 7fb190cb8700 1 -- 172.21.15.60:0/2974536654 wait complete. 2024-03-04T12:15:13.133 INFO:teuthology.orchestra.run.smithi060.stdout:Restarting the monitor... 2024-03-04T12:15:13.383 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:13 smithi060 systemd[1]: Stopping Ceph mon.a for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:15:13.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:13 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a[32356]: 2024-03-04T12:15:13.380+0000 7fc19f950700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:15:13.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:13 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a[32356]: 2024-03-04T12:15:13.380+0000 7fc19f950700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-04T12:15:13.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:13 smithi060 podman[32618]: 2024-03-04 12:15:13.534289895 +0000 UTC m=+0.242578021 container died ec79852b63f7cdd1d5bcc32bdbafb2ead2623a155d51e5674d0ff657efef058e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, ceph=True) 2024-03-04T12:15:14.167 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:13 smithi060 podman[32618]: 2024-03-04 12:15:13.905605192 +0000 UTC m=+0.613893316 container cleanup ec79852b63f7cdd1d5bcc32bdbafb2ead2623a155d51e5674d0ff657efef058e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-03-04T12:15:14.167 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:13 smithi060 bash[32618]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a 2024-03-04T12:15:14.421 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 podman[32632]: 2024-03-04 12:15:14.164783461 +0000 UTC m=+0.629745023 container remove ec79852b63f7cdd1d5bcc32bdbafb2ead2623a155d51e5674d0ff657efef058e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5, ceph=True) 2024-03-04T12:15:14.421 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 systemd[1]: ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-04T12:15:14.421 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 systemd[1]: Stopped Ceph mon.a for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:15:14.421 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 systemd[1]: Starting Ceph mon.a for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:15:14.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 podman[32726]: 2024-03-04 12:15:14.532664981 +0000 UTC m=+0.108324565 container create 421225a59508cb2b1a0566bfd30e7843d4ff6ce9a9846904fb394ecf477e8025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:15:14.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 podman[32726]: 2024-03-04 12:15:14.443168308 +0000 UTC m=+0.018827856 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:15:14.841 INFO:teuthology.orchestra.run.smithi060.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-04T12:15:15.039 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 podman[32726]: 2024-03-04 12:15:14.751686514 +0000 UTC m=+0.327346057 container init 421225a59508cb2b1a0566bfd30e7843d4ff6ce9a9846904fb394ecf477e8025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0) 2024-03-04T12:15:15.039 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 podman[32726]: 2024-03-04 12:15:14.754443611 +0000 UTC m=+0.330103153 container start 421225a59508cb2b1a0566bfd30e7843d4ff6ce9a9846904fb394ecf477e8025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev), process ceph-mon, pid 2 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: pidfile_write: ignore empty --pid-file 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: load: jerasure load: lrc 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: RocksDB version: 7.9.2 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Git sha 0 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Compile date 2024-03-04 04:43:24 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: DB SUMMARY 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: DB Session ID: SR8HATJJBCRU2X6L2QMU 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: CURRENT file: CURRENT 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-04T12:15:15.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100322 ; 2024-03-04T12:15:15.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.error_if_exists: 0 2024-03-04T12:15:15.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.create_if_missing: 0 2024-03-04T12:15:15.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.paranoid_checks: 1 2024-03-04T12:15:15.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T12:15:15.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T12:15:15.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T12:15:15.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.env: 0x55d457f22b20 2024-03-04T12:15:15.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T12:15:15.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.info_log: 0x55d4592078e0 2024-03-04T12:15:15.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.statistics: (nil) 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.use_fsync: 0 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_log_file_size: 0 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.allow_fallocate: 1 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.use_direct_reads: 0 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T12:15:15.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.db_log_dir: 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.wal_dir: 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.write_buffer_manager: 0x55d4591db400 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T12:15:15.044 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.unordered_write: 0 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.row_cache: None 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.wal_filter: None 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T12:15:15.045 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.two_write_queues: 0 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.wal_compression: 0 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.atomic_flush: 0 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.log_readahead_size: 0 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_background_jobs: 2 2024-03-04T12:15:15.046 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_background_compactions: -1 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_subcompactions: 1 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_open_files: -1 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_background_flushes: -1 2024-03-04T12:15:15.047 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Compression algorithms supported: 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: kZSTD supported: 0 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: kXpressCompression supported: 0 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: kZlibCompression supported: 1 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: kSnappyCompression supported: 1 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: kLZ4Compression supported: 1 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: kBZip2Compression supported: 0 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T12:15:15.048 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.merge_operator: 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_filter: None 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d4592079e0) 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout: cache_index_and_filter_blocks: 1 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout: pin_top_level_index_and_filter: 1 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout: index_type: 0 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout: data_block_index_type: 0 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout: index_shortening: 1 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T12:15:15.049 INFO:journalctl@ceph.mon.a.smithi060.stdout: checksum: 4 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: no_block_cache: 0 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_cache: 0x55d45920d090 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_cache_name: BinnedLRUCache 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_cache_options: 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: capacity : 536870912 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: num_shard_bits : 4 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: strict_capacity_limit : 0 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: high_pri_pool_ratio: 0.000 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_cache_compressed: (nil) 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: persistent_cache: (nil) 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_size: 4096 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_size_deviation: 10 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_restart_interval: 16 2024-03-04T12:15:15.050 INFO:journalctl@ceph.mon.a.smithi060.stdout: index_block_restart_interval: 1 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: metadata_block_size: 4096 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: partition_filters: 0 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: use_delta_encoding: 1 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: filter_policy: bloomfilter 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: whole_key_filtering: 1 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: verify_compression: 0 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: read_amp_bytes_per_bit: 0 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: format_version: 5 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: enable_index_compression: 1 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: block_align: 0 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: max_auto_readahead_size: 262144 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: prepopulate_block_cache: 0 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: initial_auto_readahead_size: 8192 2024-03-04T12:15:15.051 INFO:journalctl@ceph.mon.a.smithi060.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compression: NoCompression 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.num_levels: 7 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T12:15:15.052 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T12:15:15.053 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T12:15:15.054 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T12:15:15.055 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.table_properties_collectors: 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.inplace_update_support: 0 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.bloom_locality: 0 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.max_successive_merges: 0 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T12:15:15.056 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.ttl: 2592000 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.enable_blob_files: false 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.min_blob_size: 0 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T12:15:15.057 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dae1d522-f944-4186-94b3-7f4d695cb3ec 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709554514782715, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709554514783662, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95451, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 318, "table_properties": {"data_size": 93437, "index_size": 296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13912, "raw_average_key_size": 50, "raw_value_size": 85881, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 277, "num_filter_entries": 277, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709554514, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dae1d522-f944-4186-94b3-7f4d695cb3ec", "db_session_id": "SR8HATJJBCRU2X6L2QMU", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709554514783745, "job": 1, "event": "recovery_finished"} 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d459300000 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: DB pointer 0x55d4592ea000 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout: ** DB Stats ** 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T12:15:15.058 INFO:journalctl@ceph.mon.a.smithi060.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: ** Compaction Stats [default] ** 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: L0 2/0 95.05 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: Sum 2/0 95.05 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: ** Compaction Stats [default] ** 2024-03-04T12:15:15.059 INFO:journalctl@ceph.mon.a.smithi060.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 156.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: Cumulative compaction: 0.00 GB write, 31.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: Interval compaction: 0.00 GB write, 31.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T12:15:15.060 INFO:journalctl@ceph.mon.a.smithi060.stdout: Block cache BinnedLRUCache@0x55d45920d090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout: 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: starting mon.a rank 0 at public addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] at bind addrs [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: mon.a@-1(???) e1 preinit fsid a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: mon.a@-1(???).mds e1 new map 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: mon.a@-1(???).mds e1 print_map 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout: e1 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout: legacy client fscid: -1 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout: 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout: No filesystems configured 2024-03-04T12:15:15.061 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T12:15:15.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T12:15:15.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T12:15:15.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T12:15:15.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-04T12:15:15.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 bash[32726]: 421225a59508cb2b1a0566bfd30e7843d4ff6ce9a9846904fb394ecf477e8025 2024-03-04T12:15:15.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 systemd[1]: Started Ceph mon.a for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:15:15.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T12:15:15.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: monmap e1: 1 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T12:15:15.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: fsmap 2024-03-04T12:15:15.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: osdmap e1: 0 total, 0 up, 0 in 2024-03-04T12:15:15.062 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:14 smithi060 ceph-mon[32745]: mgrmap e1: no daemons active 2024-03-04T12:15:16.396 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.249+0000 7f0aedb59700 1 Processor -- start 2024-03-04T12:15:16.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.249+0000 7f0aedb59700 1 -- start start 2024-03-04T12:15:16.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.249+0000 7f0aedb59700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 0x7f0ae807b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:16.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.249+0000 7f0aedb59700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae807b8b0 con 0x7f0ae807ce30 2024-03-04T12:15:16.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.250+0000 7f0ae77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 0x7f0ae807b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:16.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.250+0000 7f0ae77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 0x7f0ae807b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43644/0 (socket says 172.21.15.60:43644) 2024-03-04T12:15:16.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.250+0000 7f0ae77fe700 1 -- 172.21.15.60:0/1056426418 learned_addr learned my addr 172.21.15.60:0/1056426418 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:16.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.250+0000 7f0ae77fe700 1 -- 172.21.15.60:0/1056426418 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae807ba30 con 0x7f0ae807ce30 2024-03-04T12:15:16.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.250+0000 7f0ae77fe700 1 --2- 172.21.15.60:0/1056426418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 0x7f0ae807b2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80098a0 tx=0x7f0ad8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f333d9e9d583c8f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:16.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.250+0000 7f0ae67fc700 1 -- 172.21.15.60:0/1056426418 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ad8004160 con 0x7f0ae807ce30 2024-03-04T12:15:16.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.250+0000 7f0ae67fc700 1 -- 172.21.15.60:0/1056426418 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ad8004300 con 0x7f0ae807ce30 2024-03-04T12:15:16.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.250+0000 7f0ae67fc700 1 -- 172.21.15.60:0/1056426418 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ad8011b50 con 0x7f0ae807ce30 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.251+0000 7f0aedb59700 1 -- 172.21.15.60:0/1056426418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 msgr2=0x7f0ae807b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.251+0000 7f0aedb59700 1 --2- 172.21.15.60:0/1056426418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 0x7f0ae807b2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0ad80098a0 tx=0x7f0ad8009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.251+0000 7f0aedb59700 1 -- 172.21.15.60:0/1056426418 shutdown_connections 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.251+0000 7f0aedb59700 1 --2- 172.21.15.60:0/1056426418 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 0x7f0ae807b2e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.251+0000 7f0aedb59700 1 -- 172.21.15.60:0/1056426418 >> 172.21.15.60:0/1056426418 conn(0x7f0ae8102eb0 msgr2=0x7f0ae81052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.251+0000 7f0aedb59700 1 -- 172.21.15.60:0/1056426418 shutdown_connections 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.251+0000 7f0aedb59700 1 -- 172.21.15.60:0/1056426418 wait complete. 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.252+0000 7f0aedb59700 1 Processor -- start 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.252+0000 7f0aedb59700 1 -- start start 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.252+0000 7f0aedb59700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 0x7f0ae819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.252+0000 7f0aedb59700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae810dda0 con 0x7f0ae807ce30 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.252+0000 7f0ae77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 0x7f0ae819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.252+0000 7f0ae77fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 0x7f0ae819d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43656/0 (socket says 172.21.15.60:43656) 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.252+0000 7f0ae77fe700 1 -- 172.21.15.60:0/365519414 learned_addr learned my addr 172.21.15.60:0/365519414 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.252+0000 7f0ae77fe700 1 -- 172.21.15.60:0/365519414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae819de40 con 0x7f0ae807ce30 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.252+0000 7f0ae77fe700 1 --2- 172.21.15.60:0/365519414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 0x7f0ae819d900 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8011b60 tx=0x7f0ad8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.253+0000 7f0ae4ff9700 1 -- 172.21.15.60:0/365519414 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ad8003b20 con 0x7f0ae807ce30 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.253+0000 7f0ae4ff9700 1 -- 172.21.15.60:0/365519414 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ad8003cc0 con 0x7f0ae807ce30 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.253+0000 7f0aedb59700 1 -- 172.21.15.60:0/365519414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae819e0d0 con 0x7f0ae807ce30 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.253+0000 7f0ae4ff9700 1 -- 172.21.15.60:0/365519414 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ad8024840 con 0x7f0ae807ce30 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.253+0000 7f0aedb59700 1 -- 172.21.15.60:0/365519414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae81a0dc0 con 0x7f0ae807ce30 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.253+0000 7f0ae4ff9700 1 -- 172.21.15.60:0/365519414 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0ad8022020 con 0x7f0ae807ce30 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.253+0000 7f0ae4ff9700 1 -- 172.21.15.60:0/365519414 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0ad8011e00 con 0x7f0ae807ce30 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.254+0000 7f0aedb59700 1 -- 172.21.15.60:0/365519414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae8063450 con 0x7f0ae807ce30 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.256+0000 7f0ae4ff9700 1 -- 172.21.15.60:0/365519414 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0ad8043dd0 con 0x7f0ae807ce30 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.326+0000 7f0aedb59700 1 -- 172.21.15.60:0/365519414 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f0ae81a0f90 con 0x7f0ae807ce30 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.328+0000 7f0ae4ff9700 1 -- 172.21.15.60:0/365519414 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0ad8032030 con 0x7f0ae807ce30 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.329+0000 7f0aedb59700 1 -- 172.21.15.60:0/365519414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 msgr2=0x7f0ae819d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.329+0000 7f0aedb59700 1 --2- 172.21.15.60:0/365519414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 0x7f0ae819d900 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8011b60 tx=0x7f0ad8011c40 comp rx=0 tx=0).stop 2024-03-04T12:15:16.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.329+0000 7f0aedb59700 1 -- 172.21.15.60:0/365519414 shutdown_connections 2024-03-04T12:15:16.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.329+0000 7f0aedb59700 1 --2- 172.21.15.60:0/365519414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ae807ce30 0x7f0ae819d900 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:16.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.329+0000 7f0aedb59700 1 -- 172.21.15.60:0/365519414 >> 172.21.15.60:0/365519414 conn(0x7f0ae8102eb0 msgr2=0x7f0ae81039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:16.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.329+0000 7f0aedb59700 1 -- 172.21.15.60:0/365519414 shutdown_connections 2024-03-04T12:15:16.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:15.329+0000 7f0aedb59700 1 -- 172.21.15.60:0/365519414 wait complete. 2024-03-04T12:15:16.404 INFO:teuthology.orchestra.run.smithi060.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-04T12:15:16.418 INFO:teuthology.orchestra.run.smithi060.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-04T12:15:16.418 INFO:teuthology.orchestra.run.smithi060.stdout:Creating mgr... 2024-03-04T12:15:16.419 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-04T12:15:16.419 INFO:teuthology.orchestra.run.smithi060.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-04T12:15:16.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:16 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/365519414' entity='client.admin' 2024-03-04T12:15:16.676 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.x 2024-03-04T12:15:16.676 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.x.service: Unit ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.x.service not loaded. 2024-03-04T12:15:16.822 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a1c49686-da20-11ee-95c6-87774f69a715.target.wants/ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-a1c49686-da20-11ee-95c6-87774f69a715@.service. 2024-03-04T12:15:17.046 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:16 smithi060 systemd[1]: Starting Ceph mgr.x for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:15:17.315 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:17 smithi060 podman[32958]: 2024-03-04 12:15:17.043731921 +0000 UTC m=+0.108275798 container create fbd5b7e7c4abcab1de820af83c91af70913d0633755ea8909216636527974333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-04T12:15:17.315 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:17 smithi060 podman[32958]: 2024-03-04 12:15:16.952992309 +0000 UTC m=+0.017536205 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:15:17.315 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:17 smithi060 podman[32958]: 2024-03-04 12:15:17.254231229 +0000 UTC m=+0.318775114 container init fbd5b7e7c4abcab1de820af83c91af70913d0633755ea8909216636527974333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-03-04T12:15:17.316 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:17 smithi060 podman[32958]: 2024-03-04 12:15:17.256644021 +0000 UTC m=+0.321187900 container start fbd5b7e7c4abcab1de820af83c91af70913d0633755ea8909216636527974333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-04T12:15:17.316 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:17 smithi060 bash[32958]: fbd5b7e7c4abcab1de820af83c91af70913d0633755ea8909216636527974333 2024-03-04T12:15:17.329 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T12:15:17.329 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout disabled 2024-03-04T12:15:17.337 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T12:15:17.337 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout inactive 2024-03-04T12:15:17.337 INFO:teuthology.orchestra.run.smithi060.stdout:firewalld.service is not enabled 2024-03-04T12:15:17.337 INFO:teuthology.orchestra.run.smithi060.stdout:Not possible to enable service . firewalld.service is not available 2024-03-04T12:15:17.346 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T12:15:17.346 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout disabled 2024-03-04T12:15:17.353 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T12:15:17.354 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout inactive 2024-03-04T12:15:17.354 INFO:teuthology.orchestra.run.smithi060.stdout:firewalld.service is not enabled 2024-03-04T12:15:17.354 INFO:teuthology.orchestra.run.smithi060.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-04T12:15:17.354 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for mgr to start... 2024-03-04T12:15:17.354 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for mgr... 2024-03-04T12:15:17.571 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:17 smithi060 systemd[1]: Started Ceph mgr.x for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:15:18.138 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:17 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3762868947' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T12:15:18.138 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:17 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:17.835+0000 7fe85192d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T12:15:18.138 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:17 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:17.940+0000 7fe85192d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T12:15:18.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:18.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2024-03-04T12:15:18.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:15:18.401 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 0 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-04T12:15:18.402 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T12:15:18.403 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:18.404 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T12:15:08.419171+0000", 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T12:15:18.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.775+0000 7f13f181c700 1 Processor -- start 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.775+0000 7f13f181c700 1 -- start start 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.775+0000 7f13f181c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 0x7f13ec107c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.775+0000 7f13f181c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec1081f0 con 0x7f13ec107800 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.775+0000 7f13eaffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 0x7f13ec107c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.775+0000 7f13eaffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 0x7f13ec107c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50448/0 (socket says 172.21.15.60:50448) 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.775+0000 7f13eaffd700 1 -- 172.21.15.60:0/400912363 learned_addr learned my addr 172.21.15.60:0/400912363 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.775+0000 7f13eaffd700 1 -- 172.21.15.60:0/400912363 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec1089b0 con 0x7f13ec107800 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.775+0000 7f13eaffd700 1 --2- 172.21.15.60:0/400912363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 0x7f13ec107c20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0098a0 tx=0x7f13dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898711275ed48488 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.776+0000 7f13e9ffb700 1 -- 172.21.15.60:0/400912363 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13dc004160 con 0x7f13ec107800 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.776+0000 7f13e9ffb700 1 -- 172.21.15.60:0/400912363 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13dc004300 con 0x7f13ec107800 2024-03-04T12:15:18.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.776+0000 7f13e9ffb700 1 -- 172.21.15.60:0/400912363 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13dc011b50 con 0x7f13ec107800 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.776+0000 7f13f181c700 1 -- 172.21.15.60:0/400912363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 msgr2=0x7f13ec107c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.776+0000 7f13f181c700 1 --2- 172.21.15.60:0/400912363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 0x7f13ec107c20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0098a0 tx=0x7f13dc009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.776+0000 7f13f181c700 1 -- 172.21.15.60:0/400912363 shutdown_connections 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.776+0000 7f13f181c700 1 --2- 172.21.15.60:0/400912363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 0x7f13ec107c20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.776+0000 7f13f181c700 1 -- 172.21.15.60:0/400912363 >> 172.21.15.60:0/400912363 conn(0x7f13ec102e20 msgr2=0x7f13ec105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.776+0000 7f13f181c700 1 -- 172.21.15.60:0/400912363 shutdown_connections 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.776+0000 7f13f181c700 1 -- 172.21.15.60:0/400912363 wait complete. 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.777+0000 7f13f181c700 1 Processor -- start 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.777+0000 7f13f181c700 1 -- start start 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.777+0000 7f13f181c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 0x7f13ec19b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.777+0000 7f13f181c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec10bd60 con 0x7f13ec107800 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.777+0000 7f13eaffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 0x7f13ec19b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.777+0000 7f13eaffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 0x7f13ec19b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50464/0 (socket says 172.21.15.60:50464) 2024-03-04T12:15:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.777+0000 7f13eaffd700 1 -- 172.21.15.60:0/3762868947 learned_addr learned my addr 172.21.15.60:0/3762868947 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.778+0000 7f13eaffd700 1 -- 172.21.15.60:0/3762868947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec19bd30 con 0x7f13ec107800 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.778+0000 7f13eaffd700 1 --2- 172.21.15.60:0/3762868947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 0x7f13ec19b7f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f13dc011850 tx=0x7f13dc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.778+0000 7f13f081a700 1 -- 172.21.15.60:0/3762868947 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13dc003af0 con 0x7f13ec107800 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.778+0000 7f13f081a700 1 -- 172.21.15.60:0/3762868947 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f13dc020070 con 0x7f13ec107800 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.778+0000 7f13f181c700 1 -- 172.21.15.60:0/3762868947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ec19bfc0 con 0x7f13ec107800 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.778+0000 7f13f081a700 1 -- 172.21.15.60:0/3762868947 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13dc0195a0 con 0x7f13ec107800 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.778+0000 7f13f181c700 1 -- 172.21.15.60:0/3762868947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec19ecb0 con 0x7f13ec107800 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.778+0000 7f13f081a700 1 -- 172.21.15.60:0/3762868947 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f13dc019be0 con 0x7f13ec107800 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.778+0000 7f13f081a700 1 -- 172.21.15.60:0/3762868947 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f13dc011e00 con 0x7f13ec107800 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.779+0000 7f13f181c700 1 -- 172.21.15.60:0/3762868947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ec068f00 con 0x7f13ec107800 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.781+0000 7f13f081a700 1 -- 172.21.15.60:0/3762868947 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f13dc003c90 con 0x7f13ec107800 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.855+0000 7f13f181c700 1 -- 172.21.15.60:0/3762868947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f13ec19ee80 con 0x7f13ec107800 2024-03-04T12:15:18.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.855+0000 7f13f081a700 1 -- 172.21.15.60:0/3762868947 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f13dc00bad0 con 0x7f13ec107800 2024-03-04T12:15:18.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.856+0000 7f13f181c700 1 -- 172.21.15.60:0/3762868947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 msgr2=0x7f13ec19b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:18.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.856+0000 7f13f181c700 1 --2- 172.21.15.60:0/3762868947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 0x7f13ec19b7f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f13dc011850 tx=0x7f13dc011c10 comp rx=0 tx=0).stop 2024-03-04T12:15:18.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.857+0000 7f13f181c700 1 -- 172.21.15.60:0/3762868947 shutdown_connections 2024-03-04T12:15:18.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.857+0000 7f13f181c700 1 --2- 172.21.15.60:0/3762868947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13ec107800 0x7f13ec19b7f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:18.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.857+0000 7f13f181c700 1 -- 172.21.15.60:0/3762868947 >> 172.21.15.60:0/3762868947 conn(0x7f13ec102e20 msgr2=0x7f13ec1039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:18.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.857+0000 7f13f181c700 1 -- 172.21.15.60:0/3762868947 shutdown_connections 2024-03-04T12:15:18.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:17.857+0000 7f13f181c700 1 -- 172.21.15.60:0/3762868947 wait complete. 2024-03-04T12:15:18.409 INFO:teuthology.orchestra.run.smithi060.stdout:mgr not available, waiting (1/15)... 2024-03-04T12:15:18.675 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:18 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:18.609+0000 7fe85192d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T12:15:19.029 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:18 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:18.703+0000 7fe85192d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T12:15:19.029 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:18 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:18.845+0000 7fe85192d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T12:15:19.030 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:18 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:18.942+0000 7fe85192d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T12:15:19.030 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:19 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:19.026+0000 7fe85192d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T12:15:19.600 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:19 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:19.302+0000 7fe85192d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T12:15:19.600 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:19 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:19.414+0000 7fe85192d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T12:15:19.600 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:19 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:19.501+0000 7fe85192d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T12:15:19.600 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:19 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:19.597+0000 7fe85192d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T12:15:19.925 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:19 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:19.774+0000 7fe85192d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T12:15:21.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:20 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/101644010' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T12:15:21.175 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:20 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:20.780+0000 7fe85192d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T12:15:21.175 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:20 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:20.919+0000 7fe85192d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T12:15:21.475 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:21.475 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 0 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-04T12:15:21.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T12:15:21.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:21.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T12:15:21.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T12:15:21.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:21.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T12:15:21.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:21.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T12:15:21.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T12:15:21.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:21.485 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:21.486 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T12:15:08.419171+0000", 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.780+0000 7f9faa378700 1 Processor -- start 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.780+0000 7f9faa378700 1 -- start start 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.780+0000 7f9faa378700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 0x7f9fa41070f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.780+0000 7f9faa378700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa41076c0 con 0x7f9fa4106cd0 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.781+0000 7f9fa3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 0x7f9fa41070f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.781+0000 7f9fa3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 0x7f9fa41070f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50474/0 (socket says 172.21.15.60:50474) 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.781+0000 7f9fa3fff700 1 -- 172.21.15.60:0/3718217330 learned_addr learned my addr 172.21.15.60:0/3718217330 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.781+0000 7f9fa3fff700 1 -- 172.21.15.60:0/3718217330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa4107960 con 0x7f9fa4106cd0 2024-03-04T12:15:21.487 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.781+0000 7f9fa3fff700 1 --2- 172.21.15.60:0/3718217330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 0x7f9fa41070f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0098a0 tx=0x7f9f8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3050ca984f074e71 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.781+0000 7f9fa2ffd700 1 -- 172.21.15.60:0/3718217330 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f8c004160 con 0x7f9fa4106cd0 2024-03-04T12:15:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.781+0000 7f9fa2ffd700 1 -- 172.21.15.60:0/3718217330 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f8c004300 con 0x7f9fa4106cd0 2024-03-04T12:15:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.781+0000 7f9fa2ffd700 1 -- 172.21.15.60:0/3718217330 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f8c011b50 con 0x7f9fa4106cd0 2024-03-04T12:15:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.782+0000 7f9faa378700 1 -- 172.21.15.60:0/3718217330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 msgr2=0x7f9fa41070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.782+0000 7f9faa378700 1 --2- 172.21.15.60:0/3718217330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 0x7f9fa41070f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0098a0 tx=0x7f9f8c009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.782+0000 7f9faa378700 1 -- 172.21.15.60:0/3718217330 shutdown_connections 2024-03-04T12:15:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.782+0000 7f9faa378700 1 --2- 172.21.15.60:0/3718217330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 0x7f9fa41070f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.782+0000 7f9faa378700 1 -- 172.21.15.60:0/3718217330 >> 172.21.15.60:0/3718217330 conn(0x7f9fa4101500 msgr2=0x7f9fa4103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.782+0000 7f9faa378700 1 -- 172.21.15.60:0/3718217330 shutdown_connections 2024-03-04T12:15:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.782+0000 7f9faa378700 1 -- 172.21.15.60:0/3718217330 wait complete. 2024-03-04T12:15:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.783+0000 7f9faa378700 1 Processor -- start 2024-03-04T12:15:21.488 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.783+0000 7f9faa378700 1 -- start start 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.783+0000 7f9faa378700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 0x7f9fa419d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.783+0000 7f9faa378700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa4105920 con 0x7f9fa4106cd0 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.783+0000 7f9fa3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 0x7f9fa419d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.783+0000 7f9fa3fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 0x7f9fa419d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50488/0 (socket says 172.21.15.60:50488) 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.783+0000 7f9fa3fff700 1 -- 172.21.15.60:0/101644010 learned_addr learned my addr 172.21.15.60:0/101644010 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.783+0000 7f9fa3fff700 1 -- 172.21.15.60:0/101644010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa419df30 con 0x7f9fa4106cd0 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.784+0000 7f9fa3fff700 1 --2- 172.21.15.60:0/101644010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 0x7f9fa419d9f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c011b60 tx=0x7f9f8c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.784+0000 7f9fa17fa700 1 -- 172.21.15.60:0/101644010 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f8c003970 con 0x7f9fa4106cd0 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.784+0000 7f9fa17fa700 1 -- 172.21.15.60:0/101644010 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f8c003b10 con 0x7f9fa4106cd0 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.784+0000 7f9faa378700 1 -- 172.21.15.60:0/101644010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa419e1c0 con 0x7f9fa4106cd0 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.784+0000 7f9fa17fa700 1 -- 172.21.15.60:0/101644010 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f8c024610 con 0x7f9fa4106cd0 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.784+0000 7f9faa378700 1 -- 172.21.15.60:0/101644010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa41a0eb0 con 0x7f9fa4106cd0 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.784+0000 7f9fa17fa700 1 -- 172.21.15.60:0/101644010 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9f8c024c50 con 0x7f9fa4106cd0 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.784+0000 7f9fa17fa700 1 -- 172.21.15.60:0/101644010 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9f8c011e00 con 0x7f9fa4106cd0 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.785+0000 7f9faa378700 1 -- 172.21.15.60:0/101644010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa4068f00 con 0x7f9fa4106cd0 2024-03-04T12:15:21.489 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.787+0000 7f9fa17fa700 1 -- 172.21.15.60:0/101644010 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9f8c01a9f0 con 0x7f9fa4106cd0 2024-03-04T12:15:21.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.861+0000 7f9faa378700 1 -- 172.21.15.60:0/101644010 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9fa4196ff0 con 0x7f9fa4106cd0 2024-03-04T12:15:21.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.861+0000 7f9fa17fa700 1 -- 172.21.15.60:0/101644010 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f9f8c01a430 con 0x7f9fa4106cd0 2024-03-04T12:15:21.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.862+0000 7f9faa378700 1 -- 172.21.15.60:0/101644010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 msgr2=0x7f9fa419d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:21.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.862+0000 7f9faa378700 1 --2- 172.21.15.60:0/101644010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 0x7f9fa419d9f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c011b60 tx=0x7f9f8c011c40 comp rx=0 tx=0).stop 2024-03-04T12:15:21.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.862+0000 7f9faa378700 1 -- 172.21.15.60:0/101644010 shutdown_connections 2024-03-04T12:15:21.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.862+0000 7f9faa378700 1 --2- 172.21.15.60:0/101644010 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9fa4106cd0 0x7f9fa419d9f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:21.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.862+0000 7f9faa378700 1 -- 172.21.15.60:0/101644010 >> 172.21.15.60:0/101644010 conn(0x7f9fa4101500 msgr2=0x7f9fa41940b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:21.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.862+0000 7f9faa378700 1 -- 172.21.15.60:0/101644010 shutdown_connections 2024-03-04T12:15:21.490 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:20.862+0000 7f9faa378700 1 -- 172.21.15.60:0/101644010 wait complete. 2024-03-04T12:15:21.490 INFO:teuthology.orchestra.run.smithi060.stdout:mgr not available, waiting (2/15)... 2024-03-04T12:15:21.525 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:21 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:21.255+0000 7fe85192d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T12:15:21.525 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:21 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:21.347+0000 7fe85192d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T12:15:21.887 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:21 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:21.544+0000 7fe85192d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:15:21.887 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:21 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:21.731+0000 7fe85192d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T12:15:22.175 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:22 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:22.020+0000 7fe85192d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:15:22.675 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:22 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:22.281+0000 7fe85192d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T12:15:22.675 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:22 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:22.527+0000 7fe85192d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T12:15:23.606 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:23 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:23.348+0000 7fe85192d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T12:15:24.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:24 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/4200181912' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T12:15:24.175 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:23 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:23.942+0000 7fe85192d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T12:15:24.175 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:24 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:24.053+0000 7fe85192d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T12:15:24.456 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:24.456 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 0 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2024-03-04T12:15:24.457 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-04T12:15:24.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T12:15:24.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:24.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T12:15:24.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T12:15:24.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:24.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T12:15:24.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:24.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T12:15:24.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T12:15:24.458 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T12:15:24.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T12:15:24.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T12:15:24.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T12:15:24.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:24.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T12:15:24.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T12:15:08.419171+0000", 2024-03-04T12:15:24.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.877+0000 7f54bb011700 1 Processor -- start 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.877+0000 7f54bb011700 1 -- start start 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.878+0000 7f54bb011700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 0x7f54b410a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.878+0000 7f54bb011700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b407ae50 con 0x7f54b41082c0 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.878+0000 7f54b8dad700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 0x7f54b410a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.878+0000 7f54b8dad700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 0x7f54b410a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50494/0 (socket says 172.21.15.60:50494) 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.878+0000 7f54b8dad700 1 -- 172.21.15.60:0/3490043450 learned_addr learned my addr 172.21.15.60:0/3490043450 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.878+0000 7f54b8dad700 1 -- 172.21.15.60:0/3490043450 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b410abf0 con 0x7f54b41082c0 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.878+0000 7f54b8dad700 1 --2- 172.21.15.60:0/3490043450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 0x7f54b410a6b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f54a80098a0 tx=0x7f54a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f9ee11731585a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.878+0000 7f54b37fe700 1 -- 172.21.15.60:0/3490043450 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54a8004160 con 0x7f54b41082c0 2024-03-04T12:15:24.469 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.878+0000 7f54b37fe700 1 -- 172.21.15.60:0/3490043450 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54a8004300 con 0x7f54b41082c0 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.878+0000 7f54b37fe700 1 -- 172.21.15.60:0/3490043450 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54a8011b50 con 0x7f54b41082c0 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.879+0000 7f54bb011700 1 -- 172.21.15.60:0/3490043450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 msgr2=0x7f54b410a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.879+0000 7f54bb011700 1 --2- 172.21.15.60:0/3490043450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 0x7f54b410a6b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f54a80098a0 tx=0x7f54a8009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.879+0000 7f54bb011700 1 -- 172.21.15.60:0/3490043450 shutdown_connections 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.879+0000 7f54bb011700 1 --2- 172.21.15.60:0/3490043450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 0x7f54b410a6b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.879+0000 7f54bb011700 1 -- 172.21.15.60:0/3490043450 >> 172.21.15.60:0/3490043450 conn(0x7f54b4101dd0 msgr2=0x7f54b41041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.879+0000 7f54bb011700 1 -- 172.21.15.60:0/3490043450 shutdown_connections 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.879+0000 7f54bb011700 1 -- 172.21.15.60:0/3490043450 wait complete. 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.880+0000 7f54bb011700 1 Processor -- start 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.880+0000 7f54bb011700 1 -- start start 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.880+0000 7f54bb011700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 0x7f54b4107b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.880+0000 7f54bb011700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b410bba0 con 0x7f54b41082c0 2024-03-04T12:15:24.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.880+0000 7f54b8dad700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 0x7f54b4107b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.880+0000 7f54b8dad700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 0x7f54b4107b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:50506/0 (socket says 172.21.15.60:50506) 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.880+0000 7f54b8dad700 1 -- 172.21.15.60:0/4200181912 learned_addr learned my addr 172.21.15.60:0/4200181912 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.880+0000 7f54b8dad700 1 -- 172.21.15.60:0/4200181912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b4108080 con 0x7f54b41082c0 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.881+0000 7f54b8dad700 1 --2- 172.21.15.60:0/4200181912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 0x7f54b4107b40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009fb0 tx=0x7f54a8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.881+0000 7f54b1ffb700 1 -- 172.21.15.60:0/4200181912 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54a80038b0 con 0x7f54b41082c0 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.881+0000 7f54b1ffb700 1 -- 172.21.15.60:0/4200181912 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54a8011e50 con 0x7f54b41082c0 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.881+0000 7f54bb011700 1 -- 172.21.15.60:0/4200181912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b4106270 con 0x7f54b41082c0 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.881+0000 7f54b1ffb700 1 -- 172.21.15.60:0/4200181912 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54a8019420 con 0x7f54b41082c0 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.881+0000 7f54bb011700 1 -- 172.21.15.60:0/4200181912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b41067b0 con 0x7f54b41082c0 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.881+0000 7f54b1ffb700 1 -- 172.21.15.60:0/4200181912 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f54a8019ab0 con 0x7f54b41082c0 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.881+0000 7f54b1ffb700 1 -- 172.21.15.60:0/4200181912 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f54a8024730 con 0x7f54b41082c0 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.882+0000 7f54bb011700 1 -- 172.21.15.60:0/4200181912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b4068f00 con 0x7f54b41082c0 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.884+0000 7f54b1ffb700 1 -- 172.21.15.60:0/4200181912 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f54a8043dd0 con 0x7f54b41082c0 2024-03-04T12:15:24.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.957+0000 7f54bb011700 1 -- 172.21.15.60:0/4200181912 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f54b4106af0 con 0x7f54b41082c0 2024-03-04T12:15:24.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.958+0000 7f54b1ffb700 1 -- 172.21.15.60:0/4200181912 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f54a8032030 con 0x7f54b41082c0 2024-03-04T12:15:24.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.959+0000 7f54bb011700 1 -- 172.21.15.60:0/4200181912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 msgr2=0x7f54b4107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:24.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.959+0000 7f54bb011700 1 --2- 172.21.15.60:0/4200181912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 0x7f54b4107b40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009fb0 tx=0x7f54a8011bf0 comp rx=0 tx=0).stop 2024-03-04T12:15:24.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.959+0000 7f54bb011700 1 -- 172.21.15.60:0/4200181912 shutdown_connections 2024-03-04T12:15:24.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.959+0000 7f54bb011700 1 --2- 172.21.15.60:0/4200181912 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54b41082c0 0x7f54b4107b40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:24.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.959+0000 7f54bb011700 1 -- 172.21.15.60:0/4200181912 >> 172.21.15.60:0/4200181912 conn(0x7f54b4101dd0 msgr2=0x7f54b41028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:24.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.959+0000 7f54bb011700 1 -- 172.21.15.60:0/4200181912 shutdown_connections 2024-03-04T12:15:24.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:23.959+0000 7f54bb011700 1 -- 172.21.15.60:0/4200181912 wait complete. 2024-03-04T12:15:24.472 INFO:teuthology.orchestra.run.smithi060.stdout:mgr not available, waiting (3/15)... 2024-03-04T12:15:24.582 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:24 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:24.278+0000 7fe85192d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T12:15:25.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: Activating manager daemon x 2024-03-04T12:15:25.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: mgrmap e2: x(active, starting, since 0.00321032s) 2024-03-04T12:15:25.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: from='mgr.14100 172.21.15.60:0/3630362059' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T12:15:25.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: from='mgr.14100 172.21.15.60:0/3630362059' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T12:15:25.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: from='mgr.14100 172.21.15.60:0/3630362059' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T12:15:25.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: from='mgr.14100 172.21.15.60:0/3630362059' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:15:25.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: from='mgr.14100 172.21.15.60:0/3630362059' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T12:15:25.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: Manager daemon x is now available 2024-03-04T12:15:25.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: from='mgr.14100 172.21.15.60:0/3630362059' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T12:15:25.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: from='mgr.14100 172.21.15.60:0/3630362059' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T12:15:25.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: from='mgr.14100 172.21.15.60:0/3630362059' entity='mgr.x' 2024-03-04T12:15:25.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: from='mgr.14100 172.21.15.60:0/3630362059' entity='mgr.x' 2024-03-04T12:15:25.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:25 smithi060 ceph-mon[32745]: from='mgr.14100 172.21.15.60:0/3630362059' entity='mgr.x' 2024-03-04T12:15:26.569 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:26 smithi060 ceph-mon[32745]: mgrmap e3: x(active, since 1.0073s) 2024-03-04T12:15:27.572 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:27 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2424922947' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T12:15:27.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:27.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2024-03-04T12:15:27.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:15:27.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T12:15:27.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T12:15:27.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T12:15:27.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T12:15:27.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:27.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T12:15:27.713 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 0 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T12:15:27.714 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T12:15:27.723 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T12:15:27.723 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T12:15:27.723 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T12:15:27.723 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T12:15:27.723 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:27.724 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ], 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T12:15:27.725 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T12:15:08.419171+0000", 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout }, 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.876+0000 7fbcae59e700 1 Processor -- start 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.876+0000 7fbcae59e700 1 -- start start 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.876+0000 7fbcae59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 0x7fbca810a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.876+0000 7fbcae59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca807ae10 con 0x7fbca81080d0 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.877+0000 7fbcad59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 0x7fbca810a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.877+0000 7fbcad59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 0x7fbca810a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40356/0 (socket says 172.21.15.60:40356) 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.877+0000 7fbcad59c700 1 -- 172.21.15.60:0/3231068453 learned_addr learned my addr 172.21.15.60:0/3231068453 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.877+0000 7fbcad59c700 1 -- 172.21.15.60:0/3231068453 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca810aa00 con 0x7fbca81080d0 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.877+0000 7fbcad59c700 1 --2- 172.21.15.60:0/3231068453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 0x7fbca810a4c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0098a0 tx=0x7fbc9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f42b270cac229d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.877+0000 7fbc9bfff700 1 -- 172.21.15.60:0/3231068453 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc9c004160 con 0x7fbca81080d0 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.877+0000 7fbc9bfff700 1 -- 172.21.15.60:0/3231068453 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc9c004300 con 0x7fbca81080d0 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.877+0000 7fbc9bfff700 1 -- 172.21.15.60:0/3231068453 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc9c011b50 con 0x7fbca81080d0 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.878+0000 7fbcae59e700 1 -- 172.21.15.60:0/3231068453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 msgr2=0x7fbca810a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.878+0000 7fbcae59e700 1 --2- 172.21.15.60:0/3231068453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 0x7fbca810a4c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0098a0 tx=0x7fbc9c009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.878+0000 7fbcae59e700 1 -- 172.21.15.60:0/3231068453 shutdown_connections 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.878+0000 7fbcae59e700 1 --2- 172.21.15.60:0/3231068453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 0x7fbca810a4c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.878+0000 7fbcae59e700 1 -- 172.21.15.60:0/3231068453 >> 172.21.15.60:0/3231068453 conn(0x7fbca8101ae0 msgr2=0x7fbca8103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.878+0000 7fbcae59e700 1 -- 172.21.15.60:0/3231068453 shutdown_connections 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.878+0000 7fbcae59e700 1 -- 172.21.15.60:0/3231068453 wait complete. 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.879+0000 7fbcae59e700 1 Processor -- start 2024-03-04T12:15:27.727 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.879+0000 7fbcae59e700 1 -- start start 2024-03-04T12:15:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.879+0000 7fbcae59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 0x7fbca8107950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.879+0000 7fbcae59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca810b9c0 con 0x7fbca81080d0 2024-03-04T12:15:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.879+0000 7fbcad59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 0x7fbca8107950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.879+0000 7fbcad59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 0x7fbca8107950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40360/0 (socket says 172.21.15.60:40360) 2024-03-04T12:15:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.879+0000 7fbcad59c700 1 -- 172.21.15.60:0/2424922947 learned_addr learned my addr 172.21.15.60:0/2424922947 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.879+0000 7fbcad59c700 1 -- 172.21.15.60:0/2424922947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca8107e90 con 0x7fbca81080d0 2024-03-04T12:15:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.880+0000 7fbcad59c700 1 --2- 172.21.15.60:0/2424922947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 0x7fbca8107950 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c011b60 tx=0x7fbc9c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.880+0000 7fbc9a7fc700 1 -- 172.21.15.60:0/2424922947 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc9c003970 con 0x7fbca81080d0 2024-03-04T12:15:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.880+0000 7fbc9a7fc700 1 -- 172.21.15.60:0/2424922947 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc9c003b10 con 0x7fbca81080d0 2024-03-04T12:15:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.880+0000 7fbcae59e700 1 -- 172.21.15.60:0/2424922947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca8106080 con 0x7fbca81080d0 2024-03-04T12:15:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.880+0000 7fbc9a7fc700 1 -- 172.21.15.60:0/2424922947 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc9c024610 con 0x7fbca81080d0 2024-03-04T12:15:27.728 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.880+0000 7fbcae59e700 1 -- 172.21.15.60:0/2424922947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca81065c0 con 0x7fbca81080d0 2024-03-04T12:15:27.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.881+0000 7fbc9a7fc700 1 -- 172.21.15.60:0/2424922947 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 3) v1 ==== 45810+0+0 (secure 0 0 0) 0x7fbc9c024910 con 0x7fbca81080d0 2024-03-04T12:15:27.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.881+0000 7fbc9a7fc700 1 --2- 172.21.15.60:0/2424922947 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fbc94038a30 0x7fbc9403aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:27.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.881+0000 7fbcae59e700 1 -- 172.21.15.60:0/2424922947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca8068f00 con 0x7fbca81080d0 2024-03-04T12:15:27.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.881+0000 7fbc9a7fc700 1 -- 172.21.15.60:0/2424922947 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbc9c011e00 con 0x7fbca81080d0 2024-03-04T12:15:27.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.881+0000 7fbcacd9b700 1 --2- 172.21.15.60:0/2424922947 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fbc94038a30 0x7fbc9403aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:27.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.881+0000 7fbcacd9b700 1 --2- 172.21.15.60:0/2424922947 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fbc94038a30 0x7fbc9403aef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbca4006fd0 tx=0x7fbca4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:27.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:26.886+0000 7fbc9a7fc700 1 -- 172.21.15.60:0/2424922947 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbc9c00ba00 con 0x7fbca81080d0 2024-03-04T12:15:27.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:27.174+0000 7fbcae59e700 1 -- 172.21.15.60:0/2424922947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbca8106840 con 0x7fbca81080d0 2024-03-04T12:15:27.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:27.174+0000 7fbc9a7fc700 1 -- 172.21.15.60:0/2424922947 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fbc9c00bc60 con 0x7fbca81080d0 2024-03-04T12:15:27.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:27.176+0000 7fbcae59e700 1 -- 172.21.15.60:0/2424922947 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fbc94038a30 msgr2=0x7fbc9403aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:27.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:27.176+0000 7fbcae59e700 1 --2- 172.21.15.60:0/2424922947 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fbc94038a30 0x7fbc9403aef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fbca4006fd0 tx=0x7fbca4006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:27.731 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:27.176+0000 7fbcae59e700 1 -- 172.21.15.60:0/2424922947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 msgr2=0x7fbca8107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:27.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:27.176+0000 7fbcae59e700 1 --2- 172.21.15.60:0/2424922947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 0x7fbca8107950 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c011b60 tx=0x7fbc9c011c40 comp rx=0 tx=0).stop 2024-03-04T12:15:27.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:27.176+0000 7fbcae59e700 1 -- 172.21.15.60:0/2424922947 shutdown_connections 2024-03-04T12:15:27.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:27.176+0000 7fbcae59e700 1 --2- 172.21.15.60:0/2424922947 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fbc94038a30 0x7fbc9403aef0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:27.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:27.176+0000 7fbcae59e700 1 --2- 172.21.15.60:0/2424922947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbca81080d0 0x7fbca8107950 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:27.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:27.177+0000 7fbcae59e700 1 -- 172.21.15.60:0/2424922947 >> 172.21.15.60:0/2424922947 conn(0x7fbca8101ae0 msgr2=0x7fbca8102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:27.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:27.177+0000 7fbcae59e700 1 -- 172.21.15.60:0/2424922947 shutdown_connections 2024-03-04T12:15:27.732 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:27.177+0000 7fbcae59e700 1 -- 172.21.15.60:0/2424922947 wait complete. 2024-03-04T12:15:27.732 INFO:teuthology.orchestra.run.smithi060.stdout:mgr is available 2024-03-04T12:15:28.365 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:28 smithi060 ceph-mon[32745]: mgrmap e4: x(active, since 3s) 2024-03-04T12:15:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout [global] 2024-03-04T12:15:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout fsid = a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:15:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-04T12:15:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-04T12:15:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-04T12:15:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-04T12:15:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-04T12:15:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-04T12:15:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 2024-03-04T12:15:28.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout [osd] 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.138+0000 7f0adbcf8700 1 Processor -- start 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.138+0000 7f0adbcf8700 1 -- start start 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.138+0000 7f0adbcf8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 0x7f0ad407b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.138+0000 7f0adbcf8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad407b8f0 con 0x7f0ad407ce70 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.138+0000 7f0ad9a94700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 0x7f0ad407b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.138+0000 7f0ad9a94700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 0x7f0ad407b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40366/0 (socket says 172.21.15.60:40366) 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.138+0000 7f0ad9a94700 1 -- 172.21.15.60:0/3614256501 learned_addr learned my addr 172.21.15.60:0/3614256501 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.139+0000 7f0ad9a94700 1 -- 172.21.15.60:0/3614256501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad407ba70 con 0x7f0ad407ce70 2024-03-04T12:15:28.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.139+0000 7f0ad9a94700 1 --2- 172.21.15.60:0/3614256501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 0x7f0ad407b320 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00098a0 tx=0x7f0ad0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b837221f7668c5c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.139+0000 7f0ad8a92700 1 -- 172.21.15.60:0/3614256501 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ad0004160 con 0x7f0ad407ce70 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.139+0000 7f0ad8a92700 1 -- 172.21.15.60:0/3614256501 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ad0004300 con 0x7f0ad407ce70 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.139+0000 7f0ad8a92700 1 -- 172.21.15.60:0/3614256501 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ad0011b50 con 0x7f0ad407ce70 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.140+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3614256501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 msgr2=0x7f0ad407b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.140+0000 7f0adbcf8700 1 --2- 172.21.15.60:0/3614256501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 0x7f0ad407b320 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00098a0 tx=0x7f0ad0009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.140+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3614256501 shutdown_connections 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.140+0000 7f0adbcf8700 1 --2- 172.21.15.60:0/3614256501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 0x7f0ad407b320 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.140+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3614256501 >> 172.21.15.60:0/3614256501 conn(0x7f0ad4102ec0 msgr2=0x7f0ad41052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.140+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3614256501 shutdown_connections 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.140+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3614256501 wait complete. 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.140+0000 7f0adbcf8700 1 Processor -- start 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.140+0000 7f0adbcf8700 1 -- start start 2024-03-04T12:15:28.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.140+0000 7f0adbcf8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 0x7f0ad41a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.141+0000 7f0adbcf8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4109a00 con 0x7f0ad407ce70 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.141+0000 7f0ad9a94700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 0x7f0ad41a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.141+0000 7f0ad9a94700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 0x7f0ad41a1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40372/0 (socket says 172.21.15.60:40372) 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.141+0000 7f0ad9a94700 1 -- 172.21.15.60:0/3531238742 learned_addr learned my addr 172.21.15.60:0/3531238742 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.141+0000 7f0ad9a94700 1 -- 172.21.15.60:0/3531238742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad41a22f0 con 0x7f0ad407ce70 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.141+0000 7f0ad9a94700 1 --2- 172.21.15.60:0/3531238742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 0x7f0ad41a1db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0009fb0 tx=0x7f0ad0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.141+0000 7f0acaffd700 1 -- 172.21.15.60:0/3531238742 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ad00038b0 con 0x7f0ad407ce70 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.141+0000 7f0acaffd700 1 -- 172.21.15.60:0/3531238742 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ad0011e50 con 0x7f0ad407ce70 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.141+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3531238742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad41a2580 con 0x7f0ad407ce70 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.141+0000 7f0acaffd700 1 -- 172.21.15.60:0/3531238742 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ad0019420 con 0x7f0ad407ce70 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.141+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3531238742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad41a5270 con 0x7f0ad407ce70 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.142+0000 7f0acaffd700 1 -- 172.21.15.60:0/3531238742 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 45916+0+0 (secure 0 0 0) 0x7f0ad00196d0 con 0x7f0ad407ce70 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.142+0000 7f0acaffd700 1 --2- 172.21.15.60:0/3531238742 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f0ac0038b50 0x7f0ac003b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.142+0000 7f0acaffd700 1 -- 172.21.15.60:0/3531238742 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0ad004cda0 con 0x7f0ad407ce70 2024-03-04T12:15:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.142+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3531238742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad419b340 con 0x7f0ad407ce70 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.142+0000 7f0ad9293700 1 --2- 172.21.15.60:0/3531238742 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f0ac0038b50 0x7f0ac003b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.143+0000 7f0ad9293700 1 --2- 172.21.15.60:0/3531238742 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f0ac0038b50 0x7f0ac003b010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4006fd0 tx=0x7f0ac4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.147+0000 7f0acaffd700 1 -- 172.21.15.60:0/3531238742 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0ad004c7c0 con 0x7f0ad407ce70 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.349+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3531238742 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f0ad402daf0 con 0x7f0ad407ce70 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.350+0000 7f0acaffd700 1 -- 172.21.15.60:0/3531238742 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f0ad0032020 con 0x7f0ad407ce70 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.352+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3531238742 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f0ac0038b50 msgr2=0x7f0ac003b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.352+0000 7f0adbcf8700 1 --2- 172.21.15.60:0/3531238742 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f0ac0038b50 0x7f0ac003b010 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4006fd0 tx=0x7f0ac4006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.352+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3531238742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 msgr2=0x7f0ad41a1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.352+0000 7f0adbcf8700 1 --2- 172.21.15.60:0/3531238742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 0x7f0ad41a1db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0009fb0 tx=0x7f0ad0011bf0 comp rx=0 tx=0).stop 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.352+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3531238742 shutdown_connections 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.352+0000 7f0adbcf8700 1 --2- 172.21.15.60:0/3531238742 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f0ac0038b50 0x7f0ac003b010 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.352+0000 7f0adbcf8700 1 --2- 172.21.15.60:0/3531238742 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ad407ce70 0x7f0ad41a1db0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.352+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3531238742 >> 172.21.15.60:0/3531238742 conn(0x7f0ad4102ec0 msgr2=0x7f0ad41052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.352+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3531238742 shutdown_connections 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:28.352+0000 7f0adbcf8700 1 -- 172.21.15.60:0/3531238742 wait complete. 2024-03-04T12:15:28.870 INFO:teuthology.orchestra.run.smithi060.stdout:Enabling cephadm module... 2024-03-04T12:15:29.542 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:29 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3531238742' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-04T12:15:30.588 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:30 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/534242805' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-04T12:15:30.588 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:30 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: ignoring --setuser ceph since I am not root 2024-03-04T12:15:30.588 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:30 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: ignoring --setgroup ceph since I am not root 2024-03-04T12:15:30.813 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.272+0000 7f0bce4af700 1 Processor -- start 2024-03-04T12:15:30.813 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.272+0000 7f0bce4af700 1 -- start start 2024-03-04T12:15:30.813 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.272+0000 7f0bce4af700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 0x7f0bc807b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:30.813 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.272+0000 7f0bce4af700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc807b8b0 con 0x7f0bc807ce30 2024-03-04T12:15:30.813 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.272+0000 7f0bc7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 0x7f0bc807b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:30.814 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.272+0000 7f0bc7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 0x7f0bc807b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40388/0 (socket says 172.21.15.60:40388) 2024-03-04T12:15:30.814 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.272+0000 7f0bc7fff700 1 -- 172.21.15.60:0/3288011722 learned_addr learned my addr 172.21.15.60:0/3288011722 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:30.814 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.272+0000 7f0bc7fff700 1 -- 172.21.15.60:0/3288011722 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc807ba30 con 0x7f0bc807ce30 2024-03-04T12:15:30.814 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.273+0000 7f0bc7fff700 1 --2- 172.21.15.60:0/3288011722 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 0x7f0bc807b2e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80098a0 tx=0x7f0bb8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480c9e4148a2524a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:30.814 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.273+0000 7f0bc6ffd700 1 -- 172.21.15.60:0/3288011722 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bb8004160 con 0x7f0bc807ce30 2024-03-04T12:15:30.814 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.273+0000 7f0bc6ffd700 1 -- 172.21.15.60:0/3288011722 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bb8004300 con 0x7f0bc807ce30 2024-03-04T12:15:30.814 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.273+0000 7f0bc6ffd700 1 -- 172.21.15.60:0/3288011722 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bb8011b50 con 0x7f0bc807ce30 2024-03-04T12:15:30.814 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.273+0000 7f0bce4af700 1 -- 172.21.15.60:0/3288011722 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 msgr2=0x7f0bc807b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:30.814 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.273+0000 7f0bce4af700 1 --2- 172.21.15.60:0/3288011722 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 0x7f0bc807b2e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80098a0 tx=0x7f0bb8009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:30.814 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.273+0000 7f0bce4af700 1 -- 172.21.15.60:0/3288011722 shutdown_connections 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.273+0000 7f0bce4af700 1 --2- 172.21.15.60:0/3288011722 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 0x7f0bc807b2e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.273+0000 7f0bce4af700 1 -- 172.21.15.60:0/3288011722 >> 172.21.15.60:0/3288011722 conn(0x7f0bc8102eb0 msgr2=0x7f0bc81052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.274+0000 7f0bce4af700 1 -- 172.21.15.60:0/3288011722 shutdown_connections 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.274+0000 7f0bce4af700 1 -- 172.21.15.60:0/3288011722 wait complete. 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.274+0000 7f0bce4af700 1 Processor -- start 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.274+0000 7f0bce4af700 1 -- start start 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.274+0000 7f0bce4af700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 0x7f0bc819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.274+0000 7f0bce4af700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc810dda0 con 0x7f0bc807ce30 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.275+0000 7f0bc7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 0x7f0bc819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.275+0000 7f0bc7fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 0x7f0bc819d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40396/0 (socket says 172.21.15.60:40396) 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.275+0000 7f0bc7fff700 1 -- 172.21.15.60:0/534242805 learned_addr learned my addr 172.21.15.60:0/534242805 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.275+0000 7f0bc7fff700 1 -- 172.21.15.60:0/534242805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc819de40 con 0x7f0bc807ce30 2024-03-04T12:15:30.815 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.275+0000 7f0bc7fff700 1 --2- 172.21.15.60:0/534242805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 0x7f0bc819d900 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8011b60 tx=0x7f0bb8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.275+0000 7f0bc57fa700 1 -- 172.21.15.60:0/534242805 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bb8003b20 con 0x7f0bc807ce30 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.275+0000 7f0bc57fa700 1 -- 172.21.15.60:0/534242805 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bb8003cc0 con 0x7f0bc807ce30 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.275+0000 7f0bce4af700 1 -- 172.21.15.60:0/534242805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc819e0d0 con 0x7f0bc807ce30 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.275+0000 7f0bc57fa700 1 -- 172.21.15.60:0/534242805 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0bb8024840 con 0x7f0bc807ce30 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.275+0000 7f0bce4af700 1 -- 172.21.15.60:0/534242805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc81a0dc0 con 0x7f0bc807ce30 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.276+0000 7f0bc57fa700 1 -- 172.21.15.60:0/534242805 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 4) v1 ==== 45916+0+0 (secure 0 0 0) 0x7f0bb8022020 con 0x7f0bc807ce30 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.276+0000 7f0bce4af700 1 -- 172.21.15.60:0/534242805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc8063450 con 0x7f0bc807ce30 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.276+0000 7f0bc57fa700 1 --2- 172.21.15.60:0/534242805 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f0bb0038b50 0x7f0bb003b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.276+0000 7f0bc57fa700 1 -- 172.21.15.60:0/534242805 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0bb8011e00 con 0x7f0bc807ce30 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.276+0000 7f0bc77fe700 1 --2- 172.21.15.60:0/534242805 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f0bb0038b50 0x7f0bb003b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.277+0000 7f0bc77fe700 1 --2- 172.21.15.60:0/534242805 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f0bb0038b50 0x7f0bb003b010 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc006fd0 tx=0x7f0bbc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.281+0000 7f0bc57fa700 1 -- 172.21.15.60:0/534242805 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0bb804c650 con 0x7f0bc807ce30 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:29.539+0000 7f0bce4af700 1 -- 172.21.15.60:0/534242805 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f0bc81a0f90 con 0x7f0bc807ce30 2024-03-04T12:15:30.816 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:30.302+0000 7f0bc57fa700 1 -- 172.21.15.60:0/534242805 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f0bc81a0f90 con 0x7f0bc807ce30 2024-03-04T12:15:30.817 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:30.304+0000 7f0bc57fa700 1 -- 172.21.15.60:0/534242805 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 5) v1 ==== 45927+0+0 (secure 0 0 0) 0x7f0bb801a960 con 0x7f0bc807ce30 2024-03-04T12:15:30.817 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:30.306+0000 7f0bce4af700 1 -- 172.21.15.60:0/534242805 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f0bb0038b50 msgr2=0x7f0bb003b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:30.817 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:30.306+0000 7f0bce4af700 1 --2- 172.21.15.60:0/534242805 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f0bb0038b50 0x7f0bb003b010 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc006fd0 tx=0x7f0bbc006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:30.817 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:30.306+0000 7f0bce4af700 1 -- 172.21.15.60:0/534242805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 msgr2=0x7f0bc819d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:30.817 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:30.306+0000 7f0bce4af700 1 --2- 172.21.15.60:0/534242805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 0x7f0bc819d900 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8011b60 tx=0x7f0bb8011c40 comp rx=0 tx=0).stop 2024-03-04T12:15:30.817 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:30.306+0000 7f0bce4af700 1 -- 172.21.15.60:0/534242805 shutdown_connections 2024-03-04T12:15:30.817 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:30.306+0000 7f0bce4af700 1 --2- 172.21.15.60:0/534242805 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f0bb0038b50 0x7f0bb003b010 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:30.817 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:30.306+0000 7f0bce4af700 1 --2- 172.21.15.60:0/534242805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0bc807ce30 0x7f0bc819d900 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:30.817 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:30.306+0000 7f0bce4af700 1 -- 172.21.15.60:0/534242805 >> 172.21.15.60:0/534242805 conn(0x7f0bc8102eb0 msgr2=0x7f0bc81039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:30.817 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:30.306+0000 7f0bce4af700 1 -- 172.21.15.60:0/534242805 shutdown_connections 2024-03-04T12:15:30.817 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:30.306+0000 7f0bce4af700 1 -- 172.21.15.60:0/534242805 wait complete. 2024-03-04T12:15:31.133 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:30 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:30.880+0000 7f26e5226200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T12:15:31.133 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:30 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:30.975+0000 7f26e5226200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T12:15:31.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:31 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/534242805' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-04T12:15:31.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:31 smithi060 ceph-mon[32745]: mgrmap e5: x(active, since 6s) 2024-03-04T12:15:31.740 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:31 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:31.643+0000 7f26e5226200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T12:15:32.067 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:31 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:31.738+0000 7f26e5226200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T12:15:32.067 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:31 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:31.881+0000 7f26e5226200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T12:15:32.067 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:31 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:31.979+0000 7f26e5226200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T12:15:32.067 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:32 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:32.064+0000 7f26e5226200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T12:15:32.090 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.229+0000 7f7e56951700 1 Processor -- start 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.229+0000 7f7e56951700 1 -- start start 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.229+0000 7f7e56951700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 0x7f7e50106b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.229+0000 7f7e56951700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50107140 con 0x7f7e50106750 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.230+0000 7f7e4ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 0x7f7e50106b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.230+0000 7f7e4ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 0x7f7e50106b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40424/0 (socket says 172.21.15.60:40424) 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.230+0000 7f7e4ffff700 1 -- 172.21.15.60:0/700199647 learned_addr learned my addr 172.21.15.60:0/700199647 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.230+0000 7f7e4ffff700 1 -- 172.21.15.60:0/700199647 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e50107950 con 0x7f7e50106750 2024-03-04T12:15:32.091 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.230+0000 7f7e4ffff700 1 --2- 172.21.15.60:0/700199647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 0x7f7e50106b70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e400098a0 tx=0x7f7e40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9f4a4edd155cae server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.230+0000 7f7e4effd700 1 -- 172.21.15.60:0/700199647 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e40004160 con 0x7f7e50106750 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.230+0000 7f7e4effd700 1 -- 172.21.15.60:0/700199647 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e40004300 con 0x7f7e50106750 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.230+0000 7f7e4effd700 1 -- 172.21.15.60:0/700199647 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e40011b50 con 0x7f7e50106750 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.231+0000 7f7e56951700 1 -- 172.21.15.60:0/700199647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 msgr2=0x7f7e50106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.231+0000 7f7e56951700 1 --2- 172.21.15.60:0/700199647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 0x7f7e50106b70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7e400098a0 tx=0x7f7e40009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.231+0000 7f7e56951700 1 -- 172.21.15.60:0/700199647 shutdown_connections 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.231+0000 7f7e56951700 1 --2- 172.21.15.60:0/700199647 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 0x7f7e50106b70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.231+0000 7f7e56951700 1 -- 172.21.15.60:0/700199647 >> 172.21.15.60:0/700199647 conn(0x7f7e50101dd0 msgr2=0x7f7e501041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.231+0000 7f7e56951700 1 -- 172.21.15.60:0/700199647 shutdown_connections 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.231+0000 7f7e56951700 1 -- 172.21.15.60:0/700199647 wait complete. 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.232+0000 7f7e56951700 1 Processor -- start 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.232+0000 7f7e56951700 1 -- start start 2024-03-04T12:15:32.092 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.232+0000 7f7e56951700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 0x7f7e5019b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.232+0000 7f7e56951700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5010ad00 con 0x7f7e50106750 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.232+0000 7f7e4ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 0x7f7e5019b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.232+0000 7f7e4ffff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 0x7f7e5019b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40428/0 (socket says 172.21.15.60:40428) 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.232+0000 7f7e4ffff700 1 -- 172.21.15.60:0/202415200 learned_addr learned my addr 172.21.15.60:0/202415200 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.232+0000 7f7e4ffff700 1 -- 172.21.15.60:0/202415200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e5019bdd0 con 0x7f7e50106750 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.233+0000 7f7e4ffff700 1 --2- 172.21.15.60:0/202415200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 0x7f7e5019b890 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7e40009fb0 tx=0x7f7e40011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.233+0000 7f7e4d7fa700 1 -- 172.21.15.60:0/202415200 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e400038b0 con 0x7f7e50106750 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.233+0000 7f7e56951700 1 -- 172.21.15.60:0/202415200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5019c060 con 0x7f7e50106750 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.233+0000 7f7e4d7fa700 1 -- 172.21.15.60:0/202415200 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e40011e50 con 0x7f7e50106750 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.233+0000 7f7e4d7fa700 1 -- 172.21.15.60:0/202415200 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e40019550 con 0x7f7e50106750 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.233+0000 7f7e56951700 1 -- 172.21.15.60:0/202415200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5019ed50 con 0x7f7e50106750 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.234+0000 7f7e56951700 1 -- 172.21.15.60:0/202415200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e50068f00 con 0x7f7e50106750 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.234+0000 7f7e4d7fa700 1 -- 172.21.15.60:0/202415200 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 45927+0+0 (secure 0 0 0) 0x7f7e40019800 con 0x7f7e50106750 2024-03-04T12:15:32.093 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.234+0000 7f7e4d7fa700 1 --2- 172.21.15.60:0/202415200 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f7e38038ba0 0x7f7e3803b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.234+0000 7f7e4d7fa700 1 -- 172.21.15.60:0/202415200 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7e4004cc60 con 0x7f7e50106750 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.234+0000 7f7e4f7fe700 1 -- 172.21.15.60:0/202415200 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f7e38038ba0 msgr2=0x7f7e3803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2061680026 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.234+0000 7f7e4f7fe700 1 --2- 172.21.15.60:0/202415200 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f7e38038ba0 0x7f7e3803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.239+0000 7f7e4d7fa700 1 -- 172.21.15.60:0/202415200 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7e4004c680 con 0x7f7e50106750 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.435+0000 7f7e4f7fe700 1 -- 172.21.15.60:0/202415200 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f7e38038ba0 msgr2=0x7f7e3803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2061680026 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.435+0000 7f7e4f7fe700 1 --2- 172.21.15.60:0/202415200 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f7e38038ba0 0x7f7e3803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.498+0000 7f7e56951700 1 -- 172.21.15.60:0/202415200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7e5002dae0 con 0x7f7e50106750 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.498+0000 7f7e4d7fa700 1 -- 172.21.15.60:0/202415200 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f7e4002d960 con 0x7f7e50106750 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.500+0000 7f7e56951700 1 -- 172.21.15.60:0/202415200 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f7e38038ba0 msgr2=0x7f7e3803b060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.500+0000 7f7e56951700 1 --2- 172.21.15.60:0/202415200 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f7e38038ba0 0x7f7e3803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.500+0000 7f7e56951700 1 -- 172.21.15.60:0/202415200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 msgr2=0x7f7e5019b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.500+0000 7f7e56951700 1 --2- 172.21.15.60:0/202415200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 0x7f7e5019b890 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7e40009fb0 tx=0x7f7e40011bf0 comp rx=0 tx=0).stop 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.500+0000 7f7e56951700 1 -- 172.21.15.60:0/202415200 shutdown_connections 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.501+0000 7f7e56951700 1 --2- 172.21.15.60:0/202415200 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7f7e38038ba0 0x7f7e3803b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:32.094 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.501+0000 7f7e56951700 1 --2- 172.21.15.60:0/202415200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e50106750 0x7f7e5019b890 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:32.095 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.501+0000 7f7e56951700 1 -- 172.21.15.60:0/202415200 >> 172.21.15.60:0/202415200 conn(0x7f7e50101dd0 msgr2=0x7f7e50102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:32.095 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.501+0000 7f7e56951700 1 -- 172.21.15.60:0/202415200 shutdown_connections 2024-03-04T12:15:32.095 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:31.501+0000 7f7e56951700 1 -- 172.21.15.60:0/202415200 wait complete. 2024-03-04T12:15:32.095 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for the mgr to restart... 2024-03-04T12:15:32.095 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for mgr epoch 5... 2024-03-04T12:15:32.340 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:32 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/202415200' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T12:15:32.641 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:32 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:32.337+0000 7f26e5226200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T12:15:32.641 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:32 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:32.451+0000 7f26e5226200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T12:15:32.642 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:32 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:32.542+0000 7f26e5226200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T12:15:32.642 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:32 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:32.638+0000 7f26e5226200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T12:15:32.925 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:32 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:32.816+0000 7f26e5226200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T12:15:34.175 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:33 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:33.800+0000 7f26e5226200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T12:15:34.175 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:33 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:33.934+0000 7f26e5226200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T12:15:34.564 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:34 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:34.271+0000 7f26e5226200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T12:15:34.564 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:34 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:34.363+0000 7f26e5226200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T12:15:34.564 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:34 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:34.561+0000 7f26e5226200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:15:34.925 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:34 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:34.749+0000 7f26e5226200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T12:15:35.305 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:35 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:35.040+0000 7f26e5226200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:15:35.305 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:35 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:35.302+0000 7f26e5226200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T12:15:35.675 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:35 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:35.545+0000 7f26e5226200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T12:15:36.675 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:36 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:36.366+0000 7f26e5226200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T12:15:37.272 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:36 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:36.948+0000 7f26e5226200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T12:15:37.272 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:37.047+0000 7f26e5226200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T12:15:37.272 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:37.268+0000 7f26e5226200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T12:15:37.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: Active manager daemon x restarted 2024-03-04T12:15:37.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: Activating manager daemon x 2024-03-04T12:15:37.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: osdmap e2: 0 total, 0 up, 0 in 2024-03-04T12:15:37.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: mgrmap e6: x(active, starting, since 0.0035617s) 2024-03-04T12:15:37.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:15:37.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T12:15:37.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T12:15:37.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T12:15:37.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T12:15:37.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: Manager daemon x is now available 2024-03-04T12:15:37.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:37.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:37.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:37 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:15:38.602 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:38 smithi060 ceph-mon[32745]: Found migration_current of "None". Setting to last migration. 2024-03-04T12:15:38.602 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:38 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:15:38.602 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:38 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:15:38.602 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:38 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T12:15:38.602 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:38 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T12:15:38.602 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:38 smithi060 ceph-mon[32745]: mgrmap e7: x(active, since 1.00619s) 2024-03-04T12:15:38.846 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2024-03-04T12:15:38.846 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-04T12:15:38.846 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-04T12:15:38.846 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2024-03-04T12:15:38.846 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.496+0000 7fa77203f700 1 Processor -- start 2024-03-04T12:15:38.846 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.496+0000 7fa77203f700 1 -- start start 2024-03-04T12:15:38.846 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.496+0000 7fa77203f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 0x7fa76c1064c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:38.846 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.496+0000 7fa77203f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c07ae30 con 0x7fa76c1060a0 2024-03-04T12:15:38.846 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.496+0000 7fa76b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 0x7fa76c1064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:38.846 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.496+0000 7fa76b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 0x7fa76c1064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40444/0 (socket says 172.21.15.60:40444) 2024-03-04T12:15:38.846 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.496+0000 7fa76b7fe700 1 -- 172.21.15.60:0/1660110200 learned_addr learned my addr 172.21.15.60:0/1660110200 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:38.846 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.497+0000 7fa76b7fe700 1 -- 172.21.15.60:0/1660110200 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c106a00 con 0x7fa76c1060a0 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.497+0000 7fa76b7fe700 1 --2- 172.21.15.60:0/1660110200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 0x7fa76c1064c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa7540098a0 tx=0x7fa754009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93e51f8cff8961f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.497+0000 7fa76a7fc700 1 -- 172.21.15.60:0/1660110200 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa754004160 con 0x7fa76c1060a0 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.497+0000 7fa76a7fc700 1 -- 172.21.15.60:0/1660110200 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa754004300 con 0x7fa76c1060a0 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.497+0000 7fa76a7fc700 1 -- 172.21.15.60:0/1660110200 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa754011b50 con 0x7fa76c1060a0 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.498+0000 7fa77203f700 1 -- 172.21.15.60:0/1660110200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 msgr2=0x7fa76c1064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.498+0000 7fa77203f700 1 --2- 172.21.15.60:0/1660110200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 0x7fa76c1064c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa7540098a0 tx=0x7fa754009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.498+0000 7fa77203f700 1 -- 172.21.15.60:0/1660110200 shutdown_connections 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.498+0000 7fa77203f700 1 --2- 172.21.15.60:0/1660110200 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 0x7fa76c1064c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.498+0000 7fa77203f700 1 -- 172.21.15.60:0/1660110200 >> 172.21.15.60:0/1660110200 conn(0x7fa76c101d40 msgr2=0x7fa76c104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.498+0000 7fa77203f700 1 -- 172.21.15.60:0/1660110200 shutdown_connections 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.498+0000 7fa77203f700 1 -- 172.21.15.60:0/1660110200 wait complete. 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.499+0000 7fa77203f700 1 Processor -- start 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.499+0000 7fa77203f700 1 -- start start 2024-03-04T12:15:38.847 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.499+0000 7fa77203f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 0x7fa76c19d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.499+0000 7fa77203f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c107a90 con 0x7fa76c1060a0 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.499+0000 7fa76b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 0x7fa76c19d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.499+0000 7fa76b7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 0x7fa76c19d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40454/0 (socket says 172.21.15.60:40454) 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.499+0000 7fa76b7fe700 1 -- 172.21.15.60:0/2086895716 learned_addr learned my addr 172.21.15.60:0/2086895716 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.499+0000 7fa76b7fe700 1 -- 172.21.15.60:0/2086895716 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c19dea0 con 0x7fa76c1060a0 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.500+0000 7fa76b7fe700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 0x7fa76c19d960 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa754011670 tx=0x7fa754011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.500+0000 7fa768ff9700 1 -- 172.21.15.60:0/2086895716 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7540039c0 con 0x7fa76c1060a0 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.500+0000 7fa768ff9700 1 -- 172.21.15.60:0/2086895716 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa754011e50 con 0x7fa76c1060a0 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.500+0000 7fa77203f700 1 -- 172.21.15.60:0/2086895716 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76c19e130 con 0x7fa76c1060a0 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.500+0000 7fa768ff9700 1 -- 172.21.15.60:0/2086895716 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa75401a5c0 con 0x7fa76c1060a0 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.500+0000 7fa77203f700 1 -- 172.21.15.60:0/2086895716 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76c1a0e20 con 0x7fa76c1060a0 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.501+0000 7fa768ff9700 1 -- 172.21.15.60:0/2086895716 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 5) v1 ==== 45927+0+0 (secure 0 0 0) 0x7fa75401a870 con 0x7fa76c1060a0 2024-03-04T12:15:38.848 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.501+0000 7fa768ff9700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 0x7fa75803b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.501+0000 7fa77203f700 1 -- 172.21.15.60:0/2086895716 --> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa76c068f00 con 0x7fa758038ba0 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.501+0000 7fa768ff9700 1 -- 172.21.15.60:0/2086895716 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa75404dce0 con 0x7fa76c1060a0 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.501+0000 7fa76affd700 1 -- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 msgr2=0x7fa75803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2061680026 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.501+0000 7fa76affd700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 0x7fa75803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.702+0000 7fa76affd700 1 -- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 msgr2=0x7fa75803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2061680026 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:32.702+0000 7fa76affd700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 0x7fa75803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:33.103+0000 7fa76affd700 1 -- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 msgr2=0x7fa75803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2061680026 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:33.103+0000 7fa76affd700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 0x7fa75803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:33.904+0000 7fa76affd700 1 -- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 msgr2=0x7fa75803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2061680026 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:33.904+0000 7fa76affd700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 0x7fa75803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:35.505+0000 7fa76affd700 1 -- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 msgr2=0x7fa75803b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2061680026 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:35.505+0000 7fa76affd700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 0x7fa75803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:37.283+0000 7fa768ff9700 1 -- 172.21.15.60:0/2086895716 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mgrmap(e 6) v1 ==== 45694+0+0 (secure 0 0 0) 0x7fa75402da00 con 0x7fa76c1060a0 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:37.283+0000 7fa768ff9700 1 -- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 msgr2=0x7fa75803b060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T12:15:38.849 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:37.283+0000 7fa768ff9700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 0x7fa75803b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.285+0000 7fa768ff9700 1 -- 172.21.15.60:0/2086895716 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 7) v1 ==== 45821+0+0 (secure 0 0 0) 0x7fa75404eae0 con 0x7fa76c1060a0 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.285+0000 7fa768ff9700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fa75803c1f0 0x7fa75803e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.285+0000 7fa768ff9700 1 -- 172.21.15.60:0/2086895716 --> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa75404d3f0 con 0x7fa75803c1f0 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.285+0000 7fa76affd700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fa75803c1f0 0x7fa75803e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.286+0000 7fa76affd700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fa75803c1f0 0x7fa75803e5e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa75c003a10 tx=0x7fa75c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.286+0000 7fa768ff9700 1 -- 172.21.15.60:0/2086895716 <== mgr.14120 v2:172.21.15.60:6800/2246601804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fa75404d3f0 con 0x7fa75803c1f0 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.293+0000 7fa77203f700 1 -- 172.21.15.60:0/2086895716 --> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa76c062260 con 0x7fa75803c1f0 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.293+0000 7fa768ff9700 1 -- 172.21.15.60:0/2086895716 <== mgr.14120 v2:172.21.15.60:6800/2246601804 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa76c062260 con 0x7fa75803c1f0 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.293+0000 7fa77203f700 1 -- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fa75803c1f0 msgr2=0x7fa75803e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.293+0000 7fa77203f700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fa75803c1f0 0x7fa75803e5e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa75c003a10 tx=0x7fa75c0092b0 comp rx=0 tx=0).stop 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.293+0000 7fa77203f700 1 -- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 msgr2=0x7fa76c19d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.293+0000 7fa77203f700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 0x7fa76c19d960 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa754011670 tx=0x7fa754011b60 comp rx=0 tx=0).stop 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.293+0000 7fa77203f700 1 -- 172.21.15.60:0/2086895716 shutdown_connections 2024-03-04T12:15:38.850 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.293+0000 7fa77203f700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fa75803c1f0 0x7fa75803e5e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:38.851 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.294+0000 7fa77203f700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c1060a0 0x7fa76c19d960 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:38.851 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.294+0000 7fa77203f700 1 --2- 172.21.15.60:0/2086895716 >> [v2:172.21.15.60:6800/2061680026,v1:172.21.15.60:6801/2061680026] conn(0x7fa758038ba0 0x7fa75803b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:38.851 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.294+0000 7fa77203f700 1 -- 172.21.15.60:0/2086895716 >> 172.21.15.60:0/2086895716 conn(0x7fa76c101d40 msgr2=0x7fa76c194600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:38.851 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.294+0000 7fa77203f700 1 -- 172.21.15.60:0/2086895716 shutdown_connections 2024-03-04T12:15:38.851 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:38.294+0000 7fa77203f700 1 -- 172.21.15.60:0/2086895716 wait complete. 2024-03-04T12:15:38.851 INFO:teuthology.orchestra.run.smithi060.stdout:mgr epoch 5 is available 2024-03-04T12:15:38.851 INFO:teuthology.orchestra.run.smithi060.stdout:Setting orchestrator backend to cephadm... 2024-03-04T12:15:39.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:39 smithi060 ceph-mon[32745]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T12:15:39.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:39 smithi060 ceph-mon[32745]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-04T12:15:39.984 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.225+0000 7f7a4dec6700 1 Processor -- start 2024-03-04T12:15:39.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.225+0000 7f7a4dec6700 1 -- start start 2024-03-04T12:15:39.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.225+0000 7f7a4dec6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 0x7f7a4810a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:39.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.225+0000 7f7a4dec6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4807ae70 con 0x7f7a48108240 2024-03-04T12:15:39.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.226+0000 7f7a477fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 0x7f7a4810a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:39.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.226+0000 7f7a477fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 0x7f7a4810a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52156/0 (socket says 172.21.15.60:52156) 2024-03-04T12:15:39.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.226+0000 7f7a477fe700 1 -- 172.21.15.60:0/1595595967 learned_addr learned my addr 172.21.15.60:0/1595595967 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:39.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.226+0000 7f7a477fe700 1 -- 172.21.15.60:0/1595595967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4810ab70 con 0x7f7a48108240 2024-03-04T12:15:39.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.226+0000 7f7a477fe700 1 --2- 172.21.15.60:0/1595595967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 0x7f7a4810a630 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7a380098a0 tx=0x7f7a38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70583a6a1f66dc1f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:39.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.226+0000 7f7a467fc700 1 -- 172.21.15.60:0/1595595967 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a38004160 con 0x7f7a48108240 2024-03-04T12:15:39.985 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.226+0000 7f7a467fc700 1 -- 172.21.15.60:0/1595595967 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a38004300 con 0x7f7a48108240 2024-03-04T12:15:39.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.226+0000 7f7a467fc700 1 -- 172.21.15.60:0/1595595967 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a38011b50 con 0x7f7a48108240 2024-03-04T12:15:39.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.227+0000 7f7a4dec6700 1 -- 172.21.15.60:0/1595595967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 msgr2=0x7f7a4810a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:39.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.227+0000 7f7a4dec6700 1 --2- 172.21.15.60:0/1595595967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 0x7f7a4810a630 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7a380098a0 tx=0x7f7a38009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:39.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.227+0000 7f7a4dec6700 1 -- 172.21.15.60:0/1595595967 shutdown_connections 2024-03-04T12:15:39.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.227+0000 7f7a4dec6700 1 --2- 172.21.15.60:0/1595595967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 0x7f7a4810a630 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:39.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.227+0000 7f7a4dec6700 1 -- 172.21.15.60:0/1595595967 >> 172.21.15.60:0/1595595967 conn(0x7f7a48101cd0 msgr2=0x7f7a481040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:39.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.227+0000 7f7a4dec6700 1 -- 172.21.15.60:0/1595595967 shutdown_connections 2024-03-04T12:15:39.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.227+0000 7f7a4dec6700 1 -- 172.21.15.60:0/1595595967 wait complete. 2024-03-04T12:15:39.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.228+0000 7f7a4dec6700 1 Processor -- start 2024-03-04T12:15:39.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.228+0000 7f7a4dec6700 1 -- start start 2024-03-04T12:15:39.986 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.228+0000 7f7a4dec6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 0x7f7a48107ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.228+0000 7f7a4dec6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4810bb20 con 0x7f7a48108240 2024-03-04T12:15:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.228+0000 7f7a477fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 0x7f7a48107ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.228+0000 7f7a477fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 0x7f7a48107ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52162/0 (socket says 172.21.15.60:52162) 2024-03-04T12:15:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.228+0000 7f7a477fe700 1 -- 172.21.15.60:0/867692427 learned_addr learned my addr 172.21.15.60:0/867692427 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.228+0000 7f7a477fe700 1 -- 172.21.15.60:0/867692427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48108000 con 0x7f7a48108240 2024-03-04T12:15:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.229+0000 7f7a477fe700 1 --2- 172.21.15.60:0/867692427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 0x7f7a48107ac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009fb0 tx=0x7f7a38011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.229+0000 7f7a44ff9700 1 -- 172.21.15.60:0/867692427 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a380038b0 con 0x7f7a48108240 2024-03-04T12:15:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.229+0000 7f7a4dec6700 1 -- 172.21.15.60:0/867692427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a481061f0 con 0x7f7a48108240 2024-03-04T12:15:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.229+0000 7f7a4dec6700 1 -- 172.21.15.60:0/867692427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a48106730 con 0x7f7a48108240 2024-03-04T12:15:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.229+0000 7f7a44ff9700 1 -- 172.21.15.60:0/867692427 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a38011e50 con 0x7f7a48108240 2024-03-04T12:15:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.230+0000 7f7a44ff9700 1 -- 172.21.15.60:0/867692427 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a38019420 con 0x7f7a48108240 2024-03-04T12:15:39.987 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.230+0000 7f7a44ff9700 1 -- 172.21.15.60:0/867692427 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 7) v1 ==== 45821+0+0 (secure 0 0 0) 0x7f7a380197a0 con 0x7f7a48108240 2024-03-04T12:15:39.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.230+0000 7f7a44ff9700 1 --2- 172.21.15.60:0/867692427 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f7a34038a40 0x7f7a3403af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:39.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.230+0000 7f7a44ff9700 1 -- 172.21.15.60:0/867692427 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7a3804cb40 con 0x7f7a48108240 2024-03-04T12:15:39.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.230+0000 7f7a46ffd700 1 --2- 172.21.15.60:0/867692427 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f7a34038a40 0x7f7a3403af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:39.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.230+0000 7f7a4dec6700 1 -- 172.21.15.60:0/867692427 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a28005360 con 0x7f7a48108240 2024-03-04T12:15:39.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.231+0000 7f7a46ffd700 1 --2- 172.21.15.60:0/867692427 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f7a34038a40 0x7f7a3403af00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7a30006fd0 tx=0x7f7a30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:39.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.236+0000 7f7a44ff9700 1 -- 172.21.15.60:0/867692427 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7a3800f2e0 con 0x7f7a48108240 2024-03-04T12:15:39.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.335+0000 7f7a44ff9700 1 -- 172.21.15.60:0/867692427 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 8) v1 ==== 45927+0+0 (secure 0 0 0) 0x7f7a38003a50 con 0x7f7a48108240 2024-03-04T12:15:39.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.454+0000 7f7a4dec6700 1 -- 172.21.15.60:0/867692427 --> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a28000bf0 con 0x7f7a34038a40 2024-03-04T12:15:39.988 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.458+0000 7f7a44ff9700 1 -- 172.21.15.60:0/867692427 <== mgr.14120 v2:172.21.15.60:6800/2246601804 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7a28000bf0 con 0x7f7a34038a40 2024-03-04T12:15:39.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.460+0000 7f7a4dec6700 1 -- 172.21.15.60:0/867692427 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f7a34038a40 msgr2=0x7f7a3403af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:39.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.460+0000 7f7a4dec6700 1 --2- 172.21.15.60:0/867692427 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f7a34038a40 0x7f7a3403af00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7a30006fd0 tx=0x7f7a30006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:39.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.460+0000 7f7a4dec6700 1 -- 172.21.15.60:0/867692427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 msgr2=0x7f7a48107ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:39.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.460+0000 7f7a4dec6700 1 --2- 172.21.15.60:0/867692427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 0x7f7a48107ac0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009fb0 tx=0x7f7a38011bf0 comp rx=0 tx=0).stop 2024-03-04T12:15:39.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.461+0000 7f7a4dec6700 1 -- 172.21.15.60:0/867692427 shutdown_connections 2024-03-04T12:15:39.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.461+0000 7f7a4dec6700 1 --2- 172.21.15.60:0/867692427 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f7a34038a40 0x7f7a3403af00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:39.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.461+0000 7f7a4dec6700 1 --2- 172.21.15.60:0/867692427 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7a48108240 0x7f7a48107ac0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:39.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.461+0000 7f7a4dec6700 1 -- 172.21.15.60:0/867692427 >> 172.21.15.60:0/867692427 conn(0x7f7a48101cd0 msgr2=0x7f7a48102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:39.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.461+0000 7f7a4dec6700 1 -- 172.21.15.60:0/867692427 shutdown_connections 2024-03-04T12:15:39.989 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:39.461+0000 7f7a4dec6700 1 -- 172.21.15.60:0/867692427 wait complete. 2024-03-04T12:15:40.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:40 smithi060 ceph-mon[32745]: mgrmap e8: x(active, since 2s) 2024-03-04T12:15:40.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:40 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:40.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:40 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:15:40.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:40 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:40.370 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:40 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:41.224 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-04T12:15:41.224 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.408+0000 7f9eed618700 1 Processor -- start 2024-03-04T12:15:41.224 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.408+0000 7f9eed618700 1 -- start start 2024-03-04T12:15:41.224 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.408+0000 7f9eed618700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 0x7f9ee8106af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:41.224 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.408+0000 7f9eed618700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee81070c0 con 0x7f9ee81066d0 2024-03-04T12:15:41.225 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.408+0000 7f9ee6ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 0x7f9ee8106af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:41.225 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.408+0000 7f9ee6ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 0x7f9ee8106af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52166/0 (socket says 172.21.15.60:52166) 2024-03-04T12:15:41.225 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.408+0000 7f9ee6ffd700 1 -- 172.21.15.60:0/2192669702 learned_addr learned my addr 172.21.15.60:0/2192669702 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:41.225 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.408+0000 7f9ee6ffd700 1 -- 172.21.15.60:0/2192669702 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee81078d0 con 0x7f9ee81066d0 2024-03-04T12:15:41.226 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.409+0000 7f9ee6ffd700 1 --2- 172.21.15.60:0/2192669702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 0x7f9ee8106af0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800bb40 tx=0x7f9ed800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03b2e9339f37173 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:41.226 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.409+0000 7f9ee5ffb700 1 -- 172.21.15.60:0/2192669702 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ed800dbe0 con 0x7f9ee81066d0 2024-03-04T12:15:41.226 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.409+0000 7f9ee5ffb700 1 -- 172.21.15.60:0/2192669702 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ed80044f0 con 0x7f9ee81066d0 2024-03-04T12:15:41.226 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.409+0000 7f9ee5ffb700 1 -- 172.21.15.60:0/2192669702 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ed8010470 con 0x7f9ee81066d0 2024-03-04T12:15:41.227 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.410+0000 7f9eed618700 1 -- 172.21.15.60:0/2192669702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 msgr2=0x7f9ee8106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:41.227 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.410+0000 7f9eed618700 1 --2- 172.21.15.60:0/2192669702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 0x7f9ee8106af0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800bb40 tx=0x7f9ed800be50 comp rx=0 tx=0).stop 2024-03-04T12:15:41.227 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.410+0000 7f9eed618700 1 -- 172.21.15.60:0/2192669702 shutdown_connections 2024-03-04T12:15:41.227 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.410+0000 7f9eed618700 1 --2- 172.21.15.60:0/2192669702 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 0x7f9ee8106af0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:41.227 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.410+0000 7f9eed618700 1 -- 172.21.15.60:0/2192669702 >> 172.21.15.60:0/2192669702 conn(0x7f9ee8101d50 msgr2=0x7f9ee8104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:41.227 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.410+0000 7f9eed618700 1 -- 172.21.15.60:0/2192669702 shutdown_connections 2024-03-04T12:15:41.227 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.410+0000 7f9eed618700 1 -- 172.21.15.60:0/2192669702 wait complete. 2024-03-04T12:15:41.227 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.410+0000 7f9eed618700 1 Processor -- start 2024-03-04T12:15:41.227 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.410+0000 7f9eed618700 1 -- start start 2024-03-04T12:15:41.227 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.411+0000 7f9eed618700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 0x7f9ee81995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:41.227 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.411+0000 7f9eed618700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee8079fb0 con 0x7f9ee81066d0 2024-03-04T12:15:41.228 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.411+0000 7f9ee6ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 0x7f9ee81995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:41.228 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.411+0000 7f9ee6ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 0x7f9ee81995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52168/0 (socket says 172.21.15.60:52168) 2024-03-04T12:15:41.228 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.411+0000 7f9ee6ffd700 1 -- 172.21.15.60:0/2294635643 learned_addr learned my addr 172.21.15.60:0/2294635643 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:41.228 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.411+0000 7f9ee6ffd700 1 -- 172.21.15.60:0/2294635643 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee8199b10 con 0x7f9ee81066d0 2024-03-04T12:15:41.228 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.411+0000 7f9ee6ffd700 1 --2- 172.21.15.60:0/2294635643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 0x7f9ee81995d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800b340 tx=0x7f9ed80040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:41.228 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.411+0000 7f9ecffff700 1 -- 172.21.15.60:0/2294635643 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ed8027070 con 0x7f9ee81066d0 2024-03-04T12:15:41.228 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.411+0000 7f9ecffff700 1 -- 172.21.15.60:0/2294635643 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ed80104e0 con 0x7f9ee81066d0 2024-03-04T12:15:41.228 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.411+0000 7f9eed618700 1 -- 172.21.15.60:0/2294635643 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee8199da0 con 0x7f9ee81066d0 2024-03-04T12:15:41.228 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.411+0000 7f9eed618700 1 -- 172.21.15.60:0/2294635643 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee819ca90 con 0x7f9ee81066d0 2024-03-04T12:15:41.228 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.411+0000 7f9ecffff700 1 -- 172.21.15.60:0/2294635643 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ed80044a0 con 0x7f9ee81066d0 2024-03-04T12:15:41.228 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.412+0000 7f9ecffff700 1 -- 172.21.15.60:0/2294635643 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45927+0+0 (secure 0 0 0) 0x7f9ed802a070 con 0x7f9ee81066d0 2024-03-04T12:15:41.228 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.412+0000 7f9eed618700 1 -- 172.21.15.60:0/2294635643 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee8068f00 con 0x7f9ee81066d0 2024-03-04T12:15:41.229 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.413+0000 7f9ecffff700 1 --2- 172.21.15.60:0/2294635643 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f9ed0038b50 0x7f9ed003b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:41.229 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.413+0000 7f9ecffff700 1 -- 172.21.15.60:0/2294635643 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9ed801f070 con 0x7f9ee81066d0 2024-03-04T12:15:41.229 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.413+0000 7f9ee67fc700 1 --2- 172.21.15.60:0/2294635643 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f9ed0038b50 0x7f9ed003b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:41.229 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.413+0000 7f9ee67fc700 1 --2- 172.21.15.60:0/2294635643 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f9ed0038b50 0x7f9ed003b010 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9edc006fd0 tx=0x7f9edc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:41.229 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.418+0000 7f9ecffff700 1 -- 172.21.15.60:0/2294635643 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9ed8015070 con 0x7f9ee81066d0 2024-03-04T12:15:41.229 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.620+0000 7f9eed618700 1 -- 172.21.15.60:0/2294635643 --> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ee8062260 con 0x7f9ed0038b50 2024-03-04T12:15:41.229 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.621+0000 7f9ecffff700 1 -- 172.21.15.60:0/2294635643 <== mgr.14120 v2:172.21.15.60:6800/2246601804 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f9ee8062260 con 0x7f9ed0038b50 2024-03-04T12:15:41.229 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.623+0000 7f9eed618700 1 -- 172.21.15.60:0/2294635643 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f9ed0038b50 msgr2=0x7f9ed003b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:41.229 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.623+0000 7f9eed618700 1 --2- 172.21.15.60:0/2294635643 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f9ed0038b50 0x7f9ed003b010 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9edc006fd0 tx=0x7f9edc006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:41.229 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.623+0000 7f9eed618700 1 -- 172.21.15.60:0/2294635643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 msgr2=0x7f9ee81995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:41.229 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.623+0000 7f9eed618700 1 --2- 172.21.15.60:0/2294635643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 0x7f9ee81995d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9ed800b340 tx=0x7f9ed80040b0 comp rx=0 tx=0).stop 2024-03-04T12:15:41.230 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.623+0000 7f9eed618700 1 -- 172.21.15.60:0/2294635643 shutdown_connections 2024-03-04T12:15:41.230 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.623+0000 7f9eed618700 1 --2- 172.21.15.60:0/2294635643 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f9ed0038b50 0x7f9ed003b010 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:41.230 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.623+0000 7f9eed618700 1 --2- 172.21.15.60:0/2294635643 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9ee81066d0 0x7f9ee81995d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:41.230 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.623+0000 7f9eed618700 1 -- 172.21.15.60:0/2294635643 >> 172.21.15.60:0/2294635643 conn(0x7f9ee8101d50 msgr2=0x7f9ee818fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:41.230 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.623+0000 7f9eed618700 1 -- 172.21.15.60:0/2294635643 shutdown_connections 2024-03-04T12:15:41.230 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:40.623+0000 7f9eed618700 1 -- 172.21.15.60:0/2294635643 wait complete. 2024-03-04T12:15:41.230 INFO:teuthology.orchestra.run.smithi060.stdout:Generating ssh key... 2024-03-04T12:15:41.602 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:41 smithi060 ceph-mon[32745]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:15:41.602 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:41 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:15:42.150 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: Generating public/private rsa key pair. 2024-03-04T12:15:42.150 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: Your identification has been saved in /tmp/tmpopm1ga9p/key. 2024-03-04T12:15:42.150 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: Your public key has been saved in /tmp/tmpopm1ga9p/key.pub. 2024-03-04T12:15:42.150 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: The key fingerprint is: 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: SHA256:e7mrPFZ/5WVDXkdU4boIEObXBR/2rtIXvXocvThpkcE ceph-a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: The key's randomart image is: 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: +---[RSA 3072]----+ 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: | o ..+ o=| 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: | o . . + + .| 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: | o . ... + | 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: | o Eo.+| 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: | S. .=o=| 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: | .o.ooo+B| 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: | ..o+ +=*=| 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: | .o. .o=++.| 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: | .ooo..oo | 2024-03-04T12:15:42.151 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: +----[SHA256]-----+ 2024-03-04T12:15:42.422 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-mon[32745]: [04/Mar/2024:12:15:40] ENGINE Bus STARTING 2024-03-04T12:15:42.422 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-mon[32745]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:15:42.422 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-mon[32745]: [04/Mar/2024:12:15:40] ENGINE Serving on http://172.21.15.60:8765 2024-03-04T12:15:42.422 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-mon[32745]: [04/Mar/2024:12:15:40] ENGINE Serving on https://172.21.15.60:7150 2024-03-04T12:15:42.422 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-mon[32745]: [04/Mar/2024:12:15:40] ENGINE Bus STARTED 2024-03-04T12:15:42.422 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:42.422 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:42 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:42.696 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.624+0000 7ff9e582e700 1 Processor -- start 2024-03-04T12:15:42.696 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.624+0000 7ff9e582e700 1 -- start start 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.624+0000 7ff9e582e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 0x7ff9e0106b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.624+0000 7ff9e582e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e0107140 con 0x7ff9e0106750 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.625+0000 7ff9deffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 0x7ff9e0106b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.625+0000 7ff9deffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 0x7ff9e0106b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52170/0 (socket says 172.21.15.60:52170) 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.625+0000 7ff9deffd700 1 -- 172.21.15.60:0/3261090747 learned_addr learned my addr 172.21.15.60:0/3261090747 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.625+0000 7ff9deffd700 1 -- 172.21.15.60:0/3261090747 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e0107950 con 0x7ff9e0106750 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.625+0000 7ff9deffd700 1 --2- 172.21.15.60:0/3261090747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 0x7ff9e0106b70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40098a0 tx=0x7ff9d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5645d15b2a51b60b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.625+0000 7ff9ddffb700 1 -- 172.21.15.60:0/3261090747 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9d4004160 con 0x7ff9e0106750 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.625+0000 7ff9ddffb700 1 -- 172.21.15.60:0/3261090747 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9d4004300 con 0x7ff9e0106750 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.625+0000 7ff9ddffb700 1 -- 172.21.15.60:0/3261090747 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9d4011b50 con 0x7ff9e0106750 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.626+0000 7ff9e582e700 1 -- 172.21.15.60:0/3261090747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 msgr2=0x7ff9e0106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.626+0000 7ff9e582e700 1 --2- 172.21.15.60:0/3261090747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 0x7ff9e0106b70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40098a0 tx=0x7ff9d4009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:42.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.626+0000 7ff9e582e700 1 -- 172.21.15.60:0/3261090747 shutdown_connections 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.626+0000 7ff9e582e700 1 --2- 172.21.15.60:0/3261090747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 0x7ff9e0106b70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.626+0000 7ff9e582e700 1 -- 172.21.15.60:0/3261090747 >> 172.21.15.60:0/3261090747 conn(0x7ff9e0101dd0 msgr2=0x7ff9e01041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.626+0000 7ff9e582e700 1 -- 172.21.15.60:0/3261090747 shutdown_connections 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.626+0000 7ff9e582e700 1 -- 172.21.15.60:0/3261090747 wait complete. 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.627+0000 7ff9e582e700 1 Processor -- start 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.627+0000 7ff9e582e700 1 -- start start 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.627+0000 7ff9e582e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 0x7ff9e019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.627+0000 7ff9e582e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e010ad00 con 0x7ff9e0106750 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.627+0000 7ff9deffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 0x7ff9e019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.627+0000 7ff9deffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 0x7ff9e019a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52186/0 (socket says 172.21.15.60:52186) 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.627+0000 7ff9deffd700 1 -- 172.21.15.60:0/3230870753 learned_addr learned my addr 172.21.15.60:0/3230870753 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.628+0000 7ff9deffd700 1 -- 172.21.15.60:0/3230870753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e019acb0 con 0x7ff9e0106750 2024-03-04T12:15:42.698 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.628+0000 7ff9deffd700 1 --2- 172.21.15.60:0/3230870753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 0x7ff9e019a770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009fb0 tx=0x7ff9d4011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.628+0000 7ff9e482c700 1 -- 172.21.15.60:0/3230870753 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9d40038b0 con 0x7ff9e0106750 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.628+0000 7ff9e482c700 1 -- 172.21.15.60:0/3230870753 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9d4011e50 con 0x7ff9e0106750 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.628+0000 7ff9e582e700 1 -- 172.21.15.60:0/3230870753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e019af40 con 0x7ff9e0106750 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.628+0000 7ff9e482c700 1 -- 172.21.15.60:0/3230870753 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9d4019550 con 0x7ff9e0106750 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.628+0000 7ff9e582e700 1 -- 172.21.15.60:0/3230870753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e019dc30 con 0x7ff9e0106750 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.629+0000 7ff9e482c700 1 -- 172.21.15.60:0/3230870753 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45927+0+0 (secure 0 0 0) 0x7ff9d4019800 con 0x7ff9e0106750 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.629+0000 7ff9e482c700 1 --2- 172.21.15.60:0/3230870753 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7ff9c8038ba0 0x7ff9c803b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.629+0000 7ff9e482c700 1 -- 172.21.15.60:0/3230870753 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff9d404cc60 con 0x7ff9e0106750 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.629+0000 7ff9de7fc700 1 --2- 172.21.15.60:0/3230870753 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7ff9c8038ba0 0x7ff9c803b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.629+0000 7ff9e582e700 1 -- 172.21.15.60:0/3230870753 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e0068f00 con 0x7ff9e0106750 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.630+0000 7ff9de7fc700 1 --2- 172.21.15.60:0/3230870753 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7ff9c8038ba0 0x7ff9c803b060 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0006fd0 tx=0x7ff9d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.634+0000 7ff9e482c700 1 -- 172.21.15.60:0/3230870753 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff9d402d960 con 0x7ff9e0106750 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:41.840+0000 7ff9e582e700 1 -- 172.21.15.60:0/3230870753 --> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9e0103040 con 0x7ff9c8038ba0 2024-03-04T12:15:42.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:42.101+0000 7ff9e482c700 1 -- 172.21.15.60:0/3230870753 <== mgr.14120 v2:172.21.15.60:6800/2246601804 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff9e0103040 con 0x7ff9c8038ba0 2024-03-04T12:15:42.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:42.103+0000 7ff9e582e700 1 -- 172.21.15.60:0/3230870753 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7ff9c8038ba0 msgr2=0x7ff9c803b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:42.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:42.103+0000 7ff9e582e700 1 --2- 172.21.15.60:0/3230870753 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7ff9c8038ba0 0x7ff9c803b060 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0006fd0 tx=0x7ff9d0006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:42.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:42.103+0000 7ff9e582e700 1 -- 172.21.15.60:0/3230870753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 msgr2=0x7ff9e019a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:42.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:42.103+0000 7ff9e582e700 1 --2- 172.21.15.60:0/3230870753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 0x7ff9e019a770 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009fb0 tx=0x7ff9d4011bf0 comp rx=0 tx=0).stop 2024-03-04T12:15:42.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:42.103+0000 7ff9e582e700 1 -- 172.21.15.60:0/3230870753 shutdown_connections 2024-03-04T12:15:42.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:42.103+0000 7ff9e582e700 1 --2- 172.21.15.60:0/3230870753 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7ff9c8038ba0 0x7ff9c803b060 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:42.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:42.103+0000 7ff9e582e700 1 --2- 172.21.15.60:0/3230870753 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0106750 0x7ff9e019a770 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:42.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:42.103+0000 7ff9e582e700 1 -- 172.21.15.60:0/3230870753 >> 172.21.15.60:0/3230870753 conn(0x7ff9e0101dd0 msgr2=0x7ff9e0102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:42.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:42.103+0000 7ff9e582e700 1 -- 172.21.15.60:0/3230870753 shutdown_connections 2024-03-04T12:15:42.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:42.103+0000 7ff9e582e700 1 -- 172.21.15.60:0/3230870753 wait complete. 2024-03-04T12:15:43.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:43 smithi060 ceph-mon[32745]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:15:43.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:43 smithi060 ceph-mon[32745]: Generating ssh key... 2024-03-04T12:15:43.862 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEXcriikNS2A5qq16UStXqFx7CW9PEeMqaT0s/JJq3DhJ6xHRlBIWzQliBPMAertGCATGk56/7z0FUX7qzUcoTpYHRwzdNVhpusBklZu7U+ymqmEFLWbSuqHKAmnt1xo5nk08ebsyLNewlnns5/A8/oO5XQqX9B+DyRjR2jSL3eDKuYgyP/QRuUcV7Wn2aI7dO53Dd+hGwmkp1gTBpv18t+5fd/vI81CJgT/9ry+KbouSVY2QOw7she8cW+VtB6IUOr/TvwgXKJ8sxQAvO7esz5a7lqrW2Pg+IihnLDVvZfIfrP7wkZV65W0neq09carXSb086id9NFJGhdRTVaQhMHDW5EubktMJTlAQSO7dC7PjE8lesIajDGN6ih68LWKCzXoCfJqifpO/UBqRNQ9MURTdDZlVsxTXeSvgdlqU6qCGXb4Mtz1kPrVIEApLmyrVxJAVrROCjUTD8UkxMBiY31SAUGapJPmQ8wNz/DVsUsEU/pJCmOUluVvex+mp9B1M= ceph-a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:15:43.862 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.107+0000 7fbe996b6700 1 Processor -- start 2024-03-04T12:15:43.862 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.107+0000 7fbe996b6700 1 -- start start 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.107+0000 7fbe996b6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 0x7fbe9407b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.107+0000 7fbe996b6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9407b8d0 con 0x7fbe9407ce50 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.107+0000 7fbe92ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 0x7fbe9407b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.107+0000 7fbe92ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 0x7fbe9407b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52202/0 (socket says 172.21.15.60:52202) 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.107+0000 7fbe92ffd700 1 -- 172.21.15.60:0/2448496825 learned_addr learned my addr 172.21.15.60:0/2448496825 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.107+0000 7fbe92ffd700 1 -- 172.21.15.60:0/2448496825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9407ba50 con 0x7fbe9407ce50 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.108+0000 7fbe92ffd700 1 --2- 172.21.15.60:0/2448496825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 0x7fbe9407b300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbe840098a0 tx=0x7fbe84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e24df9d3516ad03 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.108+0000 7fbe91ffb700 1 -- 172.21.15.60:0/2448496825 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe84004160 con 0x7fbe9407ce50 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.108+0000 7fbe91ffb700 1 -- 172.21.15.60:0/2448496825 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe84004300 con 0x7fbe9407ce50 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.108+0000 7fbe91ffb700 1 -- 172.21.15.60:0/2448496825 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe84011b50 con 0x7fbe9407ce50 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.108+0000 7fbe996b6700 1 -- 172.21.15.60:0/2448496825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 msgr2=0x7fbe9407b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.108+0000 7fbe996b6700 1 --2- 172.21.15.60:0/2448496825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 0x7fbe9407b300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbe840098a0 tx=0x7fbe84009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:43.863 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.108+0000 7fbe996b6700 1 -- 172.21.15.60:0/2448496825 shutdown_connections 2024-03-04T12:15:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.108+0000 7fbe996b6700 1 --2- 172.21.15.60:0/2448496825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 0x7fbe9407b300 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.108+0000 7fbe996b6700 1 -- 172.21.15.60:0/2448496825 >> 172.21.15.60:0/2448496825 conn(0x7fbe94102df0 msgr2=0x7fbe94105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.108+0000 7fbe996b6700 1 -- 172.21.15.60:0/2448496825 shutdown_connections 2024-03-04T12:15:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.108+0000 7fbe996b6700 1 -- 172.21.15.60:0/2448496825 wait complete. 2024-03-04T12:15:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.109+0000 7fbe996b6700 1 Processor -- start 2024-03-04T12:15:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.109+0000 7fbe996b6700 1 -- start start 2024-03-04T12:15:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.109+0000 7fbe996b6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 0x7fbe9419d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.109+0000 7fbe996b6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9410dd60 con 0x7fbe9407ce50 2024-03-04T12:15:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.109+0000 7fbe92ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 0x7fbe9419d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.109+0000 7fbe92ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 0x7fbe9419d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52212/0 (socket says 172.21.15.60:52212) 2024-03-04T12:15:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.109+0000 7fbe92ffd700 1 -- 172.21.15.60:0/2176605801 learned_addr learned my addr 172.21.15.60:0/2176605801 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:43.864 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.110+0000 7fbe92ffd700 1 -- 172.21.15.60:0/2176605801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9419dee0 con 0x7fbe9407ce50 2024-03-04T12:15:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.110+0000 7fbe92ffd700 1 --2- 172.21.15.60:0/2176605801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 0x7fbe9419d9a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009fb0 tx=0x7fbe84011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.110+0000 7fbe7bfff700 1 -- 172.21.15.60:0/2176605801 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe840038b0 con 0x7fbe9407ce50 2024-03-04T12:15:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.110+0000 7fbe7bfff700 1 -- 172.21.15.60:0/2176605801 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe84011e50 con 0x7fbe9407ce50 2024-03-04T12:15:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.110+0000 7fbe996b6700 1 -- 172.21.15.60:0/2176605801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9419e170 con 0x7fbe9407ce50 2024-03-04T12:15:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.110+0000 7fbe7bfff700 1 -- 172.21.15.60:0/2176605801 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe84019420 con 0x7fbe9407ce50 2024-03-04T12:15:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.110+0000 7fbe996b6700 1 -- 172.21.15.60:0/2176605801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe941a0e60 con 0x7fbe9407ce50 2024-03-04T12:15:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.111+0000 7fbe7bfff700 1 -- 172.21.15.60:0/2176605801 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45927+0+0 (secure 0 0 0) 0x7fbe84019720 con 0x7fbe9407ce50 2024-03-04T12:15:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.111+0000 7fbe996b6700 1 -- 172.21.15.60:0/2176605801 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe94068f00 con 0x7fbe9407ce50 2024-03-04T12:15:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.111+0000 7fbe7bfff700 1 --2- 172.21.15.60:0/2176605801 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fbe7c038b50 0x7fbe7c03b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.111+0000 7fbe7bfff700 1 -- 172.21.15.60:0/2176605801 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbe8404ce30 con 0x7fbe9407ce50 2024-03-04T12:15:43.865 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.111+0000 7fbe927fc700 1 --2- 172.21.15.60:0/2176605801 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fbe7c038b50 0x7fbe7c03b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.111+0000 7fbe927fc700 1 --2- 172.21.15.60:0/2176605801 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fbe7c038b50 0x7fbe7c03b010 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbe88006fd0 tx=0x7fbe88006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.116+0000 7fbe7bfff700 1 -- 172.21.15.60:0/2176605801 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbe8400bc70 con 0x7fbe9407ce50 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.321+0000 7fbe996b6700 1 -- 172.21.15.60:0/2176605801 --> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe94104090 con 0x7fbe7c038b50 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.321+0000 7fbe7bfff700 1 -- 172.21.15.60:0/2176605801 <== mgr.14120 v2:172.21.15.60:6800/2246601804 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fbe94104090 con 0x7fbe7c038b50 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.324+0000 7fbe996b6700 1 -- 172.21.15.60:0/2176605801 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fbe7c038b50 msgr2=0x7fbe7c03b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.324+0000 7fbe996b6700 1 --2- 172.21.15.60:0/2176605801 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fbe7c038b50 0x7fbe7c03b010 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbe88006fd0 tx=0x7fbe88006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.324+0000 7fbe996b6700 1 -- 172.21.15.60:0/2176605801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 msgr2=0x7fbe9419d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.324+0000 7fbe996b6700 1 --2- 172.21.15.60:0/2176605801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 0x7fbe9419d9a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009fb0 tx=0x7fbe84011bf0 comp rx=0 tx=0).stop 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.324+0000 7fbe996b6700 1 -- 172.21.15.60:0/2176605801 shutdown_connections 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.324+0000 7fbe996b6700 1 --2- 172.21.15.60:0/2176605801 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fbe7c038b50 0x7fbe7c03b010 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.324+0000 7fbe996b6700 1 --2- 172.21.15.60:0/2176605801 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9407ce50 0x7fbe9419d9a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.324+0000 7fbe996b6700 1 -- 172.21.15.60:0/2176605801 >> 172.21.15.60:0/2176605801 conn(0x7fbe94102df0 msgr2=0x7fbe94103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.324+0000 7fbe996b6700 1 -- 172.21.15.60:0/2176605801 shutdown_connections 2024-03-04T12:15:43.866 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:43.324+0000 7fbe996b6700 1 -- 172.21.15.60:0/2176605801 wait complete. 2024-03-04T12:15:43.867 INFO:teuthology.orchestra.run.smithi060.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-04T12:15:43.867 INFO:teuthology.orchestra.run.smithi060.stdout:Adding key to root@localhost authorized_keys... 2024-03-04T12:15:43.867 INFO:teuthology.orchestra.run.smithi060.stdout:Adding host smithi060... 2024-03-04T12:15:44.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:44 smithi060 ceph-mon[32745]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:15:45.553 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:45 smithi060 ceph-mon[32745]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi060", "addr": "172.21.15.60", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:15:45.553 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:45 smithi060 ceph-mon[32745]: Deploying cephadm binary to smithi060 2024-03-04T12:15:49.470 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout Added host 'smithi060' with addr '172.21.15.60' 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.260+0000 7fb45eddd700 1 Processor -- start 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.260+0000 7fb45eddd700 1 -- start start 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.260+0000 7fb45eddd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 0x7fb458107c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.260+0000 7fb45eddd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb458108250 con 0x7fb458107860 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.260+0000 7fb45cb79700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 0x7fb458107c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.260+0000 7fb45cb79700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 0x7fb458107c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52222/0 (socket says 172.21.15.60:52222) 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.260+0000 7fb45cb79700 1 -- 172.21.15.60:0/3041071293 learned_addr learned my addr 172.21.15.60:0/3041071293 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.261+0000 7fb45cb79700 1 -- 172.21.15.60:0/3041071293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb458108a10 con 0x7fb458107860 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.261+0000 7fb45cb79700 1 --2- 172.21.15.60:0/3041071293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 0x7fb458107c80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb44c0098a0 tx=0x7fb44c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3543004eaeb3595 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.261+0000 7fb4577fe700 1 -- 172.21.15.60:0/3041071293 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb44c004160 con 0x7fb458107860 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.261+0000 7fb4577fe700 1 -- 172.21.15.60:0/3041071293 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb44c004300 con 0x7fb458107860 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.261+0000 7fb4577fe700 1 -- 172.21.15.60:0/3041071293 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb44c011b50 con 0x7fb458107860 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.261+0000 7fb45eddd700 1 -- 172.21.15.60:0/3041071293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 msgr2=0x7fb458107c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:49.471 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.261+0000 7fb45eddd700 1 --2- 172.21.15.60:0/3041071293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 0x7fb458107c80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb44c0098a0 tx=0x7fb44c009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.262+0000 7fb45eddd700 1 -- 172.21.15.60:0/3041071293 shutdown_connections 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.262+0000 7fb45eddd700 1 --2- 172.21.15.60:0/3041071293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 0x7fb458107c80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.262+0000 7fb45eddd700 1 -- 172.21.15.60:0/3041071293 >> 172.21.15.60:0/3041071293 conn(0x7fb458102ee0 msgr2=0x7fb458105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.262+0000 7fb45eddd700 1 -- 172.21.15.60:0/3041071293 shutdown_connections 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.262+0000 7fb45eddd700 1 -- 172.21.15.60:0/3041071293 wait complete. 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.262+0000 7fb45eddd700 1 Processor -- start 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.262+0000 7fb45eddd700 1 -- start start 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.262+0000 7fb45eddd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 0x7fb45819b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.262+0000 7fb45eddd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45810bdc0 con 0x7fb458107860 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.262+0000 7fb45cb79700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 0x7fb45819b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.262+0000 7fb45cb79700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 0x7fb45819b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52236/0 (socket says 172.21.15.60:52236) 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.262+0000 7fb45cb79700 1 -- 172.21.15.60:0/2454380549 learned_addr learned my addr 172.21.15.60:0/2454380549 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:49.472 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.263+0000 7fb45cb79700 1 -- 172.21.15.60:0/2454380549 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45819bcb0 con 0x7fb458107860 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.263+0000 7fb45cb79700 1 --2- 172.21.15.60:0/2454380549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 0x7fb45819b770 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb44c000c00 tx=0x7fb44c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.263+0000 7fb455ffb700 1 -- 172.21.15.60:0/2454380549 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb44c003b20 con 0x7fb458107860 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.263+0000 7fb455ffb700 1 -- 172.21.15.60:0/2454380549 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb44c003cc0 con 0x7fb458107860 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.263+0000 7fb45eddd700 1 -- 172.21.15.60:0/2454380549 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45819bf40 con 0x7fb458107860 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.263+0000 7fb455ffb700 1 -- 172.21.15.60:0/2454380549 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb44c024910 con 0x7fb458107860 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.263+0000 7fb45eddd700 1 -- 172.21.15.60:0/2454380549 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45819ec30 con 0x7fb458107860 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.264+0000 7fb455ffb700 1 -- 172.21.15.60:0/2454380549 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45927+0+0 (secure 0 0 0) 0x7fb44c022020 con 0x7fb458107860 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.264+0000 7fb455ffb700 1 --2- 172.21.15.60:0/2454380549 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fb448038bf0 0x7fb44803b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.264+0000 7fb455ffb700 1 -- 172.21.15.60:0/2454380549 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb44c011e00 con 0x7fb458107860 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.264+0000 7fb457fff700 1 --2- 172.21.15.60:0/2454380549 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fb448038bf0 0x7fb44803b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.264+0000 7fb45eddd700 1 -- 172.21.15.60:0/2454380549 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb458063450 con 0x7fb458107860 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.265+0000 7fb457fff700 1 --2- 172.21.15.60:0/2454380549 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fb448038bf0 0x7fb44803b0b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb444006fd0 tx=0x7fb444006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.269+0000 7fb455ffb700 1 -- 172.21.15.60:0/2454380549 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb44c0172b0 con 0x7fb458107860 2024-03-04T12:15:49.473 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:44.476+0000 7fb45eddd700 1 -- 172.21.15.60:0/2454380549 --> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi060", "addr": "172.21.15.60", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4581040b0 con 0x7fb448038bf0 2024-03-04T12:15:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:48.935+0000 7fb455ffb700 1 -- 172.21.15.60:0/2454380549 <== mgr.14120 v2:172.21.15.60:6800/2246601804 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fb4581040b0 con 0x7fb448038bf0 2024-03-04T12:15:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:48.938+0000 7fb45eddd700 1 -- 172.21.15.60:0/2454380549 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fb448038bf0 msgr2=0x7fb44803b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:48.938+0000 7fb45eddd700 1 --2- 172.21.15.60:0/2454380549 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fb448038bf0 0x7fb44803b0b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb444006fd0 tx=0x7fb444006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:48.938+0000 7fb45eddd700 1 -- 172.21.15.60:0/2454380549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 msgr2=0x7fb45819b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:48.938+0000 7fb45eddd700 1 --2- 172.21.15.60:0/2454380549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 0x7fb45819b770 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb44c000c00 tx=0x7fb44c011c10 comp rx=0 tx=0).stop 2024-03-04T12:15:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:48.938+0000 7fb45eddd700 1 -- 172.21.15.60:0/2454380549 shutdown_connections 2024-03-04T12:15:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:48.938+0000 7fb45eddd700 1 --2- 172.21.15.60:0/2454380549 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fb448038bf0 0x7fb44803b0b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:48.938+0000 7fb45eddd700 1 --2- 172.21.15.60:0/2454380549 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb458107860 0x7fb45819b770 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:48.938+0000 7fb45eddd700 1 -- 172.21.15.60:0/2454380549 >> 172.21.15.60:0/2454380549 conn(0x7fb458102ee0 msgr2=0x7fb4581039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:48.938+0000 7fb45eddd700 1 -- 172.21.15.60:0/2454380549 shutdown_connections 2024-03-04T12:15:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:48.938+0000 7fb45eddd700 1 -- 172.21.15.60:0/2454380549 wait complete. 2024-03-04T12:15:49.474 INFO:teuthology.orchestra.run.smithi060.stdout:Deploying unmanaged mon service... 2024-03-04T12:15:50.102 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:49 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:50.102 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:49 smithi060 ceph-mon[32745]: Added host smithi060 2024-03-04T12:15:50.102 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:49 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:15:51.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:51 smithi060 ceph-mon[32745]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:15:51.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:51 smithi060 ceph-mon[32745]: Saving service mon spec with placement count:5 2024-03-04T12:15:51.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:51 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:51.585 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-04T12:15:51.585 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.923+0000 7f39e83ee700 1 Processor -- start 2024-03-04T12:15:51.585 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.923+0000 7f39e83ee700 1 -- start start 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.923+0000 7f39e83ee700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 0x7f39e0106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.923+0000 7f39e83ee700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0107090 con 0x7f39e01066a0 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.923+0000 7f39e618a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 0x7f39e0106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.923+0000 7f39e618a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 0x7f39e0106ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45722/0 (socket says 172.21.15.60:45722) 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.923+0000 7f39e618a700 1 -- 172.21.15.60:0/4033493116 learned_addr learned my addr 172.21.15.60:0/4033493116 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.924+0000 7f39e618a700 1 -- 172.21.15.60:0/4033493116 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e01078a0 con 0x7f39e01066a0 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.924+0000 7f39e618a700 1 --2- 172.21.15.60:0/4033493116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 0x7f39e0106ac0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0098a0 tx=0x7f39dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad9605b3f267335 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.924+0000 7f39e5188700 1 -- 172.21.15.60:0/4033493116 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39dc004160 con 0x7f39e01066a0 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.924+0000 7f39e5188700 1 -- 172.21.15.60:0/4033493116 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39dc004300 con 0x7f39e01066a0 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.924+0000 7f39e5188700 1 -- 172.21.15.60:0/4033493116 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39dc011b50 con 0x7f39e01066a0 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.924+0000 7f39e83ee700 1 -- 172.21.15.60:0/4033493116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 msgr2=0x7f39e0106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.924+0000 7f39e83ee700 1 --2- 172.21.15.60:0/4033493116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 0x7f39e0106ac0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0098a0 tx=0x7f39dc009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.925+0000 7f39e83ee700 1 -- 172.21.15.60:0/4033493116 shutdown_connections 2024-03-04T12:15:51.586 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.925+0000 7f39e83ee700 1 --2- 172.21.15.60:0/4033493116 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 0x7f39e0106ac0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.925+0000 7f39e83ee700 1 -- 172.21.15.60:0/4033493116 >> 172.21.15.60:0/4033493116 conn(0x7f39e0101d00 msgr2=0x7f39e0104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.925+0000 7f39e83ee700 1 -- 172.21.15.60:0/4033493116 shutdown_connections 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.925+0000 7f39e83ee700 1 -- 172.21.15.60:0/4033493116 wait complete. 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.925+0000 7f39e83ee700 1 Processor -- start 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.925+0000 7f39e83ee700 1 -- start start 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.925+0000 7f39e83ee700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 0x7f39e019b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.925+0000 7f39e83ee700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e010ac50 con 0x7f39e01066a0 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.926+0000 7f39e618a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 0x7f39e019b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.926+0000 7f39e618a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 0x7f39e019b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45736/0 (socket says 172.21.15.60:45736) 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.926+0000 7f39e618a700 1 -- 172.21.15.60:0/2475079497 learned_addr learned my addr 172.21.15.60:0/2475079497 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.926+0000 7f39e618a700 1 -- 172.21.15.60:0/2475079497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e019bcd0 con 0x7f39e01066a0 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.926+0000 7f39e618a700 1 --2- 172.21.15.60:0/2475079497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 0x7f39e019b790 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f39dc011670 tx=0x7f39dc011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.926+0000 7f39d77fe700 1 -- 172.21.15.60:0/2475079497 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39dc0039c0 con 0x7f39e01066a0 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.926+0000 7f39d77fe700 1 -- 172.21.15.60:0/2475079497 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39dc011e50 con 0x7f39e01066a0 2024-03-04T12:15:51.587 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.926+0000 7f39d77fe700 1 -- 172.21.15.60:0/2475079497 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39dc01a5c0 con 0x7f39e01066a0 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.926+0000 7f39e83ee700 1 -- 172.21.15.60:0/2475079497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e019bf60 con 0x7f39e01066a0 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.926+0000 7f39e83ee700 1 -- 172.21.15.60:0/2475079497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e019ec50 con 0x7f39e01066a0 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.927+0000 7f39d77fe700 1 -- 172.21.15.60:0/2475079497 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45927+0+0 (secure 0 0 0) 0x7f39dc01a870 con 0x7f39e01066a0 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.927+0000 7f39d77fe700 1 --2- 172.21.15.60:0/2475079497 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f39cc038ba0 0x7f39cc03b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.927+0000 7f39d77fe700 1 -- 172.21.15.60:0/2475079497 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f39dc04cec0 con 0x7f39e01066a0 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.927+0000 7f39e83ee700 1 -- 172.21.15.60:0/2475079497 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e0063450 con 0x7f39e01066a0 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.927+0000 7f39e5989700 1 --2- 172.21.15.60:0/2475079497 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f39cc038ba0 0x7f39cc03b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.928+0000 7f39e5989700 1 --2- 172.21.15.60:0/2475079497 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f39cc038ba0 0x7f39cc03b060 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f39d0006fd0 tx=0x7f39d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:49.932+0000 7f39d77fe700 1 -- 172.21.15.60:0/2475079497 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f39dc019c70 con 0x7f39e01066a0 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:50.159+0000 7f39e83ee700 1 -- 172.21.15.60:0/2475079497 --> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f39e002daf0 con 0x7f39cc038ba0 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:50.164+0000 7f39d77fe700 1 -- 172.21.15.60:0/2475079497 <== mgr.14120 v2:172.21.15.60:6800/2246601804 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f39e002daf0 con 0x7f39cc038ba0 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:50.166+0000 7f39e83ee700 1 -- 172.21.15.60:0/2475079497 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f39cc038ba0 msgr2=0x7f39cc03b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:51.588 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:50.166+0000 7f39e83ee700 1 --2- 172.21.15.60:0/2475079497 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f39cc038ba0 0x7f39cc03b060 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f39d0006fd0 tx=0x7f39d0006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:51.589 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:50.166+0000 7f39e83ee700 1 -- 172.21.15.60:0/2475079497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 msgr2=0x7f39e019b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:51.589 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:50.166+0000 7f39e83ee700 1 --2- 172.21.15.60:0/2475079497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 0x7f39e019b790 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f39dc011670 tx=0x7f39dc011b60 comp rx=0 tx=0).stop 2024-03-04T12:15:51.589 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:50.166+0000 7f39e83ee700 1 -- 172.21.15.60:0/2475079497 shutdown_connections 2024-03-04T12:15:51.589 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:50.166+0000 7f39e83ee700 1 --2- 172.21.15.60:0/2475079497 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f39cc038ba0 0x7f39cc03b060 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:51.589 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:50.166+0000 7f39e83ee700 1 --2- 172.21.15.60:0/2475079497 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f39e01066a0 0x7f39e019b790 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:51.589 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:50.166+0000 7f39e83ee700 1 -- 172.21.15.60:0/2475079497 >> 172.21.15.60:0/2475079497 conn(0x7f39e0101d00 msgr2=0x7f39e0102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:51.589 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:50.166+0000 7f39e83ee700 1 -- 172.21.15.60:0/2475079497 shutdown_connections 2024-03-04T12:15:51.589 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:50.166+0000 7f39e83ee700 1 -- 172.21.15.60:0/2475079497 wait complete. 2024-03-04T12:15:51.589 INFO:teuthology.orchestra.run.smithi060.stdout:Deploying unmanaged mgr service... 2024-03-04T12:15:52.225 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:52 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:52.225 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:52 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:52.867 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-04T12:15:52.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.972+0000 7fdd0322d700 1 Processor -- start 2024-03-04T12:15:52.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.972+0000 7fdd0322d700 1 -- start start 2024-03-04T12:15:52.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.972+0000 7fdd0322d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 0x7fdcfc106fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:52.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.972+0000 7fdd0322d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc1075a0 con 0x7fdcfc106bb0 2024-03-04T12:15:52.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.972+0000 7fdd00fc9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 0x7fdcfc106fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:52.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.973+0000 7fdd00fc9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 0x7fdcfc106fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45746/0 (socket says 172.21.15.60:45746) 2024-03-04T12:15:52.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.973+0000 7fdd00fc9700 1 -- 172.21.15.60:0/2181276856 learned_addr learned my addr 172.21.15.60:0/2181276856 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:52.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.973+0000 7fdd00fc9700 1 -- 172.21.15.60:0/2181276856 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc1078d0 con 0x7fdcfc106bb0 2024-03-04T12:15:52.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.973+0000 7fdd00fc9700 1 --2- 172.21.15.60:0/2181276856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 0x7fdcfc106fd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00098a0 tx=0x7fdcf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199879726c390e81 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:52.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.973+0000 7fdcfb7fe700 1 -- 172.21.15.60:0/2181276856 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdcf0004160 con 0x7fdcfc106bb0 2024-03-04T12:15:52.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.973+0000 7fdcfb7fe700 1 -- 172.21.15.60:0/2181276856 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcf0004300 con 0x7fdcfc106bb0 2024-03-04T12:15:52.868 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.973+0000 7fdcfb7fe700 1 -- 172.21.15.60:0/2181276856 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdcf0011b20 con 0x7fdcfc106bb0 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.974+0000 7fdd0322d700 1 -- 172.21.15.60:0/2181276856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 msgr2=0x7fdcfc106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.974+0000 7fdd0322d700 1 --2- 172.21.15.60:0/2181276856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 0x7fdcfc106fd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdcf00098a0 tx=0x7fdcf0009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.974+0000 7fdd0322d700 1 -- 172.21.15.60:0/2181276856 shutdown_connections 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.974+0000 7fdd0322d700 1 --2- 172.21.15.60:0/2181276856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 0x7fdcfc106fd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.974+0000 7fdd0322d700 1 -- 172.21.15.60:0/2181276856 >> 172.21.15.60:0/2181276856 conn(0x7fdcfc101450 msgr2=0x7fdcfc103870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.974+0000 7fdd0322d700 1 -- 172.21.15.60:0/2181276856 shutdown_connections 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.974+0000 7fdd0322d700 1 -- 172.21.15.60:0/2181276856 wait complete. 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.975+0000 7fdd0322d700 1 Processor -- start 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.975+0000 7fdd0322d700 1 -- start start 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.975+0000 7fdd0322d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 0x7fdcfc191880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.975+0000 7fdd0322d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc10ac80 con 0x7fdcfc106bb0 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.975+0000 7fdd00fc9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 0x7fdcfc191880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.975+0000 7fdd00fc9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 0x7fdcfc191880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45758/0 (socket says 172.21.15.60:45758) 2024-03-04T12:15:52.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.975+0000 7fdd00fc9700 1 -- 172.21.15.60:0/2235137853 learned_addr learned my addr 172.21.15.60:0/2235137853 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.975+0000 7fdd00fc9700 1 -- 172.21.15.60:0/2235137853 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc191dc0 con 0x7fdcfc106bb0 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.975+0000 7fdd00fc9700 1 --2- 172.21.15.60:0/2235137853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 0x7fdcfc191880 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0011ad0 tx=0x7fdcf0011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.975+0000 7fdcf9ffb700 1 -- 172.21.15.60:0/2235137853 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdcf0003a40 con 0x7fdcfc106bb0 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.976+0000 7fdcf9ffb700 1 -- 172.21.15.60:0/2235137853 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcf0003be0 con 0x7fdcfc106bb0 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.976+0000 7fdd0322d700 1 -- 172.21.15.60:0/2235137853 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcfc18ffb0 con 0x7fdcfc106bb0 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.976+0000 7fdcf9ffb700 1 -- 172.21.15.60:0/2235137853 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdcf0024740 con 0x7fdcfc106bb0 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.976+0000 7fdd0322d700 1 -- 172.21.15.60:0/2235137853 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcfc190470 con 0x7fdcfc106bb0 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.977+0000 7fdcf9ffb700 1 -- 172.21.15.60:0/2235137853 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45927+0+0 (secure 0 0 0) 0x7fdcf002d3f0 con 0x7fdcfc106bb0 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.977+0000 7fdd0322d700 1 -- 172.21.15.60:0/2235137853 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcfc063450 con 0x7fdcfc106bb0 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.977+0000 7fdcf9ffb700 1 --2- 172.21.15.60:0/2235137853 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fdcec038bf0 0x7fdcec03b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.977+0000 7fdcf9ffb700 1 -- 172.21.15.60:0/2235137853 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdcf004dcb0 con 0x7fdcfc106bb0 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.977+0000 7fdcfbfff700 1 --2- 172.21.15.60:0/2235137853 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fdcec038bf0 0x7fdcec03b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:52.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.977+0000 7fdcfbfff700 1 --2- 172.21.15.60:0/2235137853 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fdcec038bf0 0x7fdcec03b0b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdce8006fd0 tx=0x7fdce8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:51.982+0000 7fdcf9ffb700 1 -- 172.21.15.60:0/2235137853 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdcf00172b0 con 0x7fdcfc106bb0 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:52.220+0000 7fdd0322d700 1 -- 172.21.15.60:0/2235137853 --> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fdcfc195640 con 0x7fdcec038bf0 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:52.226+0000 7fdcf9ffb700 1 -- 172.21.15.60:0/2235137853 <== mgr.14120 v2:172.21.15.60:6800/2246601804 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdcfc195640 con 0x7fdcec038bf0 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:52.228+0000 7fdd0322d700 1 -- 172.21.15.60:0/2235137853 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fdcec038bf0 msgr2=0x7fdcec03b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:52.228+0000 7fdd0322d700 1 --2- 172.21.15.60:0/2235137853 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fdcec038bf0 0x7fdcec03b0b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdce8006fd0 tx=0x7fdce8006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:52.228+0000 7fdd0322d700 1 -- 172.21.15.60:0/2235137853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 msgr2=0x7fdcfc191880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:52.228+0000 7fdd0322d700 1 --2- 172.21.15.60:0/2235137853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 0x7fdcfc191880 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0011ad0 tx=0x7fdcf0011bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:52.229+0000 7fdd0322d700 1 -- 172.21.15.60:0/2235137853 shutdown_connections 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:52.229+0000 7fdd0322d700 1 --2- 172.21.15.60:0/2235137853 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fdcec038bf0 0x7fdcec03b0b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:52.229+0000 7fdd0322d700 1 --2- 172.21.15.60:0/2235137853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcfc106bb0 0x7fdcfc191880 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:52.229+0000 7fdd0322d700 1 -- 172.21.15.60:0/2235137853 >> 172.21.15.60:0/2235137853 conn(0x7fdcfc101450 msgr2=0x7fdcfc1022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:52.229+0000 7fdd0322d700 1 -- 172.21.15.60:0/2235137853 shutdown_connections 2024-03-04T12:15:52.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:52.229+0000 7fdd0322d700 1 -- 172.21.15.60:0/2235137853 wait complete. 2024-03-04T12:15:53.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:53 smithi060 ceph-mon[32745]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:15:53.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:53 smithi060 ceph-mon[32745]: Saving service mgr spec with placement count:2 2024-03-04T12:15:53.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:53 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:54.474 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.249+0000 7fc80e599700 1 Processor -- start 2024-03-04T12:15:54.475 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.249+0000 7fc80e599700 1 -- start start 2024-03-04T12:15:54.475 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.249+0000 7fc80e599700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 0x7fc80810a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:54.475 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.249+0000 7fc80e599700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80807adb0 con 0x7fc808108320 2024-03-04T12:15:54.475 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.249+0000 7fc807fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 0x7fc80810a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:54.475 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.249+0000 7fc807fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 0x7fc80810a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45762/0 (socket says 172.21.15.60:45762) 2024-03-04T12:15:54.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.249+0000 7fc807fff700 1 -- 172.21.15.60:0/4049027889 learned_addr learned my addr 172.21.15.60:0/4049027889 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:54.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.250+0000 7fc807fff700 1 -- 172.21.15.60:0/4049027889 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80810ac50 con 0x7fc808108320 2024-03-04T12:15:54.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.250+0000 7fc807fff700 1 --2- 172.21.15.60:0/4049027889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 0x7fc80810a710 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80098a0 tx=0x7fc7f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4dabb26950182eb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:54.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.250+0000 7fc806ffd700 1 -- 172.21.15.60:0/4049027889 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7f8004160 con 0x7fc808108320 2024-03-04T12:15:54.476 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.250+0000 7fc806ffd700 1 -- 172.21.15.60:0/4049027889 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7f8004300 con 0x7fc808108320 2024-03-04T12:15:54.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.250+0000 7fc806ffd700 1 -- 172.21.15.60:0/4049027889 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7f8011b20 con 0x7fc808108320 2024-03-04T12:15:54.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.251+0000 7fc80e599700 1 -- 172.21.15.60:0/4049027889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 msgr2=0x7fc80810a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:54.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.251+0000 7fc80e599700 1 --2- 172.21.15.60:0/4049027889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 0x7fc80810a710 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80098a0 tx=0x7fc7f8009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:54.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.251+0000 7fc80e599700 1 -- 172.21.15.60:0/4049027889 shutdown_connections 2024-03-04T12:15:54.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.251+0000 7fc80e599700 1 --2- 172.21.15.60:0/4049027889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 0x7fc80810a710 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:54.477 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.251+0000 7fc80e599700 1 -- 172.21.15.60:0/4049027889 >> 172.21.15.60:0/4049027889 conn(0x7fc808101cb0 msgr2=0x7fc8081040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:54.478 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.251+0000 7fc80e599700 1 -- 172.21.15.60:0/4049027889 shutdown_connections 2024-03-04T12:15:54.478 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.251+0000 7fc80e599700 1 -- 172.21.15.60:0/4049027889 wait complete. 2024-03-04T12:15:54.478 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.252+0000 7fc80e599700 1 Processor -- start 2024-03-04T12:15:54.478 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.252+0000 7fc80e599700 1 -- start start 2024-03-04T12:15:54.478 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.252+0000 7fc80e599700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 0x7fc808107ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:54.478 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.252+0000 7fc80e599700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc80810bbf0 con 0x7fc808108320 2024-03-04T12:15:54.479 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.252+0000 7fc807fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 0x7fc808107ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:54.479 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.252+0000 7fc807fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 0x7fc808107ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45768/0 (socket says 172.21.15.60:45768) 2024-03-04T12:15:54.479 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.252+0000 7fc807fff700 1 -- 172.21.15.60:0/3512958556 learned_addr learned my addr 172.21.15.60:0/3512958556 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:54.479 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.252+0000 7fc807fff700 1 -- 172.21.15.60:0/3512958556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8081080e0 con 0x7fc808108320 2024-03-04T12:15:54.479 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.253+0000 7fc807fff700 1 --2- 172.21.15.60:0/3512958556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 0x7fc808107ba0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8011b60 tx=0x7fc7f8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:54.479 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.253+0000 7fc8057fa700 1 -- 172.21.15.60:0/3512958556 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7f8003b60 con 0x7fc808108320 2024-03-04T12:15:54.480 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.253+0000 7fc8057fa700 1 -- 172.21.15.60:0/3512958556 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7f801a910 con 0x7fc808108320 2024-03-04T12:15:54.480 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.253+0000 7fc8057fa700 1 -- 172.21.15.60:0/3512958556 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7f8024930 con 0x7fc808108320 2024-03-04T12:15:54.480 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.253+0000 7fc80e599700 1 -- 172.21.15.60:0/3512958556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8081062d0 con 0x7fc808108320 2024-03-04T12:15:54.480 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.253+0000 7fc80e599700 1 -- 172.21.15.60:0/3512958556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc808106790 con 0x7fc808108320 2024-03-04T12:15:54.480 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.254+0000 7fc8057fa700 1 -- 172.21.15.60:0/3512958556 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45927+0+0 (secure 0 0 0) 0x7fc7f8022020 con 0x7fc808108320 2024-03-04T12:15:54.481 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.254+0000 7fc8057fa700 1 --2- 172.21.15.60:0/3512958556 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fc7f0038ba0 0x7fc7f003b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:54.481 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.254+0000 7fc8057fa700 1 -- 172.21.15.60:0/3512958556 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc7f804cee0 con 0x7fc808108320 2024-03-04T12:15:54.481 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.254+0000 7fc8077fe700 1 --2- 172.21.15.60:0/3512958556 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fc7f0038ba0 0x7fc7f003b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:54.481 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.254+0000 7fc80e599700 1 -- 172.21.15.60:0/3512958556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc808068f00 con 0x7fc808108320 2024-03-04T12:15:54.481 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.255+0000 7fc8077fe700 1 --2- 172.21.15.60:0/3512958556 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fc7f0038ba0 0x7fc7f003b060 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc006fd0 tx=0x7fc7fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:54.481 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.263+0000 7fc8057fa700 1 -- 172.21.15.60:0/3512958556 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc7f8019c70 con 0x7fc808108320 2024-03-04T12:15:54.482 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.465+0000 7fc80e599700 1 -- 172.21.15.60:0/3512958556 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fc808106db0 con 0x7fc808108320 2024-03-04T12:15:54.482 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.467+0000 7fc8057fa700 1 -- 172.21.15.60:0/3512958556 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fc7f800bc00 con 0x7fc808108320 2024-03-04T12:15:54.482 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.470+0000 7fc80e599700 1 -- 172.21.15.60:0/3512958556 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fc7f0038ba0 msgr2=0x7fc7f003b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:54.482 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.470+0000 7fc80e599700 1 --2- 172.21.15.60:0/3512958556 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fc7f0038ba0 0x7fc7f003b060 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc006fd0 tx=0x7fc7fc006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:54.482 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.470+0000 7fc80e599700 1 -- 172.21.15.60:0/3512958556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 msgr2=0x7fc808107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:54.482 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.470+0000 7fc80e599700 1 --2- 172.21.15.60:0/3512958556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 0x7fc808107ba0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8011b60 tx=0x7fc7f8011c40 comp rx=0 tx=0).stop 2024-03-04T12:15:54.483 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.470+0000 7fc80e599700 1 -- 172.21.15.60:0/3512958556 shutdown_connections 2024-03-04T12:15:54.483 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.470+0000 7fc80e599700 1 --2- 172.21.15.60:0/3512958556 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fc7f0038ba0 0x7fc7f003b060 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:54.483 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.470+0000 7fc80e599700 1 --2- 172.21.15.60:0/3512958556 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc808108320 0x7fc808107ba0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:54.483 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.470+0000 7fc80e599700 1 -- 172.21.15.60:0/3512958556 >> 172.21.15.60:0/3512958556 conn(0x7fc808101cb0 msgr2=0x7fc808102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:54.483 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.470+0000 7fc80e599700 1 -- 172.21.15.60:0/3512958556 shutdown_connections 2024-03-04T12:15:54.484 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:53.470+0000 7fc80e599700 1 -- 172.21.15.60:0/3512958556 wait complete. 2024-03-04T12:15:54.721 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:54 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3512958556' entity='client.admin' 2024-03-04T12:15:55.682 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.917+0000 7faac7c56700 1 Processor -- start 2024-03-04T12:15:55.682 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.917+0000 7faac7c56700 1 -- start start 2024-03-04T12:15:55.682 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.917+0000 7faac7c56700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 0x7faac007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:55.682 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.917+0000 7faac7c56700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac007b800 con 0x7faac007ce10 2024-03-04T12:15:55.682 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.917+0000 7faac59f2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 0x7faac007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:55.682 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.918+0000 7faac59f2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 0x7faac007b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45776/0 (socket says 172.21.15.60:45776) 2024-03-04T12:15:55.682 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.918+0000 7faac59f2700 1 -- 172.21.15.60:0/2775563910 learned_addr learned my addr 172.21.15.60:0/2775563910 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:55.682 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.918+0000 7faac59f2700 1 -- 172.21.15.60:0/2775563910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac007b980 con 0x7faac007ce10 2024-03-04T12:15:55.682 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.918+0000 7faac59f2700 1 --2- 172.21.15.60:0/2775563910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 0x7faac007b2c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faab00098a0 tx=0x7faab0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd97150a1d744acb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:55.682 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.918+0000 7faac49f0700 1 -- 172.21.15.60:0/2775563910 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faab0004160 con 0x7faac007ce10 2024-03-04T12:15:55.682 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.918+0000 7faac49f0700 1 -- 172.21.15.60:0/2775563910 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faab0004300 con 0x7faac007ce10 2024-03-04T12:15:55.682 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.918+0000 7faac49f0700 1 -- 172.21.15.60:0/2775563910 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faab0011b20 con 0x7faac007ce10 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.919+0000 7faac7c56700 1 -- 172.21.15.60:0/2775563910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 msgr2=0x7faac007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.919+0000 7faac7c56700 1 --2- 172.21.15.60:0/2775563910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 0x7faac007b2c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faab00098a0 tx=0x7faab0009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.919+0000 7faac7c56700 1 -- 172.21.15.60:0/2775563910 shutdown_connections 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.919+0000 7faac7c56700 1 --2- 172.21.15.60:0/2775563910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 0x7faac007b2c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.919+0000 7faac7c56700 1 -- 172.21.15.60:0/2775563910 >> 172.21.15.60:0/2775563910 conn(0x7faac0102e20 msgr2=0x7faac0105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.919+0000 7faac7c56700 1 -- 172.21.15.60:0/2775563910 shutdown_connections 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.919+0000 7faac7c56700 1 -- 172.21.15.60:0/2775563910 wait complete. 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.920+0000 7faac7c56700 1 Processor -- start 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.920+0000 7faac7c56700 1 -- start start 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.920+0000 7faac7c56700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 0x7faac019d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.920+0000 7faac7c56700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac010dda0 con 0x7faac007ce10 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.920+0000 7faac59f2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 0x7faac019d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:55.683 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.920+0000 7faac59f2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 0x7faac019d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45784/0 (socket says 172.21.15.60:45784) 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.920+0000 7faac59f2700 1 -- 172.21.15.60:0/2516882300 learned_addr learned my addr 172.21.15.60:0/2516882300 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.920+0000 7faac59f2700 1 -- 172.21.15.60:0/2516882300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac019dde0 con 0x7faac007ce10 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.920+0000 7faac59f2700 1 --2- 172.21.15.60:0/2516882300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 0x7faac019d8a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faab00117e0 tx=0x7faab0011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.920+0000 7faab6ffd700 1 -- 172.21.15.60:0/2516882300 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faab0003c90 con 0x7faac007ce10 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.920+0000 7faab6ffd700 1 -- 172.21.15.60:0/2516882300 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faab001a910 con 0x7faac007ce10 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.921+0000 7faac7c56700 1 -- 172.21.15.60:0/2516882300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac019e070 con 0x7faac007ce10 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.921+0000 7faab6ffd700 1 -- 172.21.15.60:0/2516882300 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faab0024a10 con 0x7faac007ce10 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.921+0000 7faac7c56700 1 -- 172.21.15.60:0/2516882300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac019e4d0 con 0x7faac007ce10 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.921+0000 7faab6ffd700 1 -- 172.21.15.60:0/2516882300 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45927+0+0 (secure 0 0 0) 0x7faab0022020 con 0x7faac007ce10 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.921+0000 7faac7c56700 1 -- 172.21.15.60:0/2516882300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac0063450 con 0x7faac007ce10 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.922+0000 7faab6ffd700 1 --2- 172.21.15.60:0/2516882300 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7faaac038ba0 0x7faaac03b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.922+0000 7faab6ffd700 1 -- 172.21.15.60:0/2516882300 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7faab004cdc0 con 0x7faac007ce10 2024-03-04T12:15:55.684 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.922+0000 7faac51f1700 1 --2- 172.21.15.60:0/2516882300 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7faaac038ba0 0x7faaac03b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:55.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.922+0000 7faac51f1700 1 --2- 172.21.15.60:0/2516882300 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7faaac038ba0 0x7faaac03b060 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faabc006fd0 tx=0x7faabc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:55.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:54.927+0000 7faab6ffd700 1 -- 172.21.15.60:0/2516882300 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faab0011e20 con 0x7faac007ce10 2024-03-04T12:15:55.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:55.135+0000 7faac7c56700 1 -- 172.21.15.60:0/2516882300 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7faac01a0f60 con 0x7faac007ce10 2024-03-04T12:15:55.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:55.138+0000 7faab6ffd700 1 -- 172.21.15.60:0/2516882300 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7faab0019c70 con 0x7faac007ce10 2024-03-04T12:15:55.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:55.141+0000 7faac7c56700 1 -- 172.21.15.60:0/2516882300 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7faaac038ba0 msgr2=0x7faaac03b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:55.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:55.141+0000 7faac7c56700 1 --2- 172.21.15.60:0/2516882300 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7faaac038ba0 0x7faaac03b060 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faabc006fd0 tx=0x7faabc006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:55.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:55.141+0000 7faac7c56700 1 -- 172.21.15.60:0/2516882300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 msgr2=0x7faac019d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:55.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:55.141+0000 7faac7c56700 1 --2- 172.21.15.60:0/2516882300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 0x7faac019d8a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faab00117e0 tx=0x7faab0011ce0 comp rx=0 tx=0).stop 2024-03-04T12:15:55.685 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:55.141+0000 7faac7c56700 1 -- 172.21.15.60:0/2516882300 shutdown_connections 2024-03-04T12:15:55.686 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:55.141+0000 7faac7c56700 1 --2- 172.21.15.60:0/2516882300 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7faaac038ba0 0x7faaac03b060 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:55.686 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:55.141+0000 7faac7c56700 1 --2- 172.21.15.60:0/2516882300 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faac007ce10 0x7faac019d8a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:55.686 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:55.141+0000 7faac7c56700 1 -- 172.21.15.60:0/2516882300 >> 172.21.15.60:0/2516882300 conn(0x7faac0102e20 msgr2=0x7faac0103880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:55.686 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:55.141+0000 7faac7c56700 1 -- 172.21.15.60:0/2516882300 shutdown_connections 2024-03-04T12:15:55.686 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:55.141+0000 7faac7c56700 1 -- 172.21.15.60:0/2516882300 wait complete. 2024-03-04T12:15:55.686 INFO:teuthology.orchestra.run.smithi060.stdout:Enabling the dashboard module... 2024-03-04T12:15:56.390 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:56 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:56.390 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:56 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2516882300' entity='client.admin' 2024-03-04T12:15:57.417 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:57 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3090263815' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-04T12:15:57.417 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:57 smithi060 ceph-mon[32745]: from='mgr.14120 172.21.15.60:0/364816955' entity='mgr.x' 2024-03-04T12:15:57.417 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:57 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: ignoring --setuser ceph since I am not root 2024-03-04T12:15:57.418 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:57 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: ignoring --setgroup ceph since I am not root 2024-03-04T12:15:57.675 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:57 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:57.609+0000 7f5a95ea3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T12:15:58.023 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:57 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:57.705+0000 7f5a95ea3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T12:15:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.113+0000 7fd0f9271700 1 Processor -- start 2024-03-04T12:15:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.113+0000 7fd0f9271700 1 -- start start 2024-03-04T12:15:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.114+0000 7fd0f9271700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 0x7fd0f407b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.114+0000 7fd0f9271700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f407b8d0 con 0x7fd0f407ce50 2024-03-04T12:15:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.114+0000 7fd0f2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 0x7fd0f407b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.114+0000 7fd0f2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 0x7fd0f407b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46742/0 (socket says 172.21.15.60:46742) 2024-03-04T12:15:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.114+0000 7fd0f2ffd700 1 -- 172.21.15.60:0/3545536109 learned_addr learned my addr 172.21.15.60:0/3545536109 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.114+0000 7fd0f2ffd700 1 -- 172.21.15.60:0/3545536109 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f407ba50 con 0x7fd0f407ce50 2024-03-04T12:15:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.114+0000 7fd0f2ffd700 1 --2- 172.21.15.60:0/3545536109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 0x7fd0f407b300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40098a0 tx=0x7fd0e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ce426ac39d63fb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:58.037 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.114+0000 7fd0f1ffb700 1 -- 172.21.15.60:0/3545536109 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0e4004160 con 0x7fd0f407ce50 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.115+0000 7fd0f1ffb700 1 -- 172.21.15.60:0/3545536109 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0e4004300 con 0x7fd0f407ce50 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.115+0000 7fd0f1ffb700 1 -- 172.21.15.60:0/3545536109 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0e4011b20 con 0x7fd0f407ce50 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.115+0000 7fd0f9271700 1 -- 172.21.15.60:0/3545536109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 msgr2=0x7fd0f407b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.115+0000 7fd0f9271700 1 --2- 172.21.15.60:0/3545536109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 0x7fd0f407b300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40098a0 tx=0x7fd0e4009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.115+0000 7fd0f9271700 1 -- 172.21.15.60:0/3545536109 shutdown_connections 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.115+0000 7fd0f9271700 1 --2- 172.21.15.60:0/3545536109 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 0x7fd0f407b300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.115+0000 7fd0f9271700 1 -- 172.21.15.60:0/3545536109 >> 172.21.15.60:0/3545536109 conn(0x7fd0f4102e20 msgr2=0x7fd0f4105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.115+0000 7fd0f9271700 1 -- 172.21.15.60:0/3545536109 shutdown_connections 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.115+0000 7fd0f9271700 1 -- 172.21.15.60:0/3545536109 wait complete. 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.116+0000 7fd0f9271700 1 Processor -- start 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.116+0000 7fd0f9271700 1 -- start start 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.116+0000 7fd0f9271700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 0x7fd0f419d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:58.038 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.116+0000 7fd0f9271700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f410ddc0 con 0x7fd0f407ce50 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.116+0000 7fd0f2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 0x7fd0f419d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.116+0000 7fd0f2ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 0x7fd0f419d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46748/0 (socket says 172.21.15.60:46748) 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.116+0000 7fd0f2ffd700 1 -- 172.21.15.60:0/3090263815 learned_addr learned my addr 172.21.15.60:0/3090263815 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.117+0000 7fd0f2ffd700 1 -- 172.21.15.60:0/3090263815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f419ddf0 con 0x7fd0f407ce50 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.117+0000 7fd0f2ffd700 1 --2- 172.21.15.60:0/3090263815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 0x7fd0f419d8b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40117e0 tx=0x7fd0e4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.117+0000 7fd0ebfff700 1 -- 172.21.15.60:0/3090263815 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0e4003ba0 con 0x7fd0f407ce50 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.117+0000 7fd0ebfff700 1 -- 172.21.15.60:0/3090263815 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0e401a910 con 0x7fd0f407ce50 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.117+0000 7fd0ebfff700 1 -- 172.21.15.60:0/3090263815 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0e40248f0 con 0x7fd0f407ce50 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.117+0000 7fd0f9271700 1 -- 172.21.15.60:0/3090263815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f419e080 con 0x7fd0f407ce50 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.117+0000 7fd0f9271700 1 -- 172.21.15.60:0/3090263815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f419e4e0 con 0x7fd0f407ce50 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.118+0000 7fd0f9271700 1 -- 172.21.15.60:0/3090263815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f4063450 con 0x7fd0f407ce50 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.118+0000 7fd0ebfff700 1 -- 172.21.15.60:0/3090263815 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 8) v1 ==== 45927+0+0 (secure 0 0 0) 0x7fd0e4022020 con 0x7fd0f407ce50 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.118+0000 7fd0ebfff700 1 --2- 172.21.15.60:0/3090263815 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fd0e0038bf0 0x7fd0e003b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:58.039 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.118+0000 7fd0ebfff700 1 -- 172.21.15.60:0/3090263815 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd0e404ce30 con 0x7fd0f407ce50 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.118+0000 7fd0f27fc700 1 --2- 172.21.15.60:0/3090263815 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fd0e0038bf0 0x7fd0e003b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.119+0000 7fd0f27fc700 1 --2- 172.21.15.60:0/3090263815 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fd0e0038bf0 0x7fd0e003b0b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc006fd0 tx=0x7fd0dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.123+0000 7fd0ebfff700 1 -- 172.21.15.60:0/3090263815 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd0e402ad60 con 0x7fd0f407ce50 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:56.386+0000 7fd0f9271700 1 -- 172.21.15.60:0/3090263815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fd0f41a0f70 con 0x7fd0f407ce50 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:57.020+0000 7fd0ebfff700 1 -- 172.21.15.60:0/3090263815 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fd0e4021d60 con 0x7fd0f407ce50 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:57.021+0000 7fd0ebfff700 1 -- 172.21.15.60:0/3090263815 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 9) v1 ==== 45940+0+0 (secure 0 0 0) 0x7fd0e402d3f0 con 0x7fd0f407ce50 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:57.023+0000 7fd0f9271700 1 -- 172.21.15.60:0/3090263815 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fd0e0038bf0 msgr2=0x7fd0e003b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:57.023+0000 7fd0f9271700 1 --2- 172.21.15.60:0/3090263815 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fd0e0038bf0 0x7fd0e003b0b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc006fd0 tx=0x7fd0dc006e40 comp rx=0 tx=0).stop 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:57.023+0000 7fd0f9271700 1 -- 172.21.15.60:0/3090263815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 msgr2=0x7fd0f419d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:57.023+0000 7fd0f9271700 1 --2- 172.21.15.60:0/3090263815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 0x7fd0f419d8b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40117e0 tx=0x7fd0e4011ba0 comp rx=0 tx=0).stop 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:57.023+0000 7fd0f9271700 1 -- 172.21.15.60:0/3090263815 shutdown_connections 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:57.023+0000 7fd0f9271700 1 --2- 172.21.15.60:0/3090263815 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7fd0e0038bf0 0x7fd0e003b0b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:58.040 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:57.023+0000 7fd0f9271700 1 --2- 172.21.15.60:0/3090263815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0f407ce50 0x7fd0f419d8b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:58.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:57.023+0000 7fd0f9271700 1 -- 172.21.15.60:0/3090263815 >> 172.21.15.60:0/3090263815 conn(0x7fd0f4102e20 msgr2=0x7fd0f41038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:58.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:57.023+0000 7fd0f9271700 1 -- 172.21.15.60:0/3090263815 shutdown_connections 2024-03-04T12:15:58.041 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:57.023+0000 7fd0f9271700 1 -- 172.21.15.60:0/3090263815 wait complete. 2024-03-04T12:15:58.333 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:58 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3090263815' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-04T12:15:58.333 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:58 smithi060 ceph-mon[32745]: mgrmap e9: x(active, since 19s) 2024-03-04T12:15:58.622 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:58 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:58.375+0000 7f5a95ea3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T12:15:58.622 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:58 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:58.473+0000 7f5a95ea3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T12:15:58.622 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:58 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:58.618+0000 7f5a95ea3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T12:15:58.925 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:58 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:58.719+0000 7f5a95ea3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T12:15:58.925 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:58 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:58.806+0000 7f5a95ea3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T12:15:59.282 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:15:59 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1562167336' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T12:15:59.282 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:59 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:59.078+0000 7f5a95ea3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T12:15:59.282 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:59 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:59.190+0000 7f5a95ea3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T12:15:59.283 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:59 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:59.279+0000 7f5a95ea3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T12:15:59.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2024-03-04T12:15:59.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-04T12:15:59.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T12:15:59.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-04T12:15:59.405 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.432+0000 7f31c81a9700 1 Processor -- start 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.432+0000 7f31c81a9700 1 -- start start 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.432+0000 7f31c81a9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 0x7f31c0107c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.432+0000 7f31c81a9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c01081f0 con 0x7f31c0107800 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.432+0000 7f31c5f45700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 0x7f31c0107c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.432+0000 7f31c5f45700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 0x7f31c0107c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46778/0 (socket says 172.21.15.60:46778) 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.432+0000 7f31c5f45700 1 -- 172.21.15.60:0/3487301195 learned_addr learned my addr 172.21.15.60:0/3487301195 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.432+0000 7f31c5f45700 1 -- 172.21.15.60:0/3487301195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c01089b0 con 0x7f31c0107800 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.433+0000 7f31c5f45700 1 --2- 172.21.15.60:0/3487301195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 0x7f31c0107c20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f31bc0098a0 tx=0x7f31bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6859de1726a1aaee server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.433+0000 7f31c4f43700 1 -- 172.21.15.60:0/3487301195 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31bc004160 con 0x7f31c0107800 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.433+0000 7f31c4f43700 1 -- 172.21.15.60:0/3487301195 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31bc004300 con 0x7f31c0107800 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.433+0000 7f31c4f43700 1 -- 172.21.15.60:0/3487301195 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31bc011b20 con 0x7f31c0107800 2024-03-04T12:15:59.406 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.433+0000 7f31c81a9700 1 -- 172.21.15.60:0/3487301195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 msgr2=0x7f31c0107c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.433+0000 7f31c81a9700 1 --2- 172.21.15.60:0/3487301195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 0x7f31c0107c20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f31bc0098a0 tx=0x7f31bc009bb0 comp rx=0 tx=0).stop 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.434+0000 7f31c81a9700 1 -- 172.21.15.60:0/3487301195 shutdown_connections 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.434+0000 7f31c81a9700 1 --2- 172.21.15.60:0/3487301195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 0x7f31c0107c20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.434+0000 7f31c81a9700 1 -- 172.21.15.60:0/3487301195 >> 172.21.15.60:0/3487301195 conn(0x7f31c0102e20 msgr2=0x7f31c0105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.434+0000 7f31c81a9700 1 -- 172.21.15.60:0/3487301195 shutdown_connections 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.434+0000 7f31c81a9700 1 -- 172.21.15.60:0/3487301195 wait complete. 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.434+0000 7f31c81a9700 1 Processor -- start 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.434+0000 7f31c81a9700 1 -- start start 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.434+0000 7f31c81a9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 0x7f31c019b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.434+0000 7f31c81a9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31c010bd60 con 0x7f31c0107800 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.434+0000 7f31c5f45700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 0x7f31c019b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.435+0000 7f31c5f45700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 0x7f31c019b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46792/0 (socket says 172.21.15.60:46792) 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.435+0000 7f31c5f45700 1 -- 172.21.15.60:0/1562167336 learned_addr learned my addr 172.21.15.60:0/1562167336 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.435+0000 7f31c5f45700 1 -- 172.21.15.60:0/1562167336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c019bca0 con 0x7f31c0107800 2024-03-04T12:15:59.407 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.435+0000 7f31c5f45700 1 --2- 172.21.15.60:0/1562167336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 0x7f31c019b760 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f31bc0116a0 tx=0x7f31bc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.435+0000 7f31b2ffd700 1 -- 172.21.15.60:0/1562167336 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31bc003a40 con 0x7f31c0107800 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.435+0000 7f31b2ffd700 1 -- 172.21.15.60:0/1562167336 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31bc003be0 con 0x7f31c0107800 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.435+0000 7f31c81a9700 1 -- 172.21.15.60:0/1562167336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31c019bf30 con 0x7f31c0107800 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.435+0000 7f31b2ffd700 1 -- 172.21.15.60:0/1562167336 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31bc01a720 con 0x7f31c0107800 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.435+0000 7f31c81a9700 1 -- 172.21.15.60:0/1562167336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31c019c390 con 0x7f31c0107800 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.436+0000 7f31b2ffd700 1 -- 172.21.15.60:0/1562167336 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 9) v1 ==== 45940+0+0 (secure 0 0 0) 0x7f31bc01aaa0 con 0x7f31c0107800 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.436+0000 7f31b2ffd700 1 --2- 172.21.15.60:0/1562167336 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f31ac038ba0 0x7f31ac03b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.436+0000 7f31b2ffd700 1 -- 172.21.15.60:0/1562167336 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f31bc04d350 con 0x7f31c0107800 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.436+0000 7f31c5744700 1 -- 172.21.15.60:0/1562167336 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f31ac038ba0 msgr2=0x7f31ac03b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2246601804 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.436+0000 7f31c5744700 1 --2- 172.21.15.60:0/1562167336 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f31ac038ba0 0x7f31ac03b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.436+0000 7f31c81a9700 1 -- 172.21.15.60:0/1562167336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31c0068f00 con 0x7f31c0107800 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.442+0000 7f31b2ffd700 1 -- 172.21.15.60:0/1562167336 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f31bc011e20 con 0x7f31c0107800 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.637+0000 7f31c5744700 1 -- 172.21.15.60:0/1562167336 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f31ac038ba0 msgr2=0x7f31ac03b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2246601804 2024-03-04T12:15:59.408 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.637+0000 7f31c5744700 1 --2- 172.21.15.60:0/1562167336 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f31ac038ba0 0x7f31ac03b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.700+0000 7f31c81a9700 1 -- 172.21.15.60:0/1562167336 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f31c019ecc0 con 0x7f31c0107800 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.701+0000 7f31b2ffd700 1 -- 172.21.15.60:0/1562167336 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f31bc032030 con 0x7f31c0107800 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.703+0000 7f31c81a9700 1 -- 172.21.15.60:0/1562167336 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f31ac038ba0 msgr2=0x7f31ac03b060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.703+0000 7f31c81a9700 1 --2- 172.21.15.60:0/1562167336 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f31ac038ba0 0x7f31ac03b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.703+0000 7f31c81a9700 1 -- 172.21.15.60:0/1562167336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 msgr2=0x7f31c019b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.703+0000 7f31c81a9700 1 --2- 172.21.15.60:0/1562167336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 0x7f31c019b760 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f31bc0116a0 tx=0x7f31bc011ba0 comp rx=0 tx=0).stop 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.703+0000 7f31c81a9700 1 -- 172.21.15.60:0/1562167336 shutdown_connections 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.703+0000 7f31c81a9700 1 --2- 172.21.15.60:0/1562167336 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f31ac038ba0 0x7f31ac03b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.703+0000 7f31c81a9700 1 --2- 172.21.15.60:0/1562167336 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f31c0107800 0x7f31c019b760 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.703+0000 7f31c81a9700 1 -- 172.21.15.60:0/1562167336 >> 172.21.15.60:0/1562167336 conn(0x7f31c0102e20 msgr2=0x7f31c0103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.703+0000 7f31c81a9700 1 -- 172.21.15.60:0/1562167336 shutdown_connections 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:58.703+0000 7f31c81a9700 1 -- 172.21.15.60:0/1562167336 wait complete. 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for the mgr to restart... 2024-03-04T12:15:59.409 INFO:teuthology.orchestra.run.smithi060.stdout:Waiting for mgr epoch 9... 2024-03-04T12:15:59.561 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:59 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:59.376+0000 7f5a95ea3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T12:15:59.562 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:15:59 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:15:59.558+0000 7f5a95ea3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T12:16:00.675 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:00.549+0000 7f5a95ea3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T12:16:01.023 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:00.683+0000 7f5a95ea3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T12:16:01.023 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:01 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:01.020+0000 7f5a95ea3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T12:16:01.319 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:01 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:01.114+0000 7f5a95ea3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T12:16:01.319 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:01 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:01.316+0000 7f5a95ea3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:16:01.675 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:01 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:01.506+0000 7f5a95ea3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T12:16:02.091 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:01 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:01.802+0000 7f5a95ea3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:16:02.091 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:02 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:02.088+0000 7f5a95ea3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T12:16:02.425 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:02 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:02.331+0000 7f5a95ea3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T12:16:03.425 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:03 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:03.158+0000 7f5a95ea3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T12:16:04.063 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:03 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:03.739+0000 7f5a95ea3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T12:16:04.063 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:03 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:03.839+0000 7f5a95ea3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T12:16:04.063 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:16:04.060+0000 7f5a95ea3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T12:16:04.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-mon[32745]: Active manager daemon x restarted 2024-03-04T12:16:04.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-mon[32745]: Activating manager daemon x 2024-03-04T12:16:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-mon[32745]: osdmap e3: 0 total, 0 up, 0 in 2024-03-04T12:16:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-mon[32745]: mgrmap e10: x(active, starting, since 0.00334297s) 2024-03-04T12:16:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:16:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T12:16:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T12:16:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T12:16:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T12:16:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-mon[32745]: Manager daemon x is now available 2024-03-04T12:16:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:16:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:16:05.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T12:16:05.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T12:16:05.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:05 smithi060 ceph-mon[32745]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T12:16:05.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:05 smithi060 ceph-mon[32745]: mgrmap e11: x(active, since 1.00705s) 2024-03-04T12:16:05.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout { 2024-03-04T12:16:05.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-04T12:16:05.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-04T12:16:05.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout } 2024-03-04T12:16:05.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.814+0000 7f5599142700 1 Processor -- start 2024-03-04T12:16:05.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.814+0000 7f5599142700 1 -- start start 2024-03-04T12:16:05.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.815+0000 7f5599142700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 0x7f559410a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:05.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.815+0000 7f5599142700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559407adb0 con 0x7f5594108320 2024-03-04T12:16:05.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.815+0000 7f5592d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 0x7f559410a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:05.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.815+0000 7f5592d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 0x7f559410a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46798/0 (socket says 172.21.15.60:46798) 2024-03-04T12:16:05.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.815+0000 7f5592d9d700 1 -- 172.21.15.60:0/2442094926 learned_addr learned my addr 172.21.15.60:0/2442094926 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:05.869 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.815+0000 7f5592d9d700 1 -- 172.21.15.60:0/2442094926 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559410ac50 con 0x7f5594108320 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.815+0000 7f5592d9d700 1 --2- 172.21.15.60:0/2442094926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 0x7f559410a710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f557c0098a0 tx=0x7f557c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fac20a11c7529dc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.816+0000 7f5591d9b700 1 -- 172.21.15.60:0/2442094926 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f557c004160 con 0x7f5594108320 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.816+0000 7f5591d9b700 1 -- 172.21.15.60:0/2442094926 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f557c004300 con 0x7f5594108320 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.816+0000 7f5591d9b700 1 -- 172.21.15.60:0/2442094926 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f557c011b20 con 0x7f5594108320 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.816+0000 7f5599142700 1 -- 172.21.15.60:0/2442094926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 msgr2=0x7f559410a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.816+0000 7f5599142700 1 --2- 172.21.15.60:0/2442094926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 0x7f559410a710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f557c0098a0 tx=0x7f557c009bb0 comp rx=0 tx=0).stop 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.816+0000 7f5599142700 1 -- 172.21.15.60:0/2442094926 shutdown_connections 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.816+0000 7f5599142700 1 --2- 172.21.15.60:0/2442094926 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 0x7f559410a710 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.816+0000 7f5599142700 1 -- 172.21.15.60:0/2442094926 >> 172.21.15.60:0/2442094926 conn(0x7f5594101cb0 msgr2=0x7f55941040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.816+0000 7f5599142700 1 -- 172.21.15.60:0/2442094926 shutdown_connections 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.816+0000 7f5599142700 1 -- 172.21.15.60:0/2442094926 wait complete. 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.817+0000 7f5599142700 1 Processor -- start 2024-03-04T12:16:05.870 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.817+0000 7f5599142700 1 -- start start 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.818+0000 7f5599142700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 0x7f559419b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.818+0000 7f5599142700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f559410bbf0 con 0x7f5594108320 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.818+0000 7f5592d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 0x7f559419b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.818+0000 7f5592d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 0x7f559419b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46804/0 (socket says 172.21.15.60:46804) 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.818+0000 7f5592d9d700 1 -- 172.21.15.60:0/4037001215 learned_addr learned my addr 172.21.15.60:0/4037001215 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.818+0000 7f5592d9d700 1 -- 172.21.15.60:0/4037001215 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559419bcb0 con 0x7f5594108320 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.818+0000 7f5592d9d700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 0x7f559419b770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f557c0116a0 tx=0x7f557c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.818+0000 7f558bfff700 1 -- 172.21.15.60:0/4037001215 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f557c003a40 con 0x7f5594108320 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.818+0000 7f558bfff700 1 -- 172.21.15.60:0/4037001215 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f557c003be0 con 0x7f5594108320 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.818+0000 7f5599142700 1 -- 172.21.15.60:0/4037001215 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559419bf40 con 0x7f5594108320 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.818+0000 7f558bfff700 1 -- 172.21.15.60:0/4037001215 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f557c01a750 con 0x7f5594108320 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.819+0000 7f5599142700 1 -- 172.21.15.60:0/4037001215 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f559419c3a0 con 0x7f5594108320 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.819+0000 7f558bfff700 1 -- 172.21.15.60:0/4037001215 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 9) v1 ==== 45940+0+0 (secure 0 0 0) 0x7f557c01aa90 con 0x7f5594108320 2024-03-04T12:16:05.871 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.820+0000 7f558bfff700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 0x7f558003b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.820+0000 7f558bfff700 1 -- 172.21.15.60:0/4037001215 --> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f558003b770 con 0x7f5580038ba0 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.820+0000 7f558bfff700 1 -- 172.21.15.60:0/4037001215 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f557c04d190 con 0x7f5594108320 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.820+0000 7f559259c700 1 -- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 msgr2=0x7f558003b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2246601804 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:15:59.820+0000 7f559259c700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 0x7f558003b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:00.021+0000 7f559259c700 1 -- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 msgr2=0x7f558003b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2246601804 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:00.021+0000 7f559259c700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 0x7f558003b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:00.422+0000 7f559259c700 1 -- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 msgr2=0x7f558003b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2246601804 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:00.422+0000 7f559259c700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 0x7f558003b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:01.223+0000 7f559259c700 1 -- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 msgr2=0x7f558003b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2246601804 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:01.223+0000 7f559259c700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 0x7f558003b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:02.824+0000 7f559259c700 1 -- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 msgr2=0x7f558003b060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/2246601804 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:02.824+0000 7f559259c700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 0x7f558003b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-04T12:16:05.872 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:04.074+0000 7f558bfff700 1 -- 172.21.15.60:0/4037001215 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mgrmap(e 10) v1 ==== 45707+0+0 (secure 0 0 0) 0x7f557c02d910 con 0x7f5594108320 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:04.075+0000 7f558bfff700 1 -- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 msgr2=0x7f558003b060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:04.075+0000 7f558bfff700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 0x7f558003b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.077+0000 7f558bfff700 1 -- 172.21.15.60:0/4037001215 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 11) v1 ==== 45834+0+0 (secure 0 0 0) 0x7f557c0208b0 con 0x7f5594108320 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.077+0000 7f558bfff700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f558003c660 0x7f558003ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.077+0000 7f558bfff700 1 -- 172.21.15.60:0/4037001215 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f557c0248d0 con 0x7f558003c660 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.077+0000 7f559259c700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f558003c660 0x7f558003ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.078+0000 7f559259c700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f558003c660 0x7f558003ea50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5584003a10 tx=0x7f55840092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.078+0000 7f558bfff700 1 -- 172.21.15.60:0/4037001215 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f557c0248d0 con 0x7f558003c660 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.085+0000 7f5599142700 1 -- 172.21.15.60:0/4037001215 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5594068f00 con 0x7f558003c660 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.086+0000 7f558bfff700 1 -- 172.21.15.60:0/4037001215 <== mgr.14152 v2:172.21.15.60:6800/1914863179 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f5594068f00 con 0x7f558003c660 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.086+0000 7f5599142700 1 -- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f558003c660 msgr2=0x7f558003ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.086+0000 7f5599142700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f558003c660 0x7f558003ea50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5584003a10 tx=0x7f55840092b0 comp rx=0 tx=0).stop 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.086+0000 7f5599142700 1 -- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 msgr2=0x7f559419b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.086+0000 7f5599142700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 0x7f559419b770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f557c0116a0 tx=0x7f557c011ba0 comp rx=0 tx=0).stop 2024-03-04T12:16:05.873 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.086+0000 7f5599142700 1 -- 172.21.15.60:0/4037001215 shutdown_connections 2024-03-04T12:16:05.874 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.086+0000 7f5599142700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f558003c660 0x7f558003ea50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:05.874 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.086+0000 7f5599142700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5594108320 0x7f559419b770 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:05.874 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.086+0000 7f5599142700 1 --2- 172.21.15.60:0/4037001215 >> [v2:172.21.15.60:6800/2246601804,v1:172.21.15.60:6801/2246601804] conn(0x7f5580038ba0 0x7f558003b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:05.874 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.086+0000 7f5599142700 1 -- 172.21.15.60:0/4037001215 >> 172.21.15.60:0/4037001215 conn(0x7f5594101cb0 msgr2=0x7f5594102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:05.874 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.086+0000 7f5599142700 1 -- 172.21.15.60:0/4037001215 shutdown_connections 2024-03-04T12:16:05.874 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:05.086+0000 7f5599142700 1 -- 172.21.15.60:0/4037001215 wait complete. 2024-03-04T12:16:05.874 INFO:teuthology.orchestra.run.smithi060.stdout:mgr epoch 9 is available 2024-03-04T12:16:05.874 INFO:teuthology.orchestra.run.smithi060.stdout:Generating a dashboard self-signed certificate... 2024-03-04T12:16:06.220 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:06 smithi060 ceph-mon[32745]: [04/Mar/2024:12:16:05] ENGINE Bus STARTING 2024-03-04T12:16:06.220 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:06 smithi060 ceph-mon[32745]: [04/Mar/2024:12:16:05] ENGINE Serving on https://172.21.15.60:7150 2024-03-04T12:16:06.220 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:06 smithi060 ceph-mon[32745]: [04/Mar/2024:12:16:05] ENGINE Serving on http://172.21.15.60:8765 2024-03-04T12:16:06.220 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:06 smithi060 ceph-mon[32745]: [04/Mar/2024:12:16:05] ENGINE Bus STARTED 2024-03-04T12:16:07.064 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-04T12:16:07.064 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.234+0000 7fda1b5f2700 1 Processor -- start 2024-03-04T12:16:07.064 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.234+0000 7fda1b5f2700 1 -- start start 2024-03-04T12:16:07.064 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.234+0000 7fda1b5f2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 0x7fda14106ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:07.064 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.234+0000 7fda1b5f2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda141070a0 con 0x7fda141066b0 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.234+0000 7fda1938e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 0x7fda14106ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.234+0000 7fda1938e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 0x7fda14106ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58256/0 (socket says 172.21.15.60:58256) 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.234+0000 7fda1938e700 1 -- 172.21.15.60:0/4029127723 learned_addr learned my addr 172.21.15.60:0/4029127723 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.234+0000 7fda1938e700 1 -- 172.21.15.60:0/4029127723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda141078b0 con 0x7fda141066b0 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.235+0000 7fda1938e700 1 --2- 172.21.15.60:0/4029127723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 0x7fda14106ad0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009e80 tx=0x7fda0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2e59985341bcc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.235+0000 7fda13fff700 1 -- 172.21.15.60:0/4029127723 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda0c017070 con 0x7fda141066b0 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.235+0000 7fda13fff700 1 -- 172.21.15.60:0/4029127723 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda0c00b8c0 con 0x7fda141066b0 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.235+0000 7fda13fff700 1 -- 172.21.15.60:0/4029127723 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda0c0045c0 con 0x7fda141066b0 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.235+0000 7fda1b5f2700 1 -- 172.21.15.60:0/4029127723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 msgr2=0x7fda14106ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.235+0000 7fda1b5f2700 1 --2- 172.21.15.60:0/4029127723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 0x7fda14106ad0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009e80 tx=0x7fda0c00b330 comp rx=0 tx=0).stop 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.235+0000 7fda1b5f2700 1 -- 172.21.15.60:0/4029127723 shutdown_connections 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.235+0000 7fda1b5f2700 1 --2- 172.21.15.60:0/4029127723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 0x7fda14106ad0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.235+0000 7fda1b5f2700 1 -- 172.21.15.60:0/4029127723 >> 172.21.15.60:0/4029127723 conn(0x7fda14101cb0 msgr2=0x7fda141040d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:07.065 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.236+0000 7fda1b5f2700 1 -- 172.21.15.60:0/4029127723 shutdown_connections 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.236+0000 7fda1b5f2700 1 -- 172.21.15.60:0/4029127723 wait complete. 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.236+0000 7fda1b5f2700 1 Processor -- start 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.236+0000 7fda1b5f2700 1 -- start start 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.236+0000 7fda1b5f2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 0x7fda1419b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.236+0000 7fda1b5f2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1410ac60 con 0x7fda141066b0 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.236+0000 7fda1938e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 0x7fda1419b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.236+0000 7fda1938e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 0x7fda1419b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58266/0 (socket says 172.21.15.60:58266) 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.236+0000 7fda1938e700 1 -- 172.21.15.60:0/2748637981 learned_addr learned my addr 172.21.15.60:0/2748637981 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.237+0000 7fda1938e700 1 -- 172.21.15.60:0/2748637981 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1419bd60 con 0x7fda141066b0 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.237+0000 7fda1938e700 1 --2- 172.21.15.60:0/2748637981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 0x7fda1419b820 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda0c003fa0 tx=0x7fda0c004080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.237+0000 7fda127fc700 1 -- 172.21.15.60:0/2748637981 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda0c020020 con 0x7fda141066b0 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.237+0000 7fda1b5f2700 1 -- 172.21.15.60:0/2748637981 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1419bff0 con 0x7fda141066b0 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.237+0000 7fda127fc700 1 -- 172.21.15.60:0/2748637981 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda0c00bba0 con 0x7fda141066b0 2024-03-04T12:16:07.066 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.237+0000 7fda127fc700 1 -- 172.21.15.60:0/2748637981 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda0c017040 con 0x7fda141066b0 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.237+0000 7fda1b5f2700 1 -- 172.21.15.60:0/2748637981 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda1419ec90 con 0x7fda141066b0 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.238+0000 7fda127fc700 1 -- 172.21.15.60:0/2748637981 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 45940+0+0 (secure 0 0 0) 0x7fda0c023020 con 0x7fda141066b0 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.238+0000 7fda1b5f2700 1 -- 172.21.15.60:0/2748637981 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda14068f00 con 0x7fda141066b0 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.238+0000 7fda127fc700 1 --2- 172.21.15.60:0/2748637981 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9fc038ba0 0x7fd9fc03b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.238+0000 7fda127fc700 1 -- 172.21.15.60:0/2748637981 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fda0c00f050 con 0x7fda141066b0 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.238+0000 7fda18b8d700 1 --2- 172.21.15.60:0/2748637981 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9fc038ba0 0x7fd9fc03b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.239+0000 7fda18b8d700 1 --2- 172.21.15.60:0/2748637981 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9fc038ba0 0x7fd9fc03b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fda08006fd0 tx=0x7fda08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.243+0000 7fda127fc700 1 -- 172.21.15.60:0/2748637981 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fda0c04ee20 con 0x7fda141066b0 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.457+0000 7fda1b5f2700 1 -- 172.21.15.60:0/2748637981 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fda1402db30 con 0x7fd9fc038ba0 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.513+0000 7fda127fc700 1 -- 172.21.15.60:0/2748637981 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fda1402db30 con 0x7fd9fc038ba0 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.516+0000 7fda1b5f2700 1 -- 172.21.15.60:0/2748637981 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9fc038ba0 msgr2=0x7fd9fc03b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.516+0000 7fda1b5f2700 1 --2- 172.21.15.60:0/2748637981 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9fc038ba0 0x7fd9fc03b060 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fda08006fd0 tx=0x7fda08006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.516+0000 7fda1b5f2700 1 -- 172.21.15.60:0/2748637981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 msgr2=0x7fda1419b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:07.067 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.516+0000 7fda1b5f2700 1 --2- 172.21.15.60:0/2748637981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 0x7fda1419b820 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fda0c003fa0 tx=0x7fda0c004080 comp rx=0 tx=0).stop 2024-03-04T12:16:07.068 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.516+0000 7fda1b5f2700 1 -- 172.21.15.60:0/2748637981 shutdown_connections 2024-03-04T12:16:07.068 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.516+0000 7fda1b5f2700 1 --2- 172.21.15.60:0/2748637981 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9fc038ba0 0x7fd9fc03b060 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:07.068 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.516+0000 7fda1b5f2700 1 --2- 172.21.15.60:0/2748637981 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda141066b0 0x7fda1419b820 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:07.068 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.516+0000 7fda1b5f2700 1 -- 172.21.15.60:0/2748637981 >> 172.21.15.60:0/2748637981 conn(0x7fda14101cb0 msgr2=0x7fda14102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:07.068 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.516+0000 7fda1b5f2700 1 -- 172.21.15.60:0/2748637981 shutdown_connections 2024-03-04T12:16:07.068 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:06.516+0000 7fda1b5f2700 1 -- 172.21.15.60:0/2748637981 wait complete. 2024-03-04T12:16:07.068 INFO:teuthology.orchestra.run.smithi060.stdout:Creating initial admin user... 2024-03-04T12:16:07.335 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:07 smithi060 ceph-mon[32745]: mgrmap e12: x(active, since 2s) 2024-03-04T12:16:07.335 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:07 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:07.335 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:07 smithi060 ceph-mon[32745]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:16:07.335 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:07 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:07.335 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:07 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:08.464 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$vh8SgWWEiqolqfmmbEg38.602X/DxzHQhcvpdw5NESNY8WJ16WWp6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1709554567, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-04T12:16:08.464 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.436+0000 7fb207b2d700 1 Processor -- start 2024-03-04T12:16:08.464 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.437+0000 7fb207b2d700 1 -- start start 2024-03-04T12:16:08.464 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.437+0000 7fb207b2d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 0x7fb200105ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.437+0000 7fb207b2d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb200106470 con 0x7fb200105a80 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.437+0000 7fb2058c9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 0x7fb200105ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.437+0000 7fb2058c9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 0x7fb200105ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58274/0 (socket says 172.21.15.60:58274) 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.437+0000 7fb2058c9700 1 -- 172.21.15.60:0/2180608438 learned_addr learned my addr 172.21.15.60:0/2180608438 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.437+0000 7fb2058c9700 1 -- 172.21.15.60:0/2180608438 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb200108a60 con 0x7fb200105a80 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.437+0000 7fb2058c9700 1 --2- 172.21.15.60:0/2180608438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 0x7fb200105ea0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009e80 tx=0x7fb1f000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a007619989167436 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.438+0000 7fb2048c7700 1 -- 172.21.15.60:0/2180608438 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1f0017070 con 0x7fb200105a80 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.438+0000 7fb2048c7700 1 -- 172.21.15.60:0/2180608438 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1f000b8c0 con 0x7fb200105a80 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.438+0000 7fb2048c7700 1 -- 172.21.15.60:0/2180608438 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1f00045c0 con 0x7fb200105a80 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.438+0000 7fb207b2d700 1 -- 172.21.15.60:0/2180608438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 msgr2=0x7fb200105ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.438+0000 7fb207b2d700 1 --2- 172.21.15.60:0/2180608438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 0x7fb200105ea0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009e80 tx=0x7fb1f000b330 comp rx=0 tx=0).stop 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.438+0000 7fb207b2d700 1 -- 172.21.15.60:0/2180608438 shutdown_connections 2024-03-04T12:16:08.465 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.438+0000 7fb207b2d700 1 --2- 172.21.15.60:0/2180608438 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 0x7fb200105ea0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.438+0000 7fb207b2d700 1 -- 172.21.15.60:0/2180608438 >> 172.21.15.60:0/2180608438 conn(0x7fb2001014a0 msgr2=0x7fb2001038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.438+0000 7fb207b2d700 1 -- 172.21.15.60:0/2180608438 shutdown_connections 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.438+0000 7fb207b2d700 1 -- 172.21.15.60:0/2180608438 wait complete. 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.439+0000 7fb207b2d700 1 Processor -- start 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.439+0000 7fb207b2d700 1 -- start start 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.439+0000 7fb207b2d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 0x7fb20019b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.439+0000 7fb207b2d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20010be10 con 0x7fb200105a80 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.439+0000 7fb2058c9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 0x7fb20019b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.439+0000 7fb2058c9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 0x7fb20019b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58290/0 (socket says 172.21.15.60:58290) 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.439+0000 7fb2058c9700 1 -- 172.21.15.60:0/2382380079 learned_addr learned my addr 172.21.15.60:0/2382380079 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.439+0000 7fb2058c9700 1 -- 172.21.15.60:0/2382380079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20019bd20 con 0x7fb200105a80 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.440+0000 7fb2058c9700 1 --2- 172.21.15.60:0/2382380079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 0x7fb20019b7e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009530 tx=0x7fb1f0003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.440+0000 7fb1feffd700 1 -- 172.21.15.60:0/2382380079 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1f0020020 con 0x7fb200105a80 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.440+0000 7fb1feffd700 1 -- 172.21.15.60:0/2382380079 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1f000bba0 con 0x7fb200105a80 2024-03-04T12:16:08.466 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.440+0000 7fb207b2d700 1 -- 172.21.15.60:0/2382380079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20019bfb0 con 0x7fb200105a80 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.440+0000 7fb1feffd700 1 -- 172.21.15.60:0/2382380079 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1f0017040 con 0x7fb200105a80 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.440+0000 7fb207b2d700 1 -- 172.21.15.60:0/2382380079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20019eca0 con 0x7fb200105a80 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.441+0000 7fb1feffd700 1 -- 172.21.15.60:0/2382380079 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 45940+0+0 (secure 0 0 0) 0x7fb1f0023020 con 0x7fb200105a80 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.441+0000 7fb207b2d700 1 -- 172.21.15.60:0/2382380079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb200068f00 con 0x7fb200105a80 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.441+0000 7fb1feffd700 1 --2- 172.21.15.60:0/2382380079 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb1e8038bf0 0x7fb1e803b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.441+0000 7fb1feffd700 1 -- 172.21.15.60:0/2382380079 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb1f000f050 con 0x7fb200105a80 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.441+0000 7fb2050c8700 1 --2- 172.21.15.60:0/2382380079 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb1e8038bf0 0x7fb1e803b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.441+0000 7fb2050c8700 1 --2- 172.21.15.60:0/2382380079 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb1e8038bf0 0x7fb1e803b0b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4006fd0 tx=0x7fb1f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.446+0000 7fb1feffd700 1 -- 172.21.15.60:0/2382380079 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb1f002a020 con 0x7fb200105a80 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.669+0000 7fb207b2d700 1 -- 172.21.15.60:0/2382380079 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb200102740 con 0x7fb1e8038bf0 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.916+0000 7fb1feffd700 1 -- 172.21.15.60:0/2382380079 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fb200102740 con 0x7fb1e8038bf0 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.918+0000 7fb207b2d700 1 -- 172.21.15.60:0/2382380079 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb1e8038bf0 msgr2=0x7fb1e803b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.918+0000 7fb207b2d700 1 --2- 172.21.15.60:0/2382380079 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb1e8038bf0 0x7fb1e803b0b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4006fd0 tx=0x7fb1f4006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:08.467 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.918+0000 7fb207b2d700 1 -- 172.21.15.60:0/2382380079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 msgr2=0x7fb20019b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:08.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.918+0000 7fb207b2d700 1 --2- 172.21.15.60:0/2382380079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 0x7fb20019b7e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009530 tx=0x7fb1f0003d00 comp rx=0 tx=0).stop 2024-03-04T12:16:08.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.918+0000 7fb207b2d700 1 -- 172.21.15.60:0/2382380079 shutdown_connections 2024-03-04T12:16:08.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.918+0000 7fb207b2d700 1 --2- 172.21.15.60:0/2382380079 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb1e8038bf0 0x7fb1e803b0b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:08.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.918+0000 7fb207b2d700 1 --2- 172.21.15.60:0/2382380079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb200105a80 0x7fb20019b7e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:08.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.918+0000 7fb207b2d700 1 -- 172.21.15.60:0/2382380079 >> 172.21.15.60:0/2382380079 conn(0x7fb2001014a0 msgr2=0x7fb200102030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:08.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.918+0000 7fb207b2d700 1 -- 172.21.15.60:0/2382380079 shutdown_connections 2024-03-04T12:16:08.468 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:07.918+0000 7fb207b2d700 1 -- 172.21.15.60:0/2382380079 wait complete. 2024-03-04T12:16:08.468 INFO:teuthology.orchestra.run.smithi060.stdout:Fetching dashboard port number... 2024-03-04T12:16:08.993 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:08 smithi060 ceph-mon[32745]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:16:08.993 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:08 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:08.993 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:08 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:09.697 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stdout 8443 2024-03-04T12:16:09.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.831+0000 7f16a083c700 1 Processor -- start 2024-03-04T12:16:09.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.831+0000 7f16a083c700 1 -- start start 2024-03-04T12:16:09.699 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.831+0000 7f16a083c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 0x7f169c105ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.831+0000 7f16a083c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c1064b0 con 0x7f169c105ac0 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.831+0000 7f169a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 0x7f169c105ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.831+0000 7f169a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 0x7f169c105ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58292/0 (socket says 172.21.15.60:58292) 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.831+0000 7f169a59c700 1 -- 172.21.15.60:0/2558852271 learned_addr learned my addr 172.21.15.60:0/2558852271 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.831+0000 7f169a59c700 1 -- 172.21.15.60:0/2558852271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c106630 con 0x7f169c105ac0 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.832+0000 7f169a59c700 1 --2- 172.21.15.60:0/2558852271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 0x7f169c105ee0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f168c0098a0 tx=0x7f168c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf062f9462258f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.832+0000 7f169959a700 1 -- 172.21.15.60:0/2558852271 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f168c004160 con 0x7f169c105ac0 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.832+0000 7f169959a700 1 -- 172.21.15.60:0/2558852271 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f168c004300 con 0x7f169c105ac0 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.832+0000 7f169959a700 1 -- 172.21.15.60:0/2558852271 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f168c011b20 con 0x7f169c105ac0 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.832+0000 7f16a083c700 1 -- 172.21.15.60:0/2558852271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 msgr2=0x7f169c105ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.832+0000 7f16a083c700 1 --2- 172.21.15.60:0/2558852271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 0x7f169c105ee0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f168c0098a0 tx=0x7f168c009bb0 comp rx=0 tx=0).stop 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.832+0000 7f16a083c700 1 -- 172.21.15.60:0/2558852271 shutdown_connections 2024-03-04T12:16:09.700 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.832+0000 7f16a083c700 1 --2- 172.21.15.60:0/2558852271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 0x7f169c105ee0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.832+0000 7f16a083c700 1 -- 172.21.15.60:0/2558852271 >> 172.21.15.60:0/2558852271 conn(0x7f169c101500 msgr2=0x7f169c103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.833+0000 7f16a083c700 1 -- 172.21.15.60:0/2558852271 shutdown_connections 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.833+0000 7f16a083c700 1 -- 172.21.15.60:0/2558852271 wait complete. 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.833+0000 7f16a083c700 1 Processor -- start 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.833+0000 7f16a083c700 1 -- start start 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.833+0000 7f16a083c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 0x7f169c1083c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.833+0000 7f16a083c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c10bd70 con 0x7f169c105ac0 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.834+0000 7f169a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 0x7f169c1083c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.834+0000 7f169a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 0x7f169c1083c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58302/0 (socket says 172.21.15.60:58302) 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.834+0000 7f169a59c700 1 -- 172.21.15.60:0/2887007546 learned_addr learned my addr 172.21.15.60:0/2887007546 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.834+0000 7f169a59c700 1 -- 172.21.15.60:0/2887007546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c108900 con 0x7f169c105ac0 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.834+0000 7f169a59c700 1 --2- 172.21.15.60:0/2887007546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 0x7f169c1083c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f168c0117e0 tx=0x7f168c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:09.701 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.834+0000 7f168b7fe700 1 -- 172.21.15.60:0/2887007546 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f168c003bf0 con 0x7f169c105ac0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.834+0000 7f168b7fe700 1 -- 172.21.15.60:0/2887007546 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f168c01a910 con 0x7f169c105ac0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.834+0000 7f16a083c700 1 -- 172.21.15.60:0/2887007546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169c106af0 con 0x7f169c105ac0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.834+0000 7f168b7fe700 1 -- 172.21.15.60:0/2887007546 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f168c0249a0 con 0x7f169c105ac0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.834+0000 7f16a083c700 1 -- 172.21.15.60:0/2887007546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169c106fb0 con 0x7f169c105ac0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.835+0000 7f16a083c700 1 -- 172.21.15.60:0/2887007546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169c063450 con 0x7f169c105ac0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.835+0000 7f168b7fe700 1 -- 172.21.15.60:0/2887007546 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 45940+0+0 (secure 0 0 0) 0x7f168c022020 con 0x7f169c105ac0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.836+0000 7f168b7fe700 1 --2- 172.21.15.60:0/2887007546 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1684038c00 0x7f168403b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.836+0000 7f168b7fe700 1 -- 172.21.15.60:0/2887007546 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f168c04cfa0 con 0x7f169c105ac0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.836+0000 7f1699d9b700 1 --2- 172.21.15.60:0/2887007546 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1684038c00 0x7f168403b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.836+0000 7f1699d9b700 1 --2- 172.21.15.60:0/2887007546 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1684038c00 0x7f168403b0c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1690006fd0 tx=0x7f1690006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:08.842+0000 7f168b7fe700 1 -- 172.21.15.60:0/2887007546 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f168c011e20 con 0x7f169c105ac0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:09.056+0000 7f16a083c700 1 -- 172.21.15.60:0/2887007546 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f169c1078b0 con 0x7f169c105ac0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:09.056+0000 7f168b7fe700 1 -- 172.21.15.60:0/2887007546 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f168c019c70 con 0x7f169c105ac0 2024-03-04T12:16:09.702 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:09.059+0000 7f16a083c700 1 -- 172.21.15.60:0/2887007546 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1684038c00 msgr2=0x7f168403b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:09.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:09.059+0000 7f16a083c700 1 --2- 172.21.15.60:0/2887007546 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1684038c00 0x7f168403b0c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1690006fd0 tx=0x7f1690006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:09.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:09.059+0000 7f16a083c700 1 -- 172.21.15.60:0/2887007546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 msgr2=0x7f169c1083c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:09.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:09.059+0000 7f16a083c700 1 --2- 172.21.15.60:0/2887007546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 0x7f169c1083c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f168c0117e0 tx=0x7f168c011c10 comp rx=0 tx=0).stop 2024-03-04T12:16:09.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:09.059+0000 7f16a083c700 1 -- 172.21.15.60:0/2887007546 shutdown_connections 2024-03-04T12:16:09.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:09.059+0000 7f16a083c700 1 --2- 172.21.15.60:0/2887007546 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1684038c00 0x7f168403b0c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:09.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:09.059+0000 7f16a083c700 1 --2- 172.21.15.60:0/2887007546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f169c105ac0 0x7f169c1083c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:09.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:09.059+0000 7f16a083c700 1 -- 172.21.15.60:0/2887007546 >> 172.21.15.60:0/2887007546 conn(0x7f169c101500 msgr2=0x7f169c101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:09.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:09.059+0000 7f16a083c700 1 -- 172.21.15.60:0/2887007546 shutdown_connections 2024-03-04T12:16:09.703 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:09.059+0000 7f16a083c700 1 -- 172.21.15.60:0/2887007546 wait complete. 2024-03-04T12:16:09.704 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T12:16:09.704 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout disabled 2024-03-04T12:16:09.711 INFO:teuthology.orchestra.run.smithi060.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T12:16:09.711 INFO:teuthology.orchestra.run.smithi060.stdout:systemctl: stdout inactive 2024-03-04T12:16:09.711 INFO:teuthology.orchestra.run.smithi060.stdout:firewalld.service is not enabled 2024-03-04T12:16:09.711 INFO:teuthology.orchestra.run.smithi060.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-04T12:16:09.713 INFO:teuthology.orchestra.run.smithi060.stdout:Ceph Dashboard is now available at: 2024-03-04T12:16:09.713 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:16:09.713 INFO:teuthology.orchestra.run.smithi060.stdout: URL: https://smithi060.front.sepia.ceph.com:8443/ 2024-03-04T12:16:09.714 INFO:teuthology.orchestra.run.smithi060.stdout: User: admin 2024-03-04T12:16:09.714 INFO:teuthology.orchestra.run.smithi060.stdout: Password: p4ln2fqh3k 2024-03-04T12:16:09.714 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:16:09.714 INFO:teuthology.orchestra.run.smithi060.stdout:Saving cluster configuration to /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config directory 2024-03-04T12:16:09.714 INFO:teuthology.orchestra.run.smithi060.stdout:Enabling autotune for osd_memory_target 2024-03-04T12:16:10.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:09 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2887007546' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-04T12:16:11.442 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.594+0000 7f1405c0b700 1 Processor -- start 2024-03-04T12:16:11.442 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.595+0000 7f1405c0b700 1 -- start start 2024-03-04T12:16:11.442 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.595+0000 7f1405c0b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 0x7f1400107c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:11.442 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.595+0000 7f1405c0b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1400108250 con 0x7f1400107860 2024-03-04T12:16:11.442 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.595+0000 7f13ff7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 0x7f1400107c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:11.442 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.595+0000 7f13ff7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 0x7f1400107c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58306/0 (socket says 172.21.15.60:58306) 2024-03-04T12:16:11.442 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.595+0000 7f13ff7fe700 1 -- 172.21.15.60:0/4277252584 learned_addr learned my addr 172.21.15.60:0/4277252584 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:11.442 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.595+0000 7f13ff7fe700 1 -- 172.21.15.60:0/4277252584 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1400108a10 con 0x7f1400107860 2024-03-04T12:16:11.442 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.595+0000 7f13ff7fe700 1 --2- 172.21.15.60:0/4277252584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 0x7f1400107c80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f13f00098a0 tx=0x7f13f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e114f853cb263ba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.596+0000 7f13fe7fc700 1 -- 172.21.15.60:0/4277252584 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13f0004160 con 0x7f1400107860 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.596+0000 7f13fe7fc700 1 -- 172.21.15.60:0/4277252584 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13f0004300 con 0x7f1400107860 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.596+0000 7f13fe7fc700 1 -- 172.21.15.60:0/4277252584 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13f0011b20 con 0x7f1400107860 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.596+0000 7f1405c0b700 1 -- 172.21.15.60:0/4277252584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 msgr2=0x7f1400107c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.596+0000 7f1405c0b700 1 --2- 172.21.15.60:0/4277252584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 0x7f1400107c80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f13f00098a0 tx=0x7f13f0009bb0 comp rx=0 tx=0).stop 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.596+0000 7f1405c0b700 1 -- 172.21.15.60:0/4277252584 shutdown_connections 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.596+0000 7f1405c0b700 1 --2- 172.21.15.60:0/4277252584 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 0x7f1400107c80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.596+0000 7f1405c0b700 1 -- 172.21.15.60:0/4277252584 >> 172.21.15.60:0/4277252584 conn(0x7f1400102ee0 msgr2=0x7f1400105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.596+0000 7f1405c0b700 1 -- 172.21.15.60:0/4277252584 shutdown_connections 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.596+0000 7f1405c0b700 1 -- 172.21.15.60:0/4277252584 wait complete. 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.597+0000 7f1405c0b700 1 Processor -- start 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.597+0000 7f1405c0b700 1 -- start start 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.597+0000 7f1405c0b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 0x7f140019b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:11.443 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.597+0000 7f1405c0b700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140010bdc0 con 0x7f1400107860 2024-03-04T12:16:11.444 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.597+0000 7f13ff7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 0x7f140019b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:11.444 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.597+0000 7f13ff7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 0x7f140019b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58322/0 (socket says 172.21.15.60:58322) 2024-03-04T12:16:11.444 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.597+0000 7f13ff7fe700 1 -- 172.21.15.60:0/1665576628 learned_addr learned my addr 172.21.15.60:0/1665576628 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:11.444 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.598+0000 7f13ff7fe700 1 -- 172.21.15.60:0/1665576628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140019bcc0 con 0x7f1400107860 2024-03-04T12:16:11.444 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.598+0000 7f13ff7fe700 1 --2- 172.21.15.60:0/1665576628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 0x7f140019b780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f13f00117e0 tx=0x7f13f0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:11.444 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.598+0000 7f13fcff9700 1 -- 172.21.15.60:0/1665576628 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13f0003bf0 con 0x7f1400107860 2024-03-04T12:16:11.444 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.598+0000 7f13fcff9700 1 -- 172.21.15.60:0/1665576628 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13f001a910 con 0x7f1400107860 2024-03-04T12:16:11.444 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.598+0000 7f1405c0b700 1 -- 172.21.15.60:0/1665576628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f140019bf50 con 0x7f1400107860 2024-03-04T12:16:11.444 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.598+0000 7f13fcff9700 1 -- 172.21.15.60:0/1665576628 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13f00249a0 con 0x7f1400107860 2024-03-04T12:16:11.444 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.598+0000 7f1405c0b700 1 -- 172.21.15.60:0/1665576628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140019c3b0 con 0x7f1400107860 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.599+0000 7f13fcff9700 1 -- 172.21.15.60:0/1665576628 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 12) v1 ==== 45940+0+0 (secure 0 0 0) 0x7f13f0022020 con 0x7f1400107860 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.599+0000 7f13fcff9700 1 --2- 172.21.15.60:0/1665576628 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f13e8038c00 0x7f13e803b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.599+0000 7f1405c0b700 1 -- 172.21.15.60:0/1665576628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1400063450 con 0x7f1400107860 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.599+0000 7f13fcff9700 1 -- 172.21.15.60:0/1665576628 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f13f004cde0 con 0x7f1400107860 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.599+0000 7f13feffd700 1 --2- 172.21.15.60:0/1665576628 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f13e8038c00 0x7f13e803b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.600+0000 7f13feffd700 1 --2- 172.21.15.60:0/1665576628 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f13e8038c00 0x7f13e803b0c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f13f4006fd0 tx=0x7f13f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.604+0000 7f13fcff9700 1 -- 172.21.15.60:0/1665576628 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f13f0011e20 con 0x7f1400107860 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.807+0000 7f1405c0b700 1 -- 172.21.15.60:0/1665576628 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f140019edf0 con 0x7f1400107860 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.807+0000 7f13fcff9700 1 -- 172.21.15.60:0/1665576628 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f13f0019ca0 con 0x7f1400107860 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.809+0000 7f1405c0b700 1 -- 172.21.15.60:0/1665576628 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f13e8038c00 msgr2=0x7f13e803b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.809+0000 7f1405c0b700 1 --2- 172.21.15.60:0/1665576628 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f13e8038c00 0x7f13e803b0c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f13f4006fd0 tx=0x7f13f4006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.809+0000 7f1405c0b700 1 -- 172.21.15.60:0/1665576628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 msgr2=0x7f140019b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.809+0000 7f1405c0b700 1 --2- 172.21.15.60:0/1665576628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 0x7f140019b780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f13f00117e0 tx=0x7f13f0011c10 comp rx=0 tx=0).stop 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.809+0000 7f1405c0b700 1 -- 172.21.15.60:0/1665576628 shutdown_connections 2024-03-04T12:16:11.445 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.809+0000 7f1405c0b700 1 --2- 172.21.15.60:0/1665576628 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f13e8038c00 0x7f13e803b0c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:11.446 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.809+0000 7f1405c0b700 1 --2- 172.21.15.60:0/1665576628 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1400107860 0x7f140019b780 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:11.446 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.809+0000 7f1405c0b700 1 -- 172.21.15.60:0/1665576628 >> 172.21.15.60:0/1665576628 conn(0x7f1400102ee0 msgr2=0x7f1400103910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:11.446 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.809+0000 7f1405c0b700 1 -- 172.21.15.60:0/1665576628 shutdown_connections 2024-03-04T12:16:11.446 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:10.809+0000 7f1405c0b700 1 -- 172.21.15.60:0/1665576628 wait complete. 2024-03-04T12:16:12.121 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:11 smithi060 ceph-mon[32745]: mgrmap e13: x(active, since 6s) 2024-03-04T12:16:12.707 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.846+0000 7faa1f407700 1 Processor -- start 2024-03-04T12:16:12.707 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.846+0000 7faa1f407700 1 -- start start 2024-03-04T12:16:12.707 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.846+0000 7faa1f407700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 0x7faa18107c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:12.707 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.846+0000 7faa1f407700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa181081d0 con 0x7faa181077e0 2024-03-04T12:16:12.707 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.847+0000 7faa1d1a3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 0x7faa18107c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:12.707 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.847+0000 7faa1d1a3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 0x7faa18107c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58332/0 (socket says 172.21.15.60:58332) 2024-03-04T12:16:12.707 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.847+0000 7faa1d1a3700 1 -- 172.21.15.60:0/3094697458 learned_addr learned my addr 172.21.15.60:0/3094697458 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:12.707 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.847+0000 7faa1d1a3700 1 -- 172.21.15.60:0/3094697458 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa18108990 con 0x7faa181077e0 2024-03-04T12:16:12.707 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.847+0000 7faa1d1a3700 1 --2- 172.21.15.60:0/3094697458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 0x7faa18107c00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faa080098a0 tx=0x7faa08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32dcf04dc307d7e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.847+0000 7faa0ffff700 1 -- 172.21.15.60:0/3094697458 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa08004160 con 0x7faa181077e0 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.847+0000 7faa0ffff700 1 -- 172.21.15.60:0/3094697458 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa08004300 con 0x7faa181077e0 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.847+0000 7faa0ffff700 1 -- 172.21.15.60:0/3094697458 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa08011b20 con 0x7faa181077e0 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.848+0000 7faa1f407700 1 -- 172.21.15.60:0/3094697458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 msgr2=0x7faa18107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.848+0000 7faa1f407700 1 --2- 172.21.15.60:0/3094697458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 0x7faa18107c00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faa080098a0 tx=0x7faa08009bb0 comp rx=0 tx=0).stop 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.848+0000 7faa1f407700 1 -- 172.21.15.60:0/3094697458 shutdown_connections 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.848+0000 7faa1f407700 1 --2- 172.21.15.60:0/3094697458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 0x7faa18107c00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.848+0000 7faa1f407700 1 -- 172.21.15.60:0/3094697458 >> 172.21.15.60:0/3094697458 conn(0x7faa18102e20 msgr2=0x7faa18105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.848+0000 7faa1f407700 1 -- 172.21.15.60:0/3094697458 shutdown_connections 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.848+0000 7faa1f407700 1 -- 172.21.15.60:0/3094697458 wait complete. 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.849+0000 7faa1f407700 1 Processor -- start 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.849+0000 7faa1f407700 1 -- start start 2024-03-04T12:16:12.708 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.849+0000 7faa1f407700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 0x7faa1819b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.849+0000 7faa1f407700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1810bd40 con 0x7faa181077e0 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.849+0000 7faa1d1a3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 0x7faa1819b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.849+0000 7faa1d1a3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 0x7faa1819b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58346/0 (socket says 172.21.15.60:58346) 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.849+0000 7faa1d1a3700 1 -- 172.21.15.60:0/3920916502 learned_addr learned my addr 172.21.15.60:0/3920916502 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.849+0000 7faa1d1a3700 1 -- 172.21.15.60:0/3920916502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1819bc40 con 0x7faa181077e0 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.849+0000 7faa1d1a3700 1 --2- 172.21.15.60:0/3920916502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 0x7faa1819b700 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faa080117e0 tx=0x7faa08011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.849+0000 7faa0e7fc700 1 -- 172.21.15.60:0/3920916502 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa08003bf0 con 0x7faa181077e0 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.850+0000 7faa0e7fc700 1 -- 172.21.15.60:0/3920916502 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa0801a910 con 0x7faa181077e0 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.850+0000 7faa1f407700 1 -- 172.21.15.60:0/3920916502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1819bed0 con 0x7faa181077e0 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.850+0000 7faa0e7fc700 1 -- 172.21.15.60:0/3920916502 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa080249a0 con 0x7faa181077e0 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.850+0000 7faa1f407700 1 -- 172.21.15.60:0/3920916502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1819c330 con 0x7faa181077e0 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.850+0000 7faa0e7fc700 1 -- 172.21.15.60:0/3920916502 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7faa08022020 con 0x7faa181077e0 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.851+0000 7faa0e7fc700 1 --2- 172.21.15.60:0/3920916502 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7faa04038c40 0x7faa0403b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:12.709 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.851+0000 7faa0e7fc700 1 -- 172.21.15.60:0/3920916502 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faa0804d480 con 0x7faa181077e0 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.851+0000 7faa1c9a2700 1 --2- 172.21.15.60:0/3920916502 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7faa04038c40 0x7faa0403b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.851+0000 7faa1f407700 1 -- 172.21.15.60:0/3920916502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa18063450 con 0x7faa181077e0 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.851+0000 7faa1c9a2700 1 --2- 172.21.15.60:0/3920916502 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7faa04038c40 0x7faa0403b100 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faa14006fd0 tx=0x7faa14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:11.856+0000 7faa0e7fc700 1 -- 172.21.15.60:0/3920916502 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faa08011e20 con 0x7faa181077e0 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:12.176+0000 7faa1f407700 1 -- 172.21.15.60:0/3920916502 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7faa1819ec60 con 0x7faa181077e0 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:12.178+0000 7faa0e7fc700 1 -- 172.21.15.60:0/3920916502 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7faa0802c350 con 0x7faa181077e0 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:12.180+0000 7faa1f407700 1 -- 172.21.15.60:0/3920916502 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7faa04038c40 msgr2=0x7faa0403b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:12.180+0000 7faa1f407700 1 --2- 172.21.15.60:0/3920916502 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7faa04038c40 0x7faa0403b100 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faa14006fd0 tx=0x7faa14006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:12.180+0000 7faa1f407700 1 -- 172.21.15.60:0/3920916502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 msgr2=0x7faa1819b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:12.180+0000 7faa1f407700 1 --2- 172.21.15.60:0/3920916502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 0x7faa1819b700 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faa080117e0 tx=0x7faa08011c10 comp rx=0 tx=0).stop 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:12.181+0000 7faa1f407700 1 -- 172.21.15.60:0/3920916502 shutdown_connections 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:12.181+0000 7faa1f407700 1 --2- 172.21.15.60:0/3920916502 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7faa04038c40 0x7faa0403b100 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:12.181+0000 7faa1f407700 1 --2- 172.21.15.60:0/3920916502 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faa181077e0 0x7faa1819b700 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:12.710 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:12.181+0000 7faa1f407700 1 -- 172.21.15.60:0/3920916502 >> 172.21.15.60:0/3920916502 conn(0x7faa18102e20 msgr2=0x7faa181038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:12.181+0000 7faa1f407700 1 -- 172.21.15.60:0/3920916502 shutdown_connections 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr 2024-03-04T12:16:12.181+0000 7faa1f407700 1 -- 172.21.15.60:0/3920916502 wait complete. 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout:Or, if you are only running a single cluster on this host: 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:16:12.711 INFO:teuthology.orchestra.run.smithi060.stdout: ceph telemetry on 2024-03-04T12:16:12.712 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:16:12.712 INFO:teuthology.orchestra.run.smithi060.stdout:For more information see: 2024-03-04T12:16:12.712 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:16:12.712 INFO:teuthology.orchestra.run.smithi060.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-04T12:16:12.712 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:16:12.712 INFO:teuthology.orchestra.run.smithi060.stdout:Bootstrap complete. 2024-03-04T12:16:12.745 INFO:tasks.cephadm:Fetching config... 2024-03-04T12:16:12.745 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:16:12.746 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-04T12:16:12.760 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-04T12:16:12.761 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:16:12.761 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-04T12:16:12.816 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-04T12:16:12.816 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:16:12.816 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-04T12:16:12.882 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-04T12:16:12.883 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:16:12.883 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-04T12:16:12.937 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-04T12:16:12.938 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEXcriikNS2A5qq16UStXqFx7CW9PEeMqaT0s/JJq3DhJ6xHRlBIWzQliBPMAertGCATGk56/7z0FUX7qzUcoTpYHRwzdNVhpusBklZu7U+ymqmEFLWbSuqHKAmnt1xo5nk08ebsyLNewlnns5/A8/oO5XQqX9B+DyRjR2jSL3eDKuYgyP/QRuUcV7Wn2aI7dO53Dd+hGwmkp1gTBpv18t+5fd/vI81CJgT/9ry+KbouSVY2QOw7she8cW+VtB6IUOr/TvwgXKJ8sxQAvO7esz5a7lqrW2Pg+IihnLDVvZfIfrP7wkZV65W0neq09carXSb086id9NFJGhdRTVaQhMHDW5EubktMJTlAQSO7dC7PjE8lesIajDGN6ih68LWKCzXoCfJqifpO/UBqRNQ9MURTdDZlVsxTXeSvgdlqU6qCGXb4Mtz1kPrVIEApLmyrVxJAVrROCjUTD8UkxMBiY31SAUGapJPmQ8wNz/DVsUsEU/pJCmOUluVvex+mp9B1M= ceph-a1c49686-da20-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T12:16:13.016 INFO:teuthology.orchestra.run.smithi060.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEXcriikNS2A5qq16UStXqFx7CW9PEeMqaT0s/JJq3DhJ6xHRlBIWzQliBPMAertGCATGk56/7z0FUX7qzUcoTpYHRwzdNVhpusBklZu7U+ymqmEFLWbSuqHKAmnt1xo5nk08ebsyLNewlnns5/A8/oO5XQqX9B+DyRjR2jSL3eDKuYgyP/QRuUcV7Wn2aI7dO53Dd+hGwmkp1gTBpv18t+5fd/vI81CJgT/9ry+KbouSVY2QOw7she8cW+VtB6IUOr/TvwgXKJ8sxQAvO7esz5a7lqrW2Pg+IihnLDVvZfIfrP7wkZV65W0neq09carXSb086id9NFJGhdRTVaQhMHDW5EubktMJTlAQSO7dC7PjE8lesIajDGN6ih68LWKCzXoCfJqifpO/UBqRNQ9MURTdDZlVsxTXeSvgdlqU6qCGXb4Mtz1kPrVIEApLmyrVxJAVrROCjUTD8UkxMBiY31SAUGapJPmQ8wNz/DVsUsEU/pJCmOUluVvex+mp9B1M= ceph-a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:16:13.030 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEXcriikNS2A5qq16UStXqFx7CW9PEeMqaT0s/JJq3DhJ6xHRlBIWzQliBPMAertGCATGk56/7z0FUX7qzUcoTpYHRwzdNVhpusBklZu7U+ymqmEFLWbSuqHKAmnt1xo5nk08ebsyLNewlnns5/A8/oO5XQqX9B+DyRjR2jSL3eDKuYgyP/QRuUcV7Wn2aI7dO53Dd+hGwmkp1gTBpv18t+5fd/vI81CJgT/9ry+KbouSVY2QOw7she8cW+VtB6IUOr/TvwgXKJ8sxQAvO7esz5a7lqrW2Pg+IihnLDVvZfIfrP7wkZV65W0neq09carXSb086id9NFJGhdRTVaQhMHDW5EubktMJTlAQSO7dC7PjE8lesIajDGN6ih68LWKCzXoCfJqifpO/UBqRNQ9MURTdDZlVsxTXeSvgdlqU6qCGXb4Mtz1kPrVIEApLmyrVxJAVrROCjUTD8UkxMBiY31SAUGapJPmQ8wNz/DVsUsEU/pJCmOUluVvex+mp9B1M= ceph-a1c49686-da20-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T12:16:13.084 INFO:teuthology.orchestra.run.smithi063.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEXcriikNS2A5qq16UStXqFx7CW9PEeMqaT0s/JJq3DhJ6xHRlBIWzQliBPMAertGCATGk56/7z0FUX7qzUcoTpYHRwzdNVhpusBklZu7U+ymqmEFLWbSuqHKAmnt1xo5nk08ebsyLNewlnns5/A8/oO5XQqX9B+DyRjR2jSL3eDKuYgyP/QRuUcV7Wn2aI7dO53Dd+hGwmkp1gTBpv18t+5fd/vI81CJgT/9ry+KbouSVY2QOw7she8cW+VtB6IUOr/TvwgXKJ8sxQAvO7esz5a7lqrW2Pg+IihnLDVvZfIfrP7wkZV65W0neq09carXSb086id9NFJGhdRTVaQhMHDW5EubktMJTlAQSO7dC7PjE8lesIajDGN6ih68LWKCzXoCfJqifpO/UBqRNQ9MURTdDZlVsxTXeSvgdlqU6qCGXb4Mtz1kPrVIEApLmyrVxJAVrROCjUTD8UkxMBiY31SAUGapJPmQ8wNz/DVsUsEU/pJCmOUluVvex+mp9B1M= ceph-a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:16:13.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEXcriikNS2A5qq16UStXqFx7CW9PEeMqaT0s/JJq3DhJ6xHRlBIWzQliBPMAertGCATGk56/7z0FUX7qzUcoTpYHRwzdNVhpusBklZu7U+ymqmEFLWbSuqHKAmnt1xo5nk08ebsyLNewlnns5/A8/oO5XQqX9B+DyRjR2jSL3eDKuYgyP/QRuUcV7Wn2aI7dO53Dd+hGwmkp1gTBpv18t+5fd/vI81CJgT/9ry+KbouSVY2QOw7she8cW+VtB6IUOr/TvwgXKJ8sxQAvO7esz5a7lqrW2Pg+IihnLDVvZfIfrP7wkZV65W0neq09carXSb086id9NFJGhdRTVaQhMHDW5EubktMJTlAQSO7dC7PjE8lesIajDGN6ih68LWKCzXoCfJqifpO/UBqRNQ9MURTdDZlVsxTXeSvgdlqU6qCGXb4Mtz1kPrVIEApLmyrVxJAVrROCjUTD8UkxMBiY31SAUGapJPmQ8wNz/DVsUsEU/pJCmOUluVvex+mp9B1M= ceph-a1c49686-da20-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T12:16:13.147 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEXcriikNS2A5qq16UStXqFx7CW9PEeMqaT0s/JJq3DhJ6xHRlBIWzQliBPMAertGCATGk56/7z0FUX7qzUcoTpYHRwzdNVhpusBklZu7U+ymqmEFLWbSuqHKAmnt1xo5nk08ebsyLNewlnns5/A8/oO5XQqX9B+DyRjR2jSL3eDKuYgyP/QRuUcV7Wn2aI7dO53Dd+hGwmkp1gTBpv18t+5fd/vI81CJgT/9ry+KbouSVY2QOw7she8cW+VtB6IUOr/TvwgXKJ8sxQAvO7esz5a7lqrW2Pg+IihnLDVvZfIfrP7wkZV65W0neq09carXSb086id9NFJGhdRTVaQhMHDW5EubktMJTlAQSO7dC7PjE8lesIajDGN6ih68LWKCzXoCfJqifpO/UBqRNQ9MURTdDZlVsxTXeSvgdlqU6qCGXb4Mtz1kPrVIEApLmyrVxJAVrROCjUTD8UkxMBiY31SAUGapJPmQ8wNz/DVsUsEU/pJCmOUluVvex+mp9B1M= ceph-a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:16:13.162 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-04T12:16:13.269 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:13 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3920916502' entity='client.admin' 2024-03-04T12:16:13.406 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:16:16.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.021+0000 7f230c889700 1 -- 172.21.15.60:0/522272615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2308101870 msgr2=0x7f2308101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:16.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.021+0000 7f230c889700 1 --2- 172.21.15.60:0/522272615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2308101870 0x7f2308101c90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009910 tx=0x7f22f8009c20 comp rx=0 tx=0).stop 2024-03-04T12:16:16.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.021+0000 7f230c889700 1 -- 172.21.15.60:0/522272615 shutdown_connections 2024-03-04T12:16:16.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.021+0000 7f230c889700 1 --2- 172.21.15.60:0/522272615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2308101870 0x7f2308101c90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:16.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.021+0000 7f230c889700 1 -- 172.21.15.60:0/522272615 >> 172.21.15.60:0/522272615 conn(0x7f23080fce10 msgr2=0x7f23080ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:16.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.021+0000 7f230c889700 1 -- 172.21.15.60:0/522272615 shutdown_connections 2024-03-04T12:16:16.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.021+0000 7f230c889700 1 -- 172.21.15.60:0/522272615 wait complete. 2024-03-04T12:16:16.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.022+0000 7f230c889700 1 Processor -- start 2024-03-04T12:16:16.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.022+0000 7f230c889700 1 -- start start 2024-03-04T12:16:16.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.022+0000 7f230c889700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2308101870 0x7f2308196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:16.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.022+0000 7f230c889700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2308107900 con 0x7f2308101870 2024-03-04T12:16:16.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.022+0000 7f230659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2308101870 0x7f2308196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:16.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.022+0000 7f230659c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2308101870 0x7f2308196f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42782/0 (socket says 172.21.15.60:42782) 2024-03-04T12:16:16.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.022+0000 7f230659c700 1 -- 172.21.15.60:0/1520092768 learned_addr learned my addr 172.21.15.60:0/1520092768 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.022+0000 7f230659c700 1 -- 172.21.15.60:0/1520092768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23081974c0 con 0x7f2308101870 2024-03-04T12:16:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.023+0000 7f230659c700 1 --2- 172.21.15.60:0/1520092768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2308101870 0x7f2308196f80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f22f8006010 tx=0x7f22f8005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.023+0000 7f22f77fe700 1 -- 172.21.15.60:0/1520092768 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22f8027070 con 0x7f2308101870 2024-03-04T12:16:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.023+0000 7f230c889700 1 -- 172.21.15.60:0/1520092768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2308197750 con 0x7f2308101870 2024-03-04T12:16:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.023+0000 7f22f77fe700 1 -- 172.21.15.60:0/1520092768 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22f800be40 con 0x7f2308101870 2024-03-04T12:16:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.023+0000 7f22f77fe700 1 -- 172.21.15.60:0/1520092768 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22f800f460 con 0x7f2308101870 2024-03-04T12:16:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.023+0000 7f230c889700 1 -- 172.21.15.60:0/1520092768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230819a440 con 0x7f2308101870 2024-03-04T12:16:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.024+0000 7f22f77fe700 1 -- 172.21.15.60:0/1520092768 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7f22f802a070 con 0x7f2308101870 2024-03-04T12:16:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.024+0000 7f230c889700 1 -- 172.21.15.60:0/1520092768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2308108080 con 0x7f2308101870 2024-03-04T12:16:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.024+0000 7f22f77fe700 1 --2- 172.21.15.60:0/1520092768 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f22f0038c40 0x7f22f003b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.024+0000 7f22f77fe700 1 -- 172.21.15.60:0/1520092768 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f22f801e030 con 0x7f2308101870 2024-03-04T12:16:16.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.024+0000 7f2305d9b700 1 --2- 172.21.15.60:0/1520092768 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f22f0038c40 0x7f22f003b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:16.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.024+0000 7f2305d9b700 1 --2- 172.21.15.60:0/1520092768 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f22f0038c40 0x7f22f003b100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f22fc006fd0 tx=0x7f22fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:16.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.029+0000 7f22f77fe700 1 -- 172.21.15.60:0/1520092768 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f22f8053080 con 0x7f2308101870 2024-03-04T12:16:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.232+0000 7f230c889700 1 -- 172.21.15.60:0/1520092768 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f230802d980 con 0x7f2308101870 2024-03-04T12:16:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.235+0000 7f22f77fe700 1 -- 172.21.15.60:0/1520092768 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f22f80470c0 con 0x7f2308101870 2024-03-04T12:16:16.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.237+0000 7f230c889700 1 -- 172.21.15.60:0/1520092768 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f22f0038c40 msgr2=0x7f22f003b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:16.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.237+0000 7f230c889700 1 --2- 172.21.15.60:0/1520092768 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f22f0038c40 0x7f22f003b100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f22fc006fd0 tx=0x7f22fc006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:16.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.237+0000 7f230c889700 1 -- 172.21.15.60:0/1520092768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2308101870 msgr2=0x7f2308196f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:16.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.237+0000 7f230c889700 1 --2- 172.21.15.60:0/1520092768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2308101870 0x7f2308196f80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f22f8006010 tx=0x7f22f8005050 comp rx=0 tx=0).stop 2024-03-04T12:16:16.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.237+0000 7f230c889700 1 -- 172.21.15.60:0/1520092768 shutdown_connections 2024-03-04T12:16:16.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.237+0000 7f230c889700 1 --2- 172.21.15.60:0/1520092768 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f22f0038c40 0x7f22f003b100 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:16.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.237+0000 7f230c889700 1 --2- 172.21.15.60:0/1520092768 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2308101870 0x7f2308196f80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:16.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.237+0000 7f230c889700 1 -- 172.21.15.60:0/1520092768 >> 172.21.15.60:0/1520092768 conn(0x7f23080fce10 msgr2=0x7f23080fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:16.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.237+0000 7f230c889700 1 -- 172.21.15.60:0/1520092768 shutdown_connections 2024-03-04T12:16:16.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:16.237+0000 7f230c889700 1 -- 172.21.15.60:0/1520092768 wait complete. 2024-03-04T12:16:16.826 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-04T12:16:16.826 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-04T12:16:17.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:17 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1520092768' entity='client.admin' 2024-03-04T12:16:17.498 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:16:18.708 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:18.709 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:18.709 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi060", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:16:18.709 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:18.709 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:16:18.709 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:18.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.944+0000 7f9d17595700 1 -- 172.21.15.60:0/3217002719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d10100770 msgr2=0x7f9d10100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:18.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.944+0000 7f9d17595700 1 --2- 172.21.15.60:0/3217002719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d10100770 0x7f9d10100b90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009910 tx=0x7f9d0c009c20 comp rx=0 tx=0).stop 2024-03-04T12:16:18.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.944+0000 7f9d17595700 1 -- 172.21.15.60:0/3217002719 shutdown_connections 2024-03-04T12:16:18.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.944+0000 7f9d17595700 1 --2- 172.21.15.60:0/3217002719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d10100770 0x7f9d10100b90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:18.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.944+0000 7f9d17595700 1 -- 172.21.15.60:0/3217002719 >> 172.21.15.60:0/3217002719 conn(0x7f9d100fbd30 msgr2=0x7f9d100fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:18.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.944+0000 7f9d17595700 1 -- 172.21.15.60:0/3217002719 shutdown_connections 2024-03-04T12:16:18.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.944+0000 7f9d17595700 1 -- 172.21.15.60:0/3217002719 wait complete. 2024-03-04T12:16:18.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.944+0000 7f9d17595700 1 Processor -- start 2024-03-04T12:16:18.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d17595700 1 -- start start 2024-03-04T12:16:18.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d17595700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d10100770 0x7f9d10199210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:18.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d17595700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d10106800 con 0x7f9d10100770 2024-03-04T12:16:18.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d15331700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d10100770 0x7f9d10199210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:18.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d15331700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d10100770 0x7f9d10199210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42794/0 (socket says 172.21.15.60:42794) 2024-03-04T12:16:18.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d15331700 1 -- 172.21.15.60:0/1557949947 learned_addr learned my addr 172.21.15.60:0/1557949947 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:18.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d15331700 1 -- 172.21.15.60:0/1557949947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d10199750 con 0x7f9d10100770 2024-03-04T12:16:18.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d15331700 1 --2- 172.21.15.60:0/1557949947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d10100770 0x7f9d10199210 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009fd0 tx=0x7f9d0c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:18.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d067fc700 1 -- 172.21.15.60:0/1557949947 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d0c027070 con 0x7f9d10100770 2024-03-04T12:16:18.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d067fc700 1 -- 172.21.15.60:0/1557949947 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c00be40 con 0x7f9d10100770 2024-03-04T12:16:18.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d17595700 1 -- 172.21.15.60:0/1557949947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d101999e0 con 0x7f9d10100770 2024-03-04T12:16:18.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d067fc700 1 -- 172.21.15.60:0/1557949947 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d0c00f460 con 0x7f9d10100770 2024-03-04T12:16:18.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.945+0000 7f9d17595700 1 -- 172.21.15.60:0/1557949947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1019c6d0 con 0x7f9d10100770 2024-03-04T12:16:18.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.946+0000 7f9d067fc700 1 -- 172.21.15.60:0/1557949947 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7f9d0c02a070 con 0x7f9d10100770 2024-03-04T12:16:18.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.946+0000 7f9d17595700 1 -- 172.21.15.60:0/1557949947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d10106f80 con 0x7f9d10100770 2024-03-04T12:16:18.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.946+0000 7f9d067fc700 1 --2- 172.21.15.60:0/1557949947 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9cfc038c40 0x7f9cfc03b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:18.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.946+0000 7f9d067fc700 1 -- 172.21.15.60:0/1557949947 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9d0c01e030 con 0x7f9d10100770 2024-03-04T12:16:18.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.947+0000 7f9d14b30700 1 --2- 172.21.15.60:0/1557949947 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9cfc038c40 0x7f9cfc03b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:18.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.947+0000 7f9d14b30700 1 --2- 172.21.15.60:0/1557949947 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9cfc038c40 0x7f9cfc03b100 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9d00006fd0 tx=0x7f9d00006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:18.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:18.951+0000 7f9d067fc700 1 -- 172.21.15.60:0/1557949947 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9d0c050030 con 0x7f9d10100770 2024-03-04T12:16:19.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:19.166+0000 7f9d17595700 1 -- 172.21.15.60:0/1557949947 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d1002d980 con 0x7f9cfc038c40 2024-03-04T12:16:19.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:19.170+0000 7f9d067fc700 1 -- 172.21.15.60:0/1557949947 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9d1002d980 con 0x7f9cfc038c40 2024-03-04T12:16:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:19.172+0000 7f9d17595700 1 -- 172.21.15.60:0/1557949947 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9cfc038c40 msgr2=0x7f9cfc03b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:19.172+0000 7f9d17595700 1 --2- 172.21.15.60:0/1557949947 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9cfc038c40 0x7f9cfc03b100 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9d00006fd0 tx=0x7f9d00006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:19.172+0000 7f9d17595700 1 -- 172.21.15.60:0/1557949947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d10100770 msgr2=0x7f9d10199210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:19.172+0000 7f9d17595700 1 --2- 172.21.15.60:0/1557949947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d10100770 0x7f9d10199210 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009fd0 tx=0x7f9d0c005050 comp rx=0 tx=0).stop 2024-03-04T12:16:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:19.172+0000 7f9d17595700 1 -- 172.21.15.60:0/1557949947 shutdown_connections 2024-03-04T12:16:19.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:19.172+0000 7f9d17595700 1 --2- 172.21.15.60:0/1557949947 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9cfc038c40 0x7f9cfc03b100 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:19.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:19.172+0000 7f9d17595700 1 --2- 172.21.15.60:0/1557949947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d10100770 0x7f9d10199210 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:19.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:19.172+0000 7f9d17595700 1 -- 172.21.15.60:0/1557949947 >> 172.21.15.60:0/1557949947 conn(0x7f9d100fbd30 msgr2=0x7f9d101902b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:19.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:19.172+0000 7f9d17595700 1 -- 172.21.15.60:0/1557949947 shutdown_connections 2024-03-04T12:16:19.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:19.172+0000 7f9d17595700 1 -- 172.21.15.60:0/1557949947 wait complete. 2024-03-04T12:16:20.107 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi063 2024-03-04T12:16:20.107 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-04T12:16:20.107 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/etc/ceph/ceph.conf 2024-03-04T12:16:20.127 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-04T12:16:20.127 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:16:20.186 INFO:tasks.cephadm:Adding host smithi063 to orchestrator... 2024-03-04T12:16:20.186 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch host add smithi063 2024-03-04T12:16:20.422 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:16:20.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:20 smithi060 ceph-mon[32745]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:16:20.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:20.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:16:20.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:16:20.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:16:20.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:20 smithi060 ceph-mon[32745]: Updating smithi060:/etc/ceph/ceph.conf 2024-03-04T12:16:20.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:20 smithi060 ceph-mon[32745]: Updating smithi060:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:16:20.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:20 smithi060 ceph-mon[32745]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:16:20.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:20 smithi060 ceph-mon[32745]: Updating smithi060:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T12:16:20.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:20.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:20.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:21.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.701+0000 7f27fb1d6700 1 -- 172.21.15.60:0/339788790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27f4106700 msgr2=0x7f27f4106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:21.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.701+0000 7f27fb1d6700 1 --2- 172.21.15.60:0/339788790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27f4106700 0x7f27f4106ae0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f27e8009910 tx=0x7f27e8009c20 comp rx=0 tx=0).stop 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.702+0000 7f27fb1d6700 1 -- 172.21.15.60:0/339788790 shutdown_connections 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.702+0000 7f27fb1d6700 1 --2- 172.21.15.60:0/339788790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27f4106700 0x7f27f4106ae0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.702+0000 7f27fb1d6700 1 -- 172.21.15.60:0/339788790 >> 172.21.15.60:0/339788790 conn(0x7f27f40fc1f0 msgr2=0x7f27f40fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.702+0000 7f27fb1d6700 1 -- 172.21.15.60:0/339788790 shutdown_connections 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.702+0000 7f27fb1d6700 1 -- 172.21.15.60:0/339788790 wait complete. 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.702+0000 7f27fb1d6700 1 Processor -- start 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.702+0000 7f27fb1d6700 1 -- start start 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.702+0000 7f27fb1d6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27f4106700 0x7f27f40fc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.702+0000 7f27fb1d6700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f410ad40 con 0x7f27f4106700 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.702+0000 7f27f8f72700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27f4106700 0x7f27f40fc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.702+0000 7f27f8f72700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27f4106700 0x7f27f40fc6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:42814/0 (socket says 172.21.15.60:42814) 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.702+0000 7f27f8f72700 1 -- 172.21.15.60:0/3366687381 learned_addr learned my addr 172.21.15.60:0/3366687381 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:21.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.703+0000 7f27f8f72700 1 -- 172.21.15.60:0/3366687381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f40fe2e0 con 0x7f27f4106700 2024-03-04T12:16:21.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.703+0000 7f27f8f72700 1 --2- 172.21.15.60:0/3366687381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27f4106700 0x7f27f40fc6a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f27e8005850 tx=0x7f27e8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:21.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.703+0000 7f27f1ffb700 1 -- 172.21.15.60:0/3366687381 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e8026070 con 0x7f27f4106700 2024-03-04T12:16:21.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.703+0000 7f27f1ffb700 1 -- 172.21.15.60:0/3366687381 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f27e800bbd0 con 0x7f27f4106700 2024-03-04T12:16:21.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.703+0000 7f27f1ffb700 1 -- 172.21.15.60:0/3366687381 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e8003970 con 0x7f27f4106700 2024-03-04T12:16:21.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.703+0000 7f27fb1d6700 1 -- 172.21.15.60:0/3366687381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f40fe5d0 con 0x7f27f4106700 2024-03-04T12:16:21.708 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.703+0000 7f27fb1d6700 1 -- 172.21.15.60:0/3366687381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f40fcdb0 con 0x7f27f4106700 2024-03-04T12:16:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.704+0000 7f27f1ffb700 1 -- 172.21.15.60:0/3366687381 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7f27e8029070 con 0x7f27f4106700 2024-03-04T12:16:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.704+0000 7f27f1ffb700 1 --2- 172.21.15.60:0/3366687381 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f27e4038c40 0x7f27e403b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.704+0000 7f27f1ffb700 1 -- 172.21.15.60:0/3366687381 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f27e801e070 con 0x7f27f4106700 2024-03-04T12:16:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.704+0000 7f27fb1d6700 1 -- 172.21.15.60:0/3366687381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f4101e40 con 0x7f27f4106700 2024-03-04T12:16:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.704+0000 7f27f3fff700 1 --2- 172.21.15.60:0/3366687381 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f27e4038c40 0x7f27e403b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.705+0000 7f27f3fff700 1 --2- 172.21.15.60:0/3366687381 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f27e4038c40 0x7f27e403b100 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f27e0006fd0 tx=0x7f27e0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:21.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.709+0000 7f27f1ffb700 1 -- 172.21.15.60:0/3366687381 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f27e8030020 con 0x7f27f4106700 2024-03-04T12:16:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:21.918+0000 7f27fb1d6700 1 -- 172.21.15.60:0/3366687381 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi063", "target": ["mon-mgr", ""]}) v1 -- 0x7f27f40fdbd0 con 0x7f27e4038c40 2024-03-04T12:16:22.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:22 smithi060 ceph-mon[32745]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi063", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:16:23.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:23 smithi060 ceph-mon[32745]: Deploying cephadm binary to smithi063 2024-03-04T12:16:25.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:25 smithi060 ceph-mon[32745]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:26.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:26.266+0000 7f27f1ffb700 1 -- 172.21.15.60:0/3366687381 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f27f40fdbd0 con 0x7f27e4038c40 2024-03-04T12:16:26.273 INFO:teuthology.orchestra.run.smithi060.stdout:Added host 'smithi063' with addr '172.21.15.63' 2024-03-04T12:16:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:26.269+0000 7f27fb1d6700 1 -- 172.21.15.60:0/3366687381 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f27e4038c40 msgr2=0x7f27e403b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:26.269+0000 7f27fb1d6700 1 --2- 172.21.15.60:0/3366687381 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f27e4038c40 0x7f27e403b100 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f27e0006fd0 tx=0x7f27e0006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:26.269+0000 7f27fb1d6700 1 -- 172.21.15.60:0/3366687381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27f4106700 msgr2=0x7f27f40fc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:26.269+0000 7f27fb1d6700 1 --2- 172.21.15.60:0/3366687381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27f4106700 0x7f27f40fc6a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f27e8005850 tx=0x7f27e8005030 comp rx=0 tx=0).stop 2024-03-04T12:16:26.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:26.269+0000 7f27fb1d6700 1 -- 172.21.15.60:0/3366687381 shutdown_connections 2024-03-04T12:16:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:26.269+0000 7f27fb1d6700 1 --2- 172.21.15.60:0/3366687381 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f27e4038c40 0x7f27e403b100 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:26.269+0000 7f27fb1d6700 1 --2- 172.21.15.60:0/3366687381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27f4106700 0x7f27f40fc6a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:26.269+0000 7f27fb1d6700 1 -- 172.21.15.60:0/3366687381 >> 172.21.15.60:0/3366687381 conn(0x7f27f40fc1f0 msgr2=0x7f27f4104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:26.269+0000 7f27fb1d6700 1 -- 172.21.15.60:0/3366687381 shutdown_connections 2024-03-04T12:16:26.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:26.269+0000 7f27fb1d6700 1 -- 172.21.15.60:0/3366687381 wait complete. 2024-03-04T12:16:27.091 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-04T12:16:27.336 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:16:27.425 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:27 smithi060 ceph-mon[32745]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:27.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:27.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:16:27.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:28.338 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:28 smithi060 ceph-mon[32745]: Added host smithi063 2024-03-04T12:16:28.338 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:28 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:28.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.826+0000 7f6571786700 1 -- 172.21.15.60:0/905032807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c107800 msgr2=0x7f656c107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:28.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.826+0000 7f6571786700 1 --2- 172.21.15.60:0/905032807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c107800 0x7f656c107be0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f655c009910 tx=0x7f655c009c20 comp rx=0 tx=0).stop 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.826+0000 7f6571786700 1 -- 172.21.15.60:0/905032807 shutdown_connections 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.826+0000 7f6571786700 1 --2- 172.21.15.60:0/905032807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c107800 0x7f656c107be0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.826+0000 7f6571786700 1 -- 172.21.15.60:0/905032807 >> 172.21.15.60:0/905032807 conn(0x7f656c0fd330 msgr2=0x7f656c0ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.826+0000 7f6571786700 1 -- 172.21.15.60:0/905032807 shutdown_connections 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.826+0000 7f6571786700 1 -- 172.21.15.60:0/905032807 wait complete. 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.826+0000 7f6571786700 1 Processor -- start 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.827+0000 7f6571786700 1 -- start start 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.827+0000 7f6571786700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c107800 0x7f656c1973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.827+0000 7f6571786700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c10be40 con 0x7f656c107800 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.827+0000 7f656affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c107800 0x7f656c1973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.827+0000 7f656affd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c107800 0x7f656c1973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51284/0 (socket says 172.21.15.60:51284) 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.827+0000 7f656affd700 1 -- 172.21.15.60:0/4096198659 learned_addr learned my addr 172.21.15.60:0/4096198659 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:28.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.827+0000 7f656affd700 1 -- 172.21.15.60:0/4096198659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c1978f0 con 0x7f656c107800 2024-03-04T12:16:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.827+0000 7f656affd700 1 --2- 172.21.15.60:0/4096198659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c107800 0x7f656c1973b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f655c006010 tx=0x7f655c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.827+0000 7f6563fff700 1 -- 172.21.15.60:0/4096198659 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f655c026070 con 0x7f656c107800 2024-03-04T12:16:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.828+0000 7f6563fff700 1 -- 172.21.15.60:0/4096198659 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f655c00bbd0 con 0x7f656c107800 2024-03-04T12:16:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.828+0000 7f6571786700 1 -- 172.21.15.60:0/4096198659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c197be0 con 0x7f656c107800 2024-03-04T12:16:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.828+0000 7f6563fff700 1 -- 172.21.15.60:0/4096198659 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f655c003910 con 0x7f656c107800 2024-03-04T12:16:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.828+0000 7f6571786700 1 -- 172.21.15.60:0/4096198659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c19bc80 con 0x7f656c107800 2024-03-04T12:16:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.828+0000 7f6563fff700 1 -- 172.21.15.60:0/4096198659 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7f655c029070 con 0x7f656c107800 2024-03-04T12:16:28.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.829+0000 7f6563fff700 1 --2- 172.21.15.60:0/4096198659 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6558038c40 0x7f655803b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.829+0000 7f6563fff700 1 -- 172.21.15.60:0/4096198659 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f655c01e070 con 0x7f656c107800 2024-03-04T12:16:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.829+0000 7f6571786700 1 -- 172.21.15.60:0/4096198659 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c102f40 con 0x7f656c107800 2024-03-04T12:16:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.829+0000 7f656a7fc700 1 --2- 172.21.15.60:0/4096198659 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6558038c40 0x7f655803b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:28.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.829+0000 7f656a7fc700 1 --2- 172.21.15.60:0/4096198659 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6558038c40 0x7f655803b100 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6554006fd0 tx=0x7f6554006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:28.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:28.834+0000 7f6563fff700 1 -- 172.21.15.60:0/4096198659 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f655c030020 con 0x7f656c107800 2024-03-04T12:16:29.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:29.043+0000 7f6571786700 1 -- 172.21.15.60:0/4096198659 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f656c063450 con 0x7f6558038c40 2024-03-04T12:16:29.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:29.044+0000 7f6563fff700 1 -- 172.21.15.60:0/4096198659 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f656c063450 con 0x7f6558038c40 2024-03-04T12:16:29.047 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:16:29.048 INFO:teuthology.orchestra.run.smithi060.stdout:[{"addr": "172.21.15.60", "hostname": "smithi060", "labels": [], "status": ""}, {"addr": "172.21.15.63", "hostname": "smithi063", "labels": [], "status": ""}] 2024-03-04T12:16:29.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:29.046+0000 7f6571786700 1 -- 172.21.15.60:0/4096198659 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6558038c40 msgr2=0x7f655803b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:29.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:29.046+0000 7f6571786700 1 --2- 172.21.15.60:0/4096198659 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6558038c40 0x7f655803b100 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6554006fd0 tx=0x7f6554006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:29.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:29.046+0000 7f6571786700 1 -- 172.21.15.60:0/4096198659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c107800 msgr2=0x7f656c1973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:29.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:29.046+0000 7f6571786700 1 --2- 172.21.15.60:0/4096198659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c107800 0x7f656c1973b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f655c006010 tx=0x7f655c005030 comp rx=0 tx=0).stop 2024-03-04T12:16:29.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:29.046+0000 7f6571786700 1 -- 172.21.15.60:0/4096198659 shutdown_connections 2024-03-04T12:16:29.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:29.046+0000 7f6571786700 1 --2- 172.21.15.60:0/4096198659 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6558038c40 0x7f655803b100 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:29.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:29.046+0000 7f6571786700 1 --2- 172.21.15.60:0/4096198659 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f656c107800 0x7f656c1973b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:29.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:29.046+0000 7f6571786700 1 -- 172.21.15.60:0/4096198659 >> 172.21.15.60:0/4096198659 conn(0x7f656c0fd330 msgr2=0x7f656c1093b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:29.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:29.046+0000 7f6571786700 1 -- 172.21.15.60:0/4096198659 shutdown_connections 2024-03-04T12:16:29.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:29.046+0000 7f6571786700 1 -- 172.21.15.60:0/4096198659 wait complete. 2024-03-04T12:16:29.352 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:29 smithi060 ceph-mon[32745]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:29.352 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:29 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:29.647 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi086 2024-03-04T12:16:29.648 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T12:16:29.648 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.conf 2024-03-04T12:16:29.666 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T12:16:29.666 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:16:29.723 INFO:tasks.cephadm:Adding host smithi086 to orchestrator... 2024-03-04T12:16:29.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch host add smithi086 2024-03-04T12:16:29.962 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:16:30.381 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:30 smithi060 ceph-mon[32745]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:16:31.279 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:31 smithi060 ceph-mon[32745]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:31.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.494+0000 7f4070eb3700 1 -- 172.21.15.60:0/465645265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f406c1029f0 msgr2=0x7f406c102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:31.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.494+0000 7f4070eb3700 1 --2- 172.21.15.60:0/465645265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f406c1029f0 0x7f406c102dd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f405c009910 tx=0x7f405c009c20 comp rx=0 tx=0).stop 2024-03-04T12:16:31.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.494+0000 7f4070eb3700 1 -- 172.21.15.60:0/465645265 shutdown_connections 2024-03-04T12:16:31.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.494+0000 7f4070eb3700 1 --2- 172.21.15.60:0/465645265 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f406c1029f0 0x7f406c102dd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:31.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.494+0000 7f4070eb3700 1 -- 172.21.15.60:0/465645265 >> 172.21.15.60:0/465645265 conn(0x7f406c0fc260 msgr2=0x7f406c0fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:31.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.494+0000 7f4070eb3700 1 -- 172.21.15.60:0/465645265 shutdown_connections 2024-03-04T12:16:31.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.494+0000 7f4070eb3700 1 -- 172.21.15.60:0/465645265 wait complete. 2024-03-04T12:16:31.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.495+0000 7f4070eb3700 1 Processor -- start 2024-03-04T12:16:31.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.495+0000 7f4070eb3700 1 -- start start 2024-03-04T12:16:31.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.495+0000 7f4070eb3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f406c1029f0 0x7f406c197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:31.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.495+0000 7f4070eb3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406c10bec0 con 0x7f406c1029f0 2024-03-04T12:16:31.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.495+0000 7f406a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f406c1029f0 0x7f406c197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:31.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.495+0000 7f406a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f406c1029f0 0x7f406c197380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51312/0 (socket says 172.21.15.60:51312) 2024-03-04T12:16:31.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.495+0000 7f406a59c700 1 -- 172.21.15.60:0/1574774041 learned_addr learned my addr 172.21.15.60:0/1574774041 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:31.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.495+0000 7f406a59c700 1 -- 172.21.15.60:0/1574774041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c1978c0 con 0x7f406c1029f0 2024-03-04T12:16:31.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.495+0000 7f406a59c700 1 --2- 172.21.15.60:0/1574774041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f406c1029f0 0x7f406c197380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f405c006010 tx=0x7f405c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:31.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.495+0000 7f405b7fe700 1 -- 172.21.15.60:0/1574774041 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f405c026070 con 0x7f406c1029f0 2024-03-04T12:16:31.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.495+0000 7f405b7fe700 1 -- 172.21.15.60:0/1574774041 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f405c00bc30 con 0x7f406c1029f0 2024-03-04T12:16:31.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.496+0000 7f405b7fe700 1 -- 172.21.15.60:0/1574774041 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f405c0039a0 con 0x7f406c1029f0 2024-03-04T12:16:31.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.496+0000 7f4070eb3700 1 -- 172.21.15.60:0/1574774041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406c197b50 con 0x7f406c1029f0 2024-03-04T12:16:31.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.496+0000 7f4070eb3700 1 -- 172.21.15.60:0/1574774041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406c197fb0 con 0x7f406c1029f0 2024-03-04T12:16:31.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.496+0000 7f405b7fe700 1 -- 172.21.15.60:0/1574774041 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7f405c029070 con 0x7f406c1029f0 2024-03-04T12:16:31.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.497+0000 7f405b7fe700 1 --2- 172.21.15.60:0/1574774041 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4054038ca0 0x7f405403b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:31.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.497+0000 7f405b7fe700 1 -- 172.21.15.60:0/1574774041 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f405c01e070 con 0x7f406c1029f0 2024-03-04T12:16:31.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.497+0000 7f4070eb3700 1 -- 172.21.15.60:0/1574774041 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406c06a540 con 0x7f406c1029f0 2024-03-04T12:16:31.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.497+0000 7f4069d9b700 1 --2- 172.21.15.60:0/1574774041 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4054038ca0 0x7f405403b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:31.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.497+0000 7f4069d9b700 1 --2- 172.21.15.60:0/1574774041 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4054038ca0 0x7f405403b160 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4060006fd0 tx=0x7f4060006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:31.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.502+0000 7f405b7fe700 1 -- 172.21.15.60:0/1574774041 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f405c030020 con 0x7f406c1029f0 2024-03-04T12:16:31.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:31.710+0000 7f4070eb3700 1 -- 172.21.15.60:0/1574774041 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}) v1 -- 0x7f406c1986e0 con 0x7f4054038ca0 2024-03-04T12:16:33.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:33 smithi060 ceph-mon[32745]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:16:33.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:33 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:33 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:33 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:33 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:33 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:16:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:33 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:16:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:33 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:16:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:33 smithi060 ceph-mon[32745]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-04T12:16:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:33 smithi060 ceph-mon[32745]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:34.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:34 smithi060 ceph-mon[32745]: Updating smithi063:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:16:34.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:34 smithi060 ceph-mon[32745]: Deploying cephadm binary to smithi086 2024-03-04T12:16:34.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:34 smithi060 ceph-mon[32745]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:16:34.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:34 smithi060 ceph-mon[32745]: Updating smithi063:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T12:16:35.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:35 smithi060 ceph-mon[32745]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:35.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:35.985+0000 7f405b7fe700 1 -- 172.21.15.60:0/1574774041 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f406c1986e0 con 0x7f4054038ca0 2024-03-04T12:16:35.989 INFO:teuthology.orchestra.run.smithi060.stdout:Added host 'smithi086' with addr '172.21.15.86' 2024-03-04T12:16:35.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:35.987+0000 7f4070eb3700 1 -- 172.21.15.60:0/1574774041 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4054038ca0 msgr2=0x7f405403b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:35.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:35.987+0000 7f4070eb3700 1 --2- 172.21.15.60:0/1574774041 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4054038ca0 0x7f405403b160 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4060006fd0 tx=0x7f4060006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:35.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:35.987+0000 7f4070eb3700 1 -- 172.21.15.60:0/1574774041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f406c1029f0 msgr2=0x7f406c197380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:35.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:35.987+0000 7f4070eb3700 1 --2- 172.21.15.60:0/1574774041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f406c1029f0 0x7f406c197380 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f405c006010 tx=0x7f405c005030 comp rx=0 tx=0).stop 2024-03-04T12:16:35.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:35.988+0000 7f4070eb3700 1 -- 172.21.15.60:0/1574774041 shutdown_connections 2024-03-04T12:16:35.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:35.988+0000 7f4070eb3700 1 --2- 172.21.15.60:0/1574774041 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4054038ca0 0x7f405403b160 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:35.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:35.988+0000 7f4070eb3700 1 --2- 172.21.15.60:0/1574774041 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f406c1029f0 0x7f406c197380 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:35.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:35.988+0000 7f4070eb3700 1 -- 172.21.15.60:0/1574774041 >> 172.21.15.60:0/1574774041 conn(0x7f406c0fc260 msgr2=0x7f406c103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:35.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:35.988+0000 7f4070eb3700 1 -- 172.21.15.60:0/1574774041 shutdown_connections 2024-03-04T12:16:35.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:35.988+0000 7f4070eb3700 1 -- 172.21.15.60:0/1574774041 wait complete. 2024-03-04T12:16:36.625 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:36 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:36.625 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:36 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:36.625 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:36 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:36.625 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:36 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:36.625 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:36 smithi060 ceph-mon[32745]: Added host smithi086 2024-03-04T12:16:36.625 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:36 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:16:36.625 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:36 smithi060 ceph-mon[32745]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:36.652 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-04T12:16:36.890 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:16:37.597 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:37 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:37.598 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:37 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:38.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.318+0000 7f1688b02700 1 -- 172.21.15.60:0/1280657781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16841029f0 msgr2=0x7f1684102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:38.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.318+0000 7f1688b02700 1 --2- 172.21.15.60:0/1280657781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16841029f0 0x7f1684102dd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1674009910 tx=0x7f1674009c20 comp rx=0 tx=0).stop 2024-03-04T12:16:38.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.318+0000 7f1688b02700 1 -- 172.21.15.60:0/1280657781 shutdown_connections 2024-03-04T12:16:38.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.318+0000 7f1688b02700 1 --2- 172.21.15.60:0/1280657781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16841029f0 0x7f1684102dd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:38.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.318+0000 7f1688b02700 1 -- 172.21.15.60:0/1280657781 >> 172.21.15.60:0/1280657781 conn(0x7f16840fc260 msgr2=0x7f16840fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:38.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.318+0000 7f1688b02700 1 -- 172.21.15.60:0/1280657781 shutdown_connections 2024-03-04T12:16:38.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.319+0000 7f1688b02700 1 -- 172.21.15.60:0/1280657781 wait complete. 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.319+0000 7f1688b02700 1 Processor -- start 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.319+0000 7f1688b02700 1 -- start start 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.319+0000 7f1688b02700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16841029f0 0x7f16841995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.319+0000 7f1688b02700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168410bec0 con 0x7f16841029f0 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.319+0000 7f168259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16841029f0 0x7f16841995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.320+0000 7f168259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16841029f0 0x7f16841995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60102/0 (socket says 172.21.15.60:60102) 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.320+0000 7f168259c700 1 -- 172.21.15.60:0/979370005 learned_addr learned my addr 172.21.15.60:0/979370005 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.320+0000 7f168259c700 1 -- 172.21.15.60:0/979370005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1684199b00 con 0x7f16841029f0 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.320+0000 7f168259c700 1 --2- 172.21.15.60:0/979370005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16841029f0 0x7f16841995c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1674006010 tx=0x7f1674005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.320+0000 7f16737fe700 1 -- 172.21.15.60:0/979370005 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1674026070 con 0x7f16841029f0 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.320+0000 7f16737fe700 1 -- 172.21.15.60:0/979370005 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f167400bc30 con 0x7f16841029f0 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.320+0000 7f16737fe700 1 -- 172.21.15.60:0/979370005 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1674003ab0 con 0x7f16841029f0 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.320+0000 7f1688b02700 1 -- 172.21.15.60:0/979370005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1684199d90 con 0x7f16841029f0 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.320+0000 7f1688b02700 1 -- 172.21.15.60:0/979370005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168419a1f0 con 0x7f16841029f0 2024-03-04T12:16:38.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.321+0000 7f16737fe700 1 -- 172.21.15.60:0/979370005 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7f1674029070 con 0x7f16841029f0 2024-03-04T12:16:38.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.321+0000 7f1688b02700 1 -- 172.21.15.60:0/979370005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168406a540 con 0x7f16841029f0 2024-03-04T12:16:38.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.321+0000 7f16737fe700 1 --2- 172.21.15.60:0/979370005 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f166c038c40 0x7f166c03b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:38.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.321+0000 7f16737fe700 1 -- 172.21.15.60:0/979370005 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f167401e070 con 0x7f16841029f0 2024-03-04T12:16:38.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.321+0000 7f1681d9b700 1 --2- 172.21.15.60:0/979370005 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f166c038c40 0x7f166c03b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:38.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.322+0000 7f1681d9b700 1 --2- 172.21.15.60:0/979370005 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f166c038c40 0x7f166c03b100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1678006fd0 tx=0x7f1678006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:38.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.326+0000 7f16737fe700 1 -- 172.21.15.60:0/979370005 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1674013070 con 0x7f16841029f0 2024-03-04T12:16:38.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.534+0000 7f1688b02700 1 -- 172.21.15.60:0/979370005 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f168419a940 con 0x7f166c038c40 2024-03-04T12:16:38.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.535+0000 7f16737fe700 1 -- 172.21.15.60:0/979370005 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+238 (secure 0 0 0) 0x7f168419a940 con 0x7f166c038c40 2024-03-04T12:16:38.538 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:16:38.538 INFO:teuthology.orchestra.run.smithi060.stdout:[{"addr": "172.21.15.60", "hostname": "smithi060", "labels": [], "status": ""}, {"addr": "172.21.15.63", "hostname": "smithi063", "labels": [], "status": ""}, {"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}] 2024-03-04T12:16:38.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.537+0000 7f1688b02700 1 -- 172.21.15.60:0/979370005 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f166c038c40 msgr2=0x7f166c03b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:38.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.537+0000 7f1688b02700 1 --2- 172.21.15.60:0/979370005 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f166c038c40 0x7f166c03b100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1678006fd0 tx=0x7f1678006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:38.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.537+0000 7f1688b02700 1 -- 172.21.15.60:0/979370005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16841029f0 msgr2=0x7f16841995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:38.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.537+0000 7f1688b02700 1 --2- 172.21.15.60:0/979370005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16841029f0 0x7f16841995c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1674006010 tx=0x7f1674005050 comp rx=0 tx=0).stop 2024-03-04T12:16:38.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.537+0000 7f1688b02700 1 -- 172.21.15.60:0/979370005 shutdown_connections 2024-03-04T12:16:38.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.537+0000 7f1688b02700 1 --2- 172.21.15.60:0/979370005 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f166c038c40 0x7f166c03b100 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:38.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.537+0000 7f1688b02700 1 --2- 172.21.15.60:0/979370005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16841029f0 0x7f16841995c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:38.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.537+0000 7f1688b02700 1 -- 172.21.15.60:0/979370005 >> 172.21.15.60:0/979370005 conn(0x7f16840fc260 msgr2=0x7f1684107b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:38.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.537+0000 7f1688b02700 1 -- 172.21.15.60:0/979370005 shutdown_connections 2024-03-04T12:16:38.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:38.537+0000 7f1688b02700 1 -- 172.21.15.60:0/979370005 wait complete. 2024-03-04T12:16:38.551 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:38 smithi060 ceph-mon[32745]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:39.023 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-04T12:16:39.023 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd crush tunables default 2024-03-04T12:16:39.259 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:16:39.657 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:39 smithi060 ceph-mon[32745]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:16:39.657 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.650+0000 7eff9761a700 1 -- 172.21.15.60:0/1616962275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff901077f0 msgr2=0x7eff90107bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.650+0000 7eff9761a700 1 --2- 172.21.15.60:0/1616962275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff901077f0 0x7eff90107bd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7eff8000b0a0 tx=0x7eff8000b3b0 comp rx=0 tx=0).stop 2024-03-04T12:16:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.650+0000 7eff9761a700 1 -- 172.21.15.60:0/1616962275 shutdown_connections 2024-03-04T12:16:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.650+0000 7eff9761a700 1 --2- 172.21.15.60:0/1616962275 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff901077f0 0x7eff90107bd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.650+0000 7eff9761a700 1 -- 172.21.15.60:0/1616962275 >> 172.21.15.60:0/1616962275 conn(0x7eff900fd2e0 msgr2=0x7eff900ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:40.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.650+0000 7eff9761a700 1 -- 172.21.15.60:0/1616962275 shutdown_connections 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.650+0000 7eff9761a700 1 -- 172.21.15.60:0/1616962275 wait complete. 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.651+0000 7eff9761a700 1 Processor -- start 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.651+0000 7eff9761a700 1 -- start start 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.651+0000 7eff9761a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff901077f0 0x7eff90074120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.651+0000 7eff9761a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9010be30 con 0x7eff901077f0 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.651+0000 7eff953b6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff901077f0 0x7eff90074120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.651+0000 7eff953b6700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff901077f0 0x7eff90074120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60118/0 (socket says 172.21.15.60:60118) 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.651+0000 7eff953b6700 1 -- 172.21.15.60:0/117774089 learned_addr learned my addr 172.21.15.60:0/117774089 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.652+0000 7eff953b6700 1 -- 172.21.15.60:0/117774089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff90076010 con 0x7eff901077f0 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.652+0000 7eff953b6700 1 --2- 172.21.15.60:0/117774089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff901077f0 0x7eff90074120 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7eff80015040 tx=0x7eff80003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.652+0000 7eff8e7fc700 1 -- 172.21.15.60:0/117774089 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff80017040 con 0x7eff901077f0 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.652+0000 7eff8e7fc700 1 -- 172.21.15.60:0/117774089 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff80022070 con 0x7eff901077f0 2024-03-04T12:16:40.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.652+0000 7eff9761a700 1 -- 172.21.15.60:0/117774089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff900746c0 con 0x7eff901077f0 2024-03-04T12:16:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.652+0000 7eff8e7fc700 1 -- 172.21.15.60:0/117774089 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff8001d5d0 con 0x7eff901077f0 2024-03-04T12:16:40.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.652+0000 7eff9761a700 1 -- 172.21.15.60:0/117774089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff90074b80 con 0x7eff901077f0 2024-03-04T12:16:40.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.653+0000 7eff8e7fc700 1 -- 172.21.15.60:0/117774089 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7eff80004000 con 0x7eff901077f0 2024-03-04T12:16:40.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.653+0000 7eff9761a700 1 -- 172.21.15.60:0/117774089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff90102f30 con 0x7eff901077f0 2024-03-04T12:16:40.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.653+0000 7eff8e7fc700 1 --2- 172.21.15.60:0/117774089 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7eff78038ca0 0x7eff7803b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:40.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.653+0000 7eff8e7fc700 1 -- 172.21.15.60:0/117774089 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7eff8000f050 con 0x7eff901077f0 2024-03-04T12:16:40.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.653+0000 7eff94bb5700 1 --2- 172.21.15.60:0/117774089 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7eff78038ca0 0x7eff7803b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:40.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.654+0000 7eff94bb5700 1 --2- 172.21.15.60:0/117774089 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7eff78038ca0 0x7eff7803b160 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7eff84006fd0 tx=0x7eff84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:40.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.658+0000 7eff8e7fc700 1 -- 172.21.15.60:0/117774089 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7eff80007e20 con 0x7eff901077f0 2024-03-04T12:16:40.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:40.866+0000 7eff9761a700 1 -- 172.21.15.60:0/117774089 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7eff90075650 con 0x7eff901077f0 2024-03-04T12:16:40.870 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:40 smithi060 ceph-mon[32745]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:41.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:41.566+0000 7eff8e7fc700 1 -- 172.21.15.60:0/117774089 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7eff800279e0 con 0x7eff901077f0 2024-03-04T12:16:41.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:41.568+0000 7eff9761a700 1 -- 172.21.15.60:0/117774089 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7eff78038ca0 msgr2=0x7eff7803b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:41.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:41.568+0000 7eff9761a700 1 --2- 172.21.15.60:0/117774089 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7eff78038ca0 0x7eff7803b160 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7eff84006fd0 tx=0x7eff84006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:41.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:41.568+0000 7eff9761a700 1 -- 172.21.15.60:0/117774089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff901077f0 msgr2=0x7eff90074120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:41.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:41.568+0000 7eff9761a700 1 --2- 172.21.15.60:0/117774089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff901077f0 0x7eff90074120 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7eff80015040 tx=0x7eff80003d50 comp rx=0 tx=0).stop 2024-03-04T12:16:41.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:41.568+0000 7eff9761a700 1 -- 172.21.15.60:0/117774089 shutdown_connections 2024-03-04T12:16:41.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:41.568+0000 7eff9761a700 1 --2- 172.21.15.60:0/117774089 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7eff78038ca0 0x7eff7803b160 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:41.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:41.568+0000 7eff9761a700 1 --2- 172.21.15.60:0/117774089 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff901077f0 0x7eff90074120 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:41.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:41.568+0000 7eff9761a700 1 -- 172.21.15.60:0/117774089 >> 172.21.15.60:0/117774089 conn(0x7eff900fd2e0 msgr2=0x7eff901092f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:41.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:41.568+0000 7eff9761a700 1 -- 172.21.15.60:0/117774089 shutdown_connections 2024-03-04T12:16:41.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:16:41.569+0000 7eff9761a700 1 -- 172.21.15.60:0/117774089 wait complete. 2024-03-04T12:16:41.573 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2024-03-04T12:16:41.773 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:41 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/117774089' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T12:16:42.205 INFO:tasks.cephadm:Adding mon.a on smithi060 2024-03-04T12:16:42.205 INFO:tasks.cephadm:Adding mon.b on smithi063 2024-03-04T12:16:42.205 INFO:tasks.cephadm:Adding mon.c on smithi086 2024-03-04T12:16:42.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch apply mon '3;smithi060:172.21.15.60=a;smithi063:172.21.15.63=b;smithi086:172.21.15.86=c' 2024-03-04T12:16:42.439 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:16:42.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:42 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/117774089' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T12:16:42.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:42 smithi060 ceph-mon[32745]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T12:16:42.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:42 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:42.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:42 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:42.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:42 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:42.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:42 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:42.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:42 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:16:42.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:42 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:16:42.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:42 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:16:42.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:42 smithi060 ceph-mon[32745]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T12:16:42.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:42 smithi060 ceph-mon[32745]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:43.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.910+0000 7fb3df451700 1 -- 172.21.15.86:0/968014431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d8106700 msgr2=0x7fb3d8106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:43.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.910+0000 7fb3df451700 1 --2- 172.21.15.86:0/968014431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d8106700 0x7fb3d8106ae0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009910 tx=0x7fb3d4009c20 comp rx=0 tx=0).stop 2024-03-04T12:16:43.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.910+0000 7fb3df451700 1 -- 172.21.15.86:0/968014431 shutdown_connections 2024-03-04T12:16:43.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.910+0000 7fb3df451700 1 --2- 172.21.15.86:0/968014431 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d8106700 0x7fb3d8106ae0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:43.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.910+0000 7fb3df451700 1 -- 172.21.15.86:0/968014431 >> 172.21.15.86:0/968014431 conn(0x7fb3d80fc210 msgr2=0x7fb3d80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:43.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.910+0000 7fb3df451700 1 -- 172.21.15.86:0/968014431 shutdown_connections 2024-03-04T12:16:43.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.910+0000 7fb3df451700 1 -- 172.21.15.86:0/968014431 wait complete. 2024-03-04T12:16:43.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.911+0000 7fb3df451700 1 Processor -- start 2024-03-04T12:16:43.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.911+0000 7fb3df451700 1 -- start start 2024-03-04T12:16:43.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.911+0000 7fb3df451700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d8106700 0x7fb3d8198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:43.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.911+0000 7fb3df451700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d810ad40 con 0x7fb3d8106700 2024-03-04T12:16:43.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.911+0000 7fb3dd1ed700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d8106700 0x7fb3d8198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:43.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.911+0000 7fb3dd1ed700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d8106700 0x7fb3d8198220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.86:35520/0 (socket says 172.21.15.86:35520) 2024-03-04T12:16:43.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.911+0000 7fb3dd1ed700 1 -- 172.21.15.86:0/482644509 learned_addr learned my addr 172.21.15.86:0/482644509 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-03-04T12:16:43.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.912+0000 7fb3dd1ed700 1 -- 172.21.15.86:0/482644509 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d8198760 con 0x7fb3d8106700 2024-03-04T12:16:43.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.912+0000 7fb3dd1ed700 1 --2- 172.21.15.86:0/482644509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d8106700 0x7fb3d8198220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4005850 tx=0x7fb3d4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:43.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.912+0000 7fb3ce7fc700 1 -- 172.21.15.86:0/482644509 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3d4026070 con 0x7fb3d8106700 2024-03-04T12:16:43.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.912+0000 7fb3df451700 1 -- 172.21.15.86:0/482644509 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d8198a50 con 0x7fb3d8106700 2024-03-04T12:16:43.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.912+0000 7fb3ce7fc700 1 -- 172.21.15.86:0/482644509 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3d400bbd0 con 0x7fb3d8106700 2024-03-04T12:16:43.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.912+0000 7fb3ce7fc700 1 -- 172.21.15.86:0/482644509 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3d4003970 con 0x7fb3d8106700 2024-03-04T12:16:43.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.912+0000 7fb3df451700 1 -- 172.21.15.86:0/482644509 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d8198f10 con 0x7fb3d8106700 2024-03-04T12:16:43.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.913+0000 7fb3df451700 1 -- 172.21.15.86:0/482644509 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d8101e40 con 0x7fb3d8106700 2024-03-04T12:16:43.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.913+0000 7fb3ce7fc700 1 -- 172.21.15.86:0/482644509 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7fb3d4029070 con 0x7fb3d8106700 2024-03-04T12:16:43.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.913+0000 7fb3ce7fc700 1 --2- 172.21.15.86:0/482644509 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb3c4038c40 0x7fb3c403b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:43.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.913+0000 7fb3ce7fc700 1 -- 172.21.15.86:0/482644509 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb3d401e070 con 0x7fb3d8106700 2024-03-04T12:16:43.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.914+0000 7fb3dc9ec700 1 --2- 172.21.15.86:0/482644509 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb3c4038c40 0x7fb3c403b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:43.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.914+0000 7fb3dc9ec700 1 --2- 172.21.15.86:0/482644509 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb3c4038c40 0x7fb3c403b100 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8006fd0 tx=0x7fb3c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:43.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:43.918+0000 7fb3ce7fc700 1 -- 172.21.15.86:0/482644509 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb3d4013070 con 0x7fb3d8106700 2024-03-04T12:16:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:44.118+0000 7fb3df451700 1 -- 172.21.15.86:0/482644509 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi060:172.21.15.60=a;smithi063:172.21.15.63=b;smithi086:172.21.15.86=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3d8063450 con 0x7fb3c4038c40 2024-03-04T12:16:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:44.125+0000 7fb3ce7fc700 1 -- 172.21.15.86:0/482644509 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb3d8063450 con 0x7fb3c4038c40 2024-03-04T12:16:44.129 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mon update... 2024-03-04T12:16:44.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:44.126+0000 7fb3df451700 1 -- 172.21.15.86:0/482644509 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb3c4038c40 msgr2=0x7fb3c403b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:44.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:44.126+0000 7fb3df451700 1 --2- 172.21.15.86:0/482644509 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb3c4038c40 0x7fb3c403b100 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8006fd0 tx=0x7fb3c8006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:44.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:44.126+0000 7fb3df451700 1 -- 172.21.15.86:0/482644509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d8106700 msgr2=0x7fb3d8198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:44.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:44.126+0000 7fb3df451700 1 --2- 172.21.15.86:0/482644509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d8106700 0x7fb3d8198220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4005850 tx=0x7fb3d4005030 comp rx=0 tx=0).stop 2024-03-04T12:16:44.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:44.127+0000 7fb3df451700 1 -- 172.21.15.86:0/482644509 shutdown_connections 2024-03-04T12:16:44.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:44.127+0000 7fb3df451700 1 --2- 172.21.15.86:0/482644509 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb3c4038c40 0x7fb3c403b100 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:44.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:44.127+0000 7fb3df451700 1 --2- 172.21.15.86:0/482644509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3d8106700 0x7fb3d8198220 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:44.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:44.127+0000 7fb3df451700 1 -- 172.21.15.86:0/482644509 >> 172.21.15.86:0/482644509 conn(0x7fb3d80fc210 msgr2=0x7fb3d81082b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:44.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:44.127+0000 7fb3df451700 1 -- 172.21.15.86:0/482644509 shutdown_connections 2024-03-04T12:16:44.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:44.127+0000 7fb3df451700 1 -- 172.21.15.86:0/482644509 wait complete. 2024-03-04T12:16:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:43 smithi060 ceph-mon[32745]: Updating smithi086:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:16:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:43 smithi060 ceph-mon[32745]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:16:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:43 smithi060 ceph-mon[32745]: Updating smithi086:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T12:16:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:44.736 DEBUG:teuthology.orchestra.run.smithi063:mon.b> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.b.service 2024-03-04T12:16:44.738 DEBUG:teuthology.orchestra.run.smithi086:mon.c> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.c.service 2024-03-04T12:16:44.741 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T12:16:44.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T12:16:45.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:44 smithi060 ceph-mon[32745]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:45.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:44 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:45.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:44 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:16:45.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:44 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:16:45.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:44 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:16:45.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:44 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:45.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:44 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T12:16:45.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:44 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:16:46.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:45 smithi060 ceph-mon[32745]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi060:172.21.15.60=a;smithi063:172.21.15.63=b;smithi086:172.21.15.86=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:16:46.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:45 smithi060 ceph-mon[32745]: Saving service mon spec with placement smithi060:172.21.15.60=a;smithi063:172.21.15.63=b;smithi086:172.21.15.86=c;count:3 2024-03-04T12:16:46.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:45 smithi060 ceph-mon[32745]: Deploying daemon mon.c on smithi086 2024-03-04T12:16:46.465 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:16:47.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:46 smithi060 ceph-mon[32745]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:48.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:48 smithi060 ceph-mon[32745]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:48.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.459+0000 7fef2c424700 1 -- 172.21.15.86:0/766859273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef241066f0 msgr2=0x7fef24106ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:48.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.459+0000 7fef2c424700 1 --2- 172.21.15.86:0/766859273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef241066f0 0x7fef24106ad0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fef20009910 tx=0x7fef20009c20 comp rx=0 tx=0).stop 2024-03-04T12:16:48.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.459+0000 7fef2c424700 1 -- 172.21.15.86:0/766859273 shutdown_connections 2024-03-04T12:16:48.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.459+0000 7fef2c424700 1 --2- 172.21.15.86:0/766859273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef241066f0 0x7fef24106ad0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:48.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.459+0000 7fef2c424700 1 -- 172.21.15.86:0/766859273 >> 172.21.15.86:0/766859273 conn(0x7fef24077360 msgr2=0x7fef240fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:48.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.459+0000 7fef2c424700 1 -- 172.21.15.86:0/766859273 shutdown_connections 2024-03-04T12:16:48.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.459+0000 7fef2c424700 1 -- 172.21.15.86:0/766859273 wait complete. 2024-03-04T12:16:48.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.460+0000 7fef2c424700 1 Processor -- start 2024-03-04T12:16:48.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.460+0000 7fef2c424700 1 -- start start 2024-03-04T12:16:48.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.460+0000 7fef2c424700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef241066f0 0x7fef241972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:48.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.460+0000 7fef2c424700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2410ad30 con 0x7fef241066f0 2024-03-04T12:16:48.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.460+0000 7fef2a1c0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef241066f0 0x7fef241972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:48.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.460+0000 7fef2a1c0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef241066f0 0x7fef241972d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.86:37392/0 (socket says 172.21.15.86:37392) 2024-03-04T12:16:48.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.460+0000 7fef2a1c0700 1 -- 172.21.15.86:0/2642574752 learned_addr learned my addr 172.21.15.86:0/2642574752 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-03-04T12:16:48.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.461+0000 7fef2a1c0700 1 -- 172.21.15.86:0/2642574752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef24197810 con 0x7fef241066f0 2024-03-04T12:16:48.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.461+0000 7fef2a1c0700 1 --2- 172.21.15.86:0/2642574752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef241066f0 0x7fef241972d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fef20005850 tx=0x7fef20005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:48.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.461+0000 7fef1b7fe700 1 -- 172.21.15.86:0/2642574752 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef20026070 con 0x7fef241066f0 2024-03-04T12:16:48.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.461+0000 7fef2c424700 1 -- 172.21.15.86:0/2642574752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef24197b00 con 0x7fef241066f0 2024-03-04T12:16:48.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.461+0000 7fef1b7fe700 1 -- 172.21.15.86:0/2642574752 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef2000bbd0 con 0x7fef241066f0 2024-03-04T12:16:48.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.461+0000 7fef1b7fe700 1 -- 172.21.15.86:0/2642574752 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef20003970 con 0x7fef241066f0 2024-03-04T12:16:48.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.461+0000 7fef2c424700 1 -- 172.21.15.86:0/2642574752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2419bba0 con 0x7fef241066f0 2024-03-04T12:16:48.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.462+0000 7fef1b7fe700 1 -- 172.21.15.86:0/2642574752 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7fef20029070 con 0x7fef241066f0 2024-03-04T12:16:48.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.462+0000 7fef2c424700 1 -- 172.21.15.86:0/2642574752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef24101e30 con 0x7fef241066f0 2024-03-04T12:16:48.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.462+0000 7fef1b7fe700 1 --2- 172.21.15.86:0/2642574752 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fef10038ca0 0x7fef1003b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:48.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.462+0000 7fef1b7fe700 1 -- 172.21.15.86:0/2642574752 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fef2001e070 con 0x7fef241066f0 2024-03-04T12:16:48.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.462+0000 7fef299bf700 1 --2- 172.21.15.86:0/2642574752 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fef10038ca0 0x7fef1003b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:48.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.463+0000 7fef299bf700 1 --2- 172.21.15.86:0/2642574752 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fef10038ca0 0x7fef1003b160 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fef14006fd0 tx=0x7fef14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:48.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.468+0000 7fef1b7fe700 1 -- 172.21.15.86:0/2642574752 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fef20013070 con 0x7fef241066f0 2024-03-04T12:16:48.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.748+0000 7fef2c424700 1 -- 172.21.15.86:0/2642574752 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef24198630 con 0x7fef241066f0 2024-03-04T12:16:48.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.749+0000 7fef1b7fe700 1 -- 172.21.15.86:0/2642574752 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7fef20030020 con 0x7fef241066f0 2024-03-04T12:16:48.753 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:16:48.753 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","modified":"2024-03-04T12:15:05.558882Z","created":"2024-03-04T12:15:05.558882Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T12:16:48.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.751+0000 7fef2c424700 1 -- 172.21.15.86:0/2642574752 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fef10038ca0 msgr2=0x7fef1003b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:48.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.751+0000 7fef2c424700 1 --2- 172.21.15.86:0/2642574752 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fef10038ca0 0x7fef1003b160 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fef14006fd0 tx=0x7fef14006e40 comp rx=0 tx=0).stop 2024-03-04T12:16:48.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.751+0000 7fef2c424700 1 -- 172.21.15.86:0/2642574752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef241066f0 msgr2=0x7fef241972d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:48.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.751+0000 7fef2c424700 1 --2- 172.21.15.86:0/2642574752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef241066f0 0x7fef241972d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fef20005850 tx=0x7fef20005030 comp rx=0 tx=0).stop 2024-03-04T12:16:48.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.751+0000 7fef2c424700 1 -- 172.21.15.86:0/2642574752 shutdown_connections 2024-03-04T12:16:48.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.751+0000 7fef2c424700 1 --2- 172.21.15.86:0/2642574752 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fef10038ca0 0x7fef1003b160 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:48.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.751+0000 7fef2c424700 1 --2- 172.21.15.86:0/2642574752 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fef241066f0 0x7fef241972d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:48.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.751+0000 7fef2c424700 1 -- 172.21.15.86:0/2642574752 >> 172.21.15.86:0/2642574752 conn(0x7fef24077360 msgr2=0x7fef24104200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:48.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.751+0000 7fef2c424700 1 -- 172.21.15.86:0/2642574752 shutdown_connections 2024-03-04T12:16:48.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:48.751+0000 7fef2c424700 1 -- 172.21.15.86:0/2642574752 wait complete. 2024-03-04T12:16:48.755 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-03-04T12:16:49.007 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:48 smithi086 ceph-mon[36851]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-04T12:16:49.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:49 smithi086 bash[36793]: 4bc60bafa8844131556585056afd940b1a1743599b3eb55cabf9c8aabe1869d6 2024-03-04T12:16:49.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:49 smithi086 systemd[1]: Started Ceph mon.c for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:16:50.411 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T12:16:50.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T12:16:50.656 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:16:54.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:16:54.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: mon.a calling monitor election 2024-03-04T12:16:54.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:54.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:54.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:54.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: mon.c calling monitor election 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: monmap e2: 2 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: fsmap 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: mgrmap e13: x(active, since 49s) 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: overall HEALTH_OK 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T12:16:54.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:16:54.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:16:54.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: mon.a calling monitor election 2024-03-04T12:16:54.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:54.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:54.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:54.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: mon.c calling monitor election 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: monmap e2: 2 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: fsmap 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: mgrmap e13: x(active, since 49s) 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: overall HEALTH_OK 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:16:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T12:16:54.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:16:55.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.221+0000 7f626dbd8700 1 -- 172.21.15.86:0/3052784480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540035c0 msgr2=0x7f6254005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:55.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.221+0000 7f626dbd8700 1 --2- 172.21.15.86:0/3052784480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540035c0 0x7f6254005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6268101830 tx=0x7f6260009e80 comp rx=0 tx=0).stop 2024-03-04T12:16:55.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.221+0000 7f626dbd8700 1 -- 172.21.15.86:0/3052784480 shutdown_connections 2024-03-04T12:16:55.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.221+0000 7f626dbd8700 1 --2- 172.21.15.86:0/3052784480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f62540035c0 0x7f6254005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:55.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.221+0000 7f626dbd8700 1 -- 172.21.15.86:0/3052784480 >> 172.21.15.86:0/3052784480 conn(0x7f62680fd3f0 msgr2=0x7f62680ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:55.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.221+0000 7f626dbd8700 1 -- 172.21.15.86:0/3052784480 shutdown_connections 2024-03-04T12:16:55.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.221+0000 7f626dbd8700 1 -- 172.21.15.86:0/3052784480 wait complete. 2024-03-04T12:16:55.227 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:54 smithi086 ceph-mon[36851]: Deploying daemon mon.b on smithi063 2024-03-04T12:16:55.227 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:54 smithi086 ceph-mon[36851]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:55.227 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:54 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:55.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.222+0000 7f626dbd8700 1 Processor -- start 2024-03-04T12:16:55.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.223+0000 7f626dbd8700 1 -- start start 2024-03-04T12:16:55.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.223+0000 7f626dbd8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62540035c0 0x7f6268074120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:55.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.223+0000 7f626dbd8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6268074660 0x7f6268074ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.223+0000 7f626dbd8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62681099a0 con 0x7f6268074660 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.223+0000 7f626dbd8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6268109820 con 0x7f62540035c0 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.223+0000 7f62677fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62540035c0 0x7f6268074120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.223+0000 7f62677fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62540035c0 0x7f6268074120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53418/0 (socket says 172.21.15.86:53418) 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.223+0000 7f62677fe700 1 -- 172.21.15.86:0/2343304790 learned_addr learned my addr 172.21.15.86:0/2343304790 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.223+0000 7f6266ffd700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6268074660 0x7f6268074ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f62677fe700 1 -- 172.21.15.86:0/2343304790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62540035c0 msgr2=0x7f6268074120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f62677fe700 1 -- 172.21.15.86:0/2343304790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62540035c0 msgr2=0x7f6268074120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f62677fe700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62540035c0 0x7f6268074120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f62677fe700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62540035c0 0x7f6268074120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f6266ffd700 1 -- 172.21.15.86:0/2343304790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62540035c0 msgr2=0x7f6268074120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f6266ffd700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62540035c0 0x7f6268074120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f6266ffd700 1 -- 172.21.15.86:0/2343304790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6268076040 con 0x7f6268074660 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f6266ffd700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6268074660 0x7f6268074ae0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f625000b4d0 tx=0x7f625000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:55.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f6264ff9700 1 -- 172.21.15.86:0/2343304790 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f6250011630 con 0x7f6268074660 2024-03-04T12:16:55.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f626dbd8700 1 -- 172.21.15.86:0/2343304790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62681a3780 con 0x7f6268074660 2024-03-04T12:16:55.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f6264ff9700 1 -- 172.21.15.86:0/2343304790 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6250011cb0 con 0x7f6268074660 2024-03-04T12:16:55.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f6264ff9700 1 -- 172.21.15.86:0/2343304790 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f6250010e60 con 0x7f6268074660 2024-03-04T12:16:55.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.224+0000 7f626dbd8700 1 -- 172.21.15.86:0/2343304790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62681a3cc0 con 0x7f6268074660 2024-03-04T12:16:55.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.225+0000 7f6264ff9700 1 -- 172.21.15.86:0/2343304790 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7f6250018020 con 0x7f6268074660 2024-03-04T12:16:55.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.226+0000 7f6264ff9700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6248038990 0x7f624803ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:16:55.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.226+0000 7f626dbd8700 1 -- 172.21.15.86:0/2343304790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6268103040 con 0x7f6268074660 2024-03-04T12:16:55.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.226+0000 7f6264ff9700 1 -- 172.21.15.86:0/2343304790 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f625004b590 con 0x7f6268074660 2024-03-04T12:16:55.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.226+0000 7f62677fe700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6248038990 0x7f624803ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:16:55.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.226+0000 7f62677fe700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6248038990 0x7f624803ae50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6260019000 tx=0x7f62600074e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:16:55.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.231+0000 7f6264ff9700 1 -- 172.21.15.86:0/2343304790 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f625001f020 con 0x7f6268074660 2024-03-04T12:16:55.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:54 smithi060 ceph-mon[32745]: Deploying daemon mon.b on smithi063 2024-03-04T12:16:55.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:54 smithi060 ceph-mon[32745]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:55.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:54 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:16:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.509+0000 7f626dbd8700 1 -- 172.21.15.86:0/2343304790 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f626802d870 con 0x7f6268074660 2024-03-04T12:16:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.510+0000 7f6264ff9700 1 -- 172.21.15.86:0/2343304790 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1011 (secure 0 0 0) 0x7f625001ed10 con 0x7f6268074660 2024-03-04T12:16:55.514 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:16:55.514 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":2,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","modified":"2024-03-04T12:16:48.826510Z","created":"2024-03-04T12:15:05.558882Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-04T12:16:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.513+0000 7f626dbd8700 1 -- 172.21.15.86:0/2343304790 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6248038990 msgr2=0x7f624803ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.513+0000 7f626dbd8700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6248038990 0x7f624803ae50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6260019000 tx=0x7f62600074e0 comp rx=0 tx=0).stop 2024-03-04T12:16:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.513+0000 7f626dbd8700 1 -- 172.21.15.86:0/2343304790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6268074660 msgr2=0x7f6268074ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:16:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.513+0000 7f626dbd8700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6268074660 0x7f6268074ae0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f625000b4d0 tx=0x7f625000b890 comp rx=0 tx=0).stop 2024-03-04T12:16:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.513+0000 7f626dbd8700 1 -- 172.21.15.86:0/2343304790 shutdown_connections 2024-03-04T12:16:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.513+0000 7f626dbd8700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62540035c0 0x7f6268074120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.513+0000 7f626dbd8700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6248038990 0x7f624803ae50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.513+0000 7f626dbd8700 1 --2- 172.21.15.86:0/2343304790 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6268074660 0x7f6268074ae0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:16:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.513+0000 7f626dbd8700 1 -- 172.21.15.86:0/2343304790 >> 172.21.15.86:0/2343304790 conn(0x7f62680fd3f0 msgr2=0x7f62680ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:16:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.513+0000 7f626dbd8700 1 -- 172.21.15.86:0/2343304790 shutdown_connections 2024-03-04T12:16:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:16:55.513+0000 7f626dbd8700 1 -- 172.21.15.86:0/2343304790 wait complete. 2024-03-04T12:16:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 2 2024-03-04T12:16:56.228 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:55 smithi086 ceph-mon[36851]: from='client.? 172.21.15.86:0/2343304790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T12:16:56.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:55 smithi060 ceph-mon[32745]: from='client.? 172.21.15.86:0/2343304790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T12:16:57.260 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T12:16:57.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T12:16:57.291 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:16:56 smithi086 ceph-mon[36851]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:57.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:16:56 smithi060 ceph-mon[32745]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:16:57.504 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:16:58.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:16:58 smithi063 systemd[1]: Started Ceph mon.b for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:17:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.260+0000 7ff65d0b0700 1 -- 172.21.15.86:0/2350823355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6400035c0 msgr2=0x7ff640005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.260+0000 7ff65d0b0700 1 --2- 172.21.15.86:0/2350823355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6400035c0 0x7ff640005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff658100750 tx=0x7ff650009e80 comp rx=0 tx=0).stop 2024-03-04T12:17:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.260+0000 7ff65d0b0700 1 -- 172.21.15.86:0/2350823355 shutdown_connections 2024-03-04T12:17:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.260+0000 7ff65d0b0700 1 --2- 172.21.15.86:0/2350823355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6400035c0 0x7ff640005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.260+0000 7ff65d0b0700 1 -- 172.21.15.86:0/2350823355 >> 172.21.15.86:0/2350823355 conn(0x7ff6580fc330 msgr2=0x7ff6580fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:17:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.261+0000 7ff65d0b0700 1 -- 172.21.15.86:0/2350823355 shutdown_connections 2024-03-04T12:17:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.261+0000 7ff65d0b0700 1 -- 172.21.15.86:0/2350823355 wait complete. 2024-03-04T12:17:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.262+0000 7ff65d0b0700 1 Processor -- start 2024-03-04T12:17:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.262+0000 7ff65d0b0700 1 -- start start 2024-03-04T12:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.262+0000 7ff65d0b0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6400035c0 0x7ff658197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.262+0000 7ff65d0b0700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff658197a50 0x7ff65819bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.262+0000 7ff65d0b0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65819c320 0x7ff65819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.262+0000 7ff65d0b0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff658108870 con 0x7ff6400035c0 2024-03-04T12:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.262+0000 7ff65d0b0700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6581086f0 con 0x7ff658197a50 2024-03-04T12:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.262+0000 7ff65d0b0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6581089f0 con 0x7ff65819c320 2024-03-04T12:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff65759e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65819c320 0x7ff65819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff656d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6400035c0 0x7ff658197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff656d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6400035c0 0x7ff658197510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.86:48988/0 (socket says 172.21.15.86:48988) 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff656d9d700 1 -- 172.21.15.86:0/96104281 learned_addr learned my addr 172.21.15.86:0/96104281 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff65659c700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff658197a50 0x7ff65819bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff65759e700 1 -- 172.21.15.86:0/96104281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65819c320 msgr2=0x7ff65819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff65759e700 1 -- 172.21.15.86:0/96104281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65819c320 msgr2=0x7ff65819c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff65759e700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65819c320 0x7ff65819c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff65759e700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65819c320 0x7ff65819c7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff65659c700 1 -- 172.21.15.86:0/96104281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff658197a50 msgr2=0x7ff65819bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff65659c700 1 -- 172.21.15.86:0/96104281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff658197a50 msgr2=0x7ff65819bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff65659c700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff658197a50 0x7ff65819bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff65659c700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff658197a50 0x7ff65819bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff656d9d700 1 -- 172.21.15.86:0/96104281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65819c320 msgr2=0x7ff65819c7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff656d9d700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65819c320 0x7ff65819c7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff656d9d700 1 -- 172.21.15.86:0/96104281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff658197a50 msgr2=0x7ff65819bde0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff656d9d700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff658197a50 0x7ff65819bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff656d9d700 1 -- 172.21.15.86:0/96104281 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6581a4090 con 0x7ff6400035c0 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff656d9d700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6400035c0 0x7ff658197510 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff650004890 tx=0x7ff650004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.263+0000 7ff63ffff700 1 -- 172.21.15.86:0/96104281 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650004be0 con 0x7ff6400035c0 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.264+0000 7ff63ffff700 1 -- 172.21.15.86:0/96104281 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff650004d80 con 0x7ff6400035c0 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.264+0000 7ff63ffff700 1 -- 172.21.15.86:0/96104281 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650003a00 con 0x7ff6400035c0 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.264+0000 7ff65d0b0700 1 -- 172.21.15.86:0/96104281 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6581a4260 con 0x7ff6400035c0 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.264+0000 7ff65d0b0700 1 -- 172.21.15.86:0/96104281 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6581a47f0 con 0x7ff6400035c0 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.264+0000 7ff63ffff700 1 -- 172.21.15.86:0/96104281 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7ff650015040 con 0x7ff6400035c0 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.265+0000 7ff63ffff700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff6440390d0 0x7ff64403b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.265+0000 7ff65d0b0700 1 -- 172.21.15.86:0/96104281 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff658063450 con 0x7ff6400035c0 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.265+0000 7ff63ffff700 1 -- 172.21.15.86:0/96104281 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff65004f270 con 0x7ff6400035c0 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.265+0000 7ff65659c700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff6440390d0 0x7ff64403b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.265+0000 7ff65659c700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff6440390d0 0x7ff64403b590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009850 tx=0x7ff64c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:17:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.270+0000 7ff63ffff700 1 -- 172.21.15.86:0/96104281 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff650027070 con 0x7ff6400035c0 2024-03-04T12:17:03.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:17:03.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: mon.a calling monitor election 2024-03-04T12:17:03.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: mon.c calling monitor election 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: monmap e3: 3 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],b=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: fsmap 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T12:17:03.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: mgrmap e13: x(active, since 59s) 2024-03-04T12:17:03.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T12:17:03.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T12:17:03.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T12:17:03.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: mon.b (rank 2) addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] is down (out of quorum) 2024-03-04T12:17:03.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:03.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:03.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:03.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:03.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.561+0000 7ff65d0b0700 1 -- 172.21.15.86:0/96104281 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6581a4a50 con 0x7ff6400035c0 2024-03-04T12:17:03.565 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:17:03.565 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: mon.a calling monitor election 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: mon.c calling monitor election 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: monmap e3: 3 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],b=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T12:17:03.566 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: fsmap 2024-03-04T12:17:03.567 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T12:17:03.567 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: mgrmap e13: x(active, since 59s) 2024-03-04T12:17:03.567 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T12:17:03.567 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T12:17:03.567 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T12:17:03.567 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: mon.b (rank 2) addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] is down (out of quorum) 2024-03-04T12:17:03.567 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:03.567 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:03.567 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:03.567 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:03.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.562+0000 7ff63ffff700 1 -- 172.21.15.86:0/96104281 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1275 (secure 0 0 0) 0x7ff650017640 con 0x7ff6400035c0 2024-03-04T12:17:03.568 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-04T12:17:03.568 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":3,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","modified":"2024-03-04T12:16:58.075649Z","created":"2024-03-04T12:15:05.558882Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:3300","nonce":0},{"type":"v1","addr":"172.21.15.60:6789","nonce":0}]},"addr":"172.21.15.60:6789/0","public_addr":"172.21.15.60:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-04T12:17:03.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.564+0000 7ff65d0b0700 1 -- 172.21.15.86:0/96104281 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff6440390d0 msgr2=0x7ff64403b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:03.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.564+0000 7ff65d0b0700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff6440390d0 0x7ff64403b590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009850 tx=0x7ff64c008040 comp rx=0 tx=0).stop 2024-03-04T12:17:03.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.564+0000 7ff65d0b0700 1 -- 172.21.15.86:0/96104281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6400035c0 msgr2=0x7ff658197510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:03.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.564+0000 7ff65d0b0700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6400035c0 0x7ff658197510 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff650004890 tx=0x7ff650004970 comp rx=0 tx=0).stop 2024-03-04T12:17:03.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.564+0000 7ff65d0b0700 1 -- 172.21.15.86:0/96104281 shutdown_connections 2024-03-04T12:17:03.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.564+0000 7ff65d0b0700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65819c320 0x7ff65819c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:03.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.564+0000 7ff65d0b0700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff6440390d0 0x7ff64403b590 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:03.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.564+0000 7ff65d0b0700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff6400035c0 0x7ff658197510 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:03.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.564+0000 7ff65d0b0700 1 --2- 172.21.15.86:0/96104281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff658197a50 0x7ff65819bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:03.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.564+0000 7ff65d0b0700 1 -- 172.21.15.86:0/96104281 >> 172.21.15.86:0/96104281 conn(0x7ff6580fc330 msgr2=0x7ff658105b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:17:03.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.564+0000 7ff65d0b0700 1 -- 172.21.15.86:0/96104281 shutdown_connections 2024-03-04T12:17:03.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:03.564+0000 7ff65d0b0700 1 -- 172.21.15.86:0/96104281 wait complete. 2024-03-04T12:17:03.570 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 3 2024-03-04T12:17:04.082 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-04T12:17:04.082 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph config generate-minimal-conf 2024-03-04T12:17:04.336 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:17:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:04 smithi060 ceph-mon[32745]: from='client.? 172.21.15.86:0/96104281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T12:17:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:04 smithi060 ceph-mon[32745]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:04.427 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:04.428 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:04 smithi086 ceph-mon[36851]: from='client.? 172.21.15.86:0/96104281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T12:17:04.428 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:04.428 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:04 smithi086 ceph-mon[36851]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:04.428 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:05.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:17:05.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: mon.a calling monitor election 2024-03-04T12:17:05.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: mon.c calling monitor election 2024-03-04T12:17:05.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:05.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: monmap e3: 3 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],b=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: fsmap 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: mgrmap e13: x(active, since 59s) 2024-03-04T12:17:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T12:17:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T12:17:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T12:17:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: mon.b (rank 2) addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] is down (out of quorum) 2024-03-04T12:17:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='client.? 172.21.15.86:0/96104281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T12:17:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:05.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.988+0000 7f7297aab700 1 -- 172.21.15.60:0/1193202867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72901077f0 msgr2=0x7f7290107bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.988+0000 7f7297aab700 1 --2- 172.21.15.60:0/1193202867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72901077f0 0x7f7290107bd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f728000b0a0 tx=0x7f728000b3b0 comp rx=0 tx=0).stop 2024-03-04T12:17:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.988+0000 7f7297aab700 1 -- 172.21.15.60:0/1193202867 shutdown_connections 2024-03-04T12:17:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.988+0000 7f7297aab700 1 --2- 172.21.15.60:0/1193202867 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72901077f0 0x7f7290107bd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.988+0000 7f7297aab700 1 -- 172.21.15.60:0/1193202867 >> 172.21.15.60:0/1193202867 conn(0x7f72900fd2e0 msgr2=0x7f72900ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:17:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.989+0000 7f7297aab700 1 -- 172.21.15.60:0/1193202867 shutdown_connections 2024-03-04T12:17:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.989+0000 7f7297aab700 1 -- 172.21.15.60:0/1193202867 wait complete. 2024-03-04T12:17:05.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.989+0000 7f7297aab700 1 Processor -- start 2024-03-04T12:17:05.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.989+0000 7f7297aab700 1 -- start start 2024-03-04T12:17:05.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.989+0000 7f7297aab700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72901077f0 0x7f7290074120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:05.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7297aab700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7290074660 0x7f7290074ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:05.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7297aab700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72901a3740 0x7f72901a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:05.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7297aab700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729010bde0 con 0x7f72901077f0 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7297aab700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729010bc60 con 0x7f7290074660 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7297aab700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f729010bf60 con 0x7f72901a3740 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7295847700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72901077f0 0x7f7290074120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7295847700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72901077f0 0x7f7290074120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51458/0 (socket says 172.21.15.60:51458) 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7295847700 1 -- 172.21.15.60:0/2689426361 learned_addr learned my addr 172.21.15.60:0/2689426361 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7295046700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7290074660 0x7f7290074ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7296048700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72901a3740 0x7f72901a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7295847700 1 -- 172.21.15.60:0/2689426361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72901a3740 msgr2=0x7f72901a5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7295847700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72901a3740 0x7f72901a5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7295847700 1 -- 172.21.15.60:0/2689426361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7290074660 msgr2=0x7f7290074ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7295847700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7290074660 0x7f7290074ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7295847700 1 -- 172.21.15.60:0/2689426361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72901a6260 con 0x7f72901077f0 2024-03-04T12:17:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7295847700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72901077f0 0x7f7290074120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7280007600 tx=0x7f72800094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:17:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f728effd700 1 -- 172.21.15.60:0/2689426361 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280020070 con 0x7f72901077f0 2024-03-04T12:17:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f728effd700 1 -- 172.21.15.60:0/2689426361 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72800055c0 con 0x7f72901077f0 2024-03-04T12:17:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f7297aab700 1 -- 172.21.15.60:0/2689426361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72901a64f0 con 0x7f72901077f0 2024-03-04T12:17:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.990+0000 7f728effd700 1 -- 172.21.15.60:0/2689426361 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7280017070 con 0x7f72901077f0 2024-03-04T12:17:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.991+0000 7f7297aab700 1 -- 172.21.15.60:0/2689426361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72901a6a80 con 0x7f72901077f0 2024-03-04T12:17:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.991+0000 7f728effd700 1 -- 172.21.15.60:0/2689426361 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7f7280012ba0 con 0x7f72901077f0 2024-03-04T12:17:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.991+0000 7f7297aab700 1 -- 172.21.15.60:0/2689426361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7290063450 con 0x7f72901077f0 2024-03-04T12:17:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.992+0000 7f728effd700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72780390d0 0x7f727803b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.992+0000 7f728effd700 1 -- 172.21.15.60:0/2689426361 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f728000f020 con 0x7f72901077f0 2024-03-04T12:17:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.992+0000 7f7295046700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72780390d0 0x7f727803b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:05.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.992+0000 7f7295046700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72780390d0 0x7f727803b590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7284009ad0 tx=0x7f7284009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:17:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:05.997+0000 7f728effd700 1 -- 172.21.15.60:0/2689426361 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f72800092e0 con 0x7f72901077f0 2024-03-04T12:17:06.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: mon.b calling monitor election 2024-03-04T12:17:06.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: mon.b calling monitor election 2024-03-04T12:17:06.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: mon.c calling monitor election 2024-03-04T12:17:06.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: mon.a calling monitor election 2024-03-04T12:17:06.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: monmap e3: 3 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],b=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: fsmap 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: mgrmap e13: x(active, since 61s) 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: Cluster is now healthy 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: overall HEALTH_OK 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T12:17:06.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T12:17:06.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:06.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.210+0000 7f7297aab700 1 -- 172.21.15.60:0/2689426361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f72901a6db0 con 0x7f72901077f0 2024-03-04T12:17:06.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.210+0000 7f728effd700 1 -- 172.21.15.60:0/2689426361 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+271 (secure 0 0 0) 0x7f7280022e20 con 0x7f72901077f0 2024-03-04T12:17:06.214 INFO:teuthology.orchestra.run.smithi060.stdout:# minimal ceph.conf for a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:17:06.214 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-03-04T12:17:06.214 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:17:06.214 INFO:teuthology.orchestra.run.smithi060.stdout: mon_host = [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-03-04T12:17:06.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.212+0000 7f7297aab700 1 -- 172.21.15.60:0/2689426361 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72780390d0 msgr2=0x7f727803b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:06.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.212+0000 7f7297aab700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72780390d0 0x7f727803b590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7284009ad0 tx=0x7f7284009210 comp rx=0 tx=0).stop 2024-03-04T12:17:06.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.212+0000 7f7297aab700 1 -- 172.21.15.60:0/2689426361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72901077f0 msgr2=0x7f7290074120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:06.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.212+0000 7f7297aab700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72901077f0 0x7f7290074120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7280007600 tx=0x7f72800094d0 comp rx=0 tx=0).stop 2024-03-04T12:17:06.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.212+0000 7f7297aab700 1 -- 172.21.15.60:0/2689426361 shutdown_connections 2024-03-04T12:17:06.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.212+0000 7f7297aab700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72901a3740 0x7f72901a5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:06.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.212+0000 7f7297aab700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72780390d0 0x7f727803b590 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:06.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.212+0000 7f7297aab700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72901077f0 0x7f7290074120 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:06.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.212+0000 7f7297aab700 1 --2- 172.21.15.60:0/2689426361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7290074660 0x7f7290074ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:06.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.212+0000 7f7297aab700 1 -- 172.21.15.60:0/2689426361 >> 172.21.15.60:0/2689426361 conn(0x7f72900fd2e0 msgr2=0x7f72901032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:17:06.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.212+0000 7f7297aab700 1 -- 172.21.15.60:0/2689426361 shutdown_connections 2024-03-04T12:17:06.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:06.212+0000 7f7297aab700 1 -- 172.21.15.60:0/2689426361 wait complete. 2024-03-04T12:17:06.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: mon.b calling monitor election 2024-03-04T12:17:06.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: mon.b calling monitor election 2024-03-04T12:17:06.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: mon.c calling monitor election 2024-03-04T12:17:06.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: mon.a calling monitor election 2024-03-04T12:17:06.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T12:17:06.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: monmap e3: 3 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],b=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T12:17:06.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: fsmap 2024-03-04T12:17:06.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T12:17:06.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: mgrmap e13: x(active, since 61s) 2024-03-04T12:17:06.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T12:17:06.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: Cluster is now healthy 2024-03-04T12:17:06.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: overall HEALTH_OK 2024-03-04T12:17:06.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T12:17:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T12:17:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:06.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: mon.b calling monitor election 2024-03-04T12:17:06.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: mon.b calling monitor election 2024-03-04T12:17:06.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: mon.c calling monitor election 2024-03-04T12:17:06.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: mon.a calling monitor election 2024-03-04T12:17:06.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T12:17:06.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: monmap e3: 3 mons at {a=[v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0],b=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],c=[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T12:17:06.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: fsmap 2024-03-04T12:17:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T12:17:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: mgrmap e13: x(active, since 61s) 2024-03-04T12:17:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T12:17:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: Cluster is now healthy 2024-03-04T12:17:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: overall HEALTH_OK 2024-03-04T12:17:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T12:17:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T12:17:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:06.854 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-04T12:17:06.854 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:17:06.855 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T12:17:06.884 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:17:06.884 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:17:06.954 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-04T12:17:06.954 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T12:17:06.994 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-04T12:17:06.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:17:07.068 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T12:17:07.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T12:17:07.103 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T12:17:07.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:17:07.173 INFO:tasks.cephadm:Adding mgr.x on smithi060 2024-03-04T12:17:07.173 INFO:tasks.cephadm:Adding mgr.y on smithi063 2024-03-04T12:17:07.173 INFO:tasks.cephadm:Adding mgr.z on smithi086 2024-03-04T12:17:07.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch apply mgr '3;smithi060=x;smithi063=y;smithi086=z' 2024-03-04T12:17:07.175 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:06 smithi060 ceph-mon[32745]: Updating smithi063:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:17:07.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:06 smithi060 ceph-mon[32745]: Updating smithi086:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:17:07.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:06 smithi060 ceph-mon[32745]: Updating smithi060:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:17:07.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:06 smithi060 ceph-mon[32745]: Reconfiguring mon.a (unknown last config time)... 2024-03-04T12:17:07.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:06 smithi060 ceph-mon[32745]: Reconfiguring daemon mon.a on smithi060 2024-03-04T12:17:07.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:06 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:07.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:06 smithi060 ceph-mon[32745]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:07.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:06 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2689426361' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:07.239 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:06 smithi086 ceph-mon[36851]: Updating smithi063:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:17:07.240 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:06 smithi086 ceph-mon[36851]: Updating smithi086:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:17:07.240 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:06 smithi086 ceph-mon[36851]: Updating smithi060:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:17:07.240 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:06 smithi086 ceph-mon[36851]: Reconfiguring mon.a (unknown last config time)... 2024-03-04T12:17:07.240 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:06 smithi086 ceph-mon[36851]: Reconfiguring daemon mon.a on smithi060 2024-03-04T12:17:07.240 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:06 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:07.240 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:06 smithi086 ceph-mon[36851]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:07.240 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:06 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2689426361' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:07.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:06 smithi063 ceph-mon[36717]: Updating smithi063:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:17:07.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:06 smithi063 ceph-mon[36717]: Updating smithi086:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:17:07.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:06 smithi063 ceph-mon[36717]: Updating smithi060:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:17:07.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:06 smithi063 ceph-mon[36717]: Reconfiguring mon.a (unknown last config time)... 2024-03-04T12:17:07.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:06 smithi063 ceph-mon[36717]: Reconfiguring daemon mon.a on smithi060 2024-03-04T12:17:07.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:06 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:17:07.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:06 smithi063 ceph-mon[36717]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:07.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:06 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2689426361' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:17:08.812 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:08 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:08.813 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:08 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:08.813 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:08 smithi063 ceph-mon[36717]: Reconfiguring mon.b (monmap changed)... 2024-03-04T12:17:08.813 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:08 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T12:17:08.813 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:08 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T12:17:08.813 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:08 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:08.813 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:08 smithi063 ceph-mon[36717]: Reconfiguring daemon mon.b on smithi063 2024-03-04T12:17:08.813 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:08 smithi063 ceph-mon[36717]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:08.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.833+0000 7f50b7e35700 1 -- 172.21.15.86:0/1339139294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b0100880 msgr2=0x7f50b0100ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:08.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.833+0000 7f50b7e35700 1 --2- 172.21.15.86:0/1339139294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b0100880 0x7f50b0100ca0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f50a0009910 tx=0x7f50a0009c20 comp rx=0 tx=0).stop 2024-03-04T12:17:08.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.833+0000 7f50b7e35700 1 -- 172.21.15.86:0/1339139294 shutdown_connections 2024-03-04T12:17:08.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.833+0000 7f50b7e35700 1 --2- 172.21.15.86:0/1339139294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b0100880 0x7f50b0100ca0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:08.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.833+0000 7f50b7e35700 1 -- 172.21.15.86:0/1339139294 >> 172.21.15.86:0/1339139294 conn(0x7f50b00fbe00 msgr2=0x7f50b00fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:17:08.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.833+0000 7f50b7e35700 1 -- 172.21.15.86:0/1339139294 shutdown_connections 2024-03-04T12:17:08.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.833+0000 7f50b7e35700 1 -- 172.21.15.86:0/1339139294 wait complete. 2024-03-04T12:17:08.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.833+0000 7f50b7e35700 1 Processor -- start 2024-03-04T12:17:08.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.833+0000 7f50b7e35700 1 -- start start 2024-03-04T12:17:08.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b7e35700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b0100880 0x7f50b0196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:08.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b7e35700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b01965d0 0x7f50b019b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:08.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b7e35700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50b0196a50 0x7f50b019dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:08.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b7e35700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b0106910 con 0x7f50b0100880 2024-03-04T12:17:08.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b7e35700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b0106790 con 0x7f50b0196a50 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b7e35700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b0106a90 con 0x7f50b01965d0 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b53d0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b01965d0 0x7f50b019b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b53d0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b01965d0 0x7f50b019b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34238/0 (socket says 172.21.15.86:34238) 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b53d0700 1 -- 172.21.15.86:0/2567721906 learned_addr learned my addr 172.21.15.86:0/2567721906 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b63d2700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50b0196a50 0x7f50b019dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b5bd1700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b0100880 0x7f50b0196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b53d0700 1 -- 172.21.15.86:0/2567721906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50b0196a50 msgr2=0x7f50b019dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b53d0700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50b0196a50 0x7f50b019dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b53d0700 1 -- 172.21.15.86:0/2567721906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b0100880 msgr2=0x7f50b0196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b53d0700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b0100880 0x7f50b0196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b53d0700 1 -- 172.21.15.86:0/2567721906 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b019e4a0 con 0x7f50b01965d0 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b53d0700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b01965d0 0x7f50b019b640 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00b780 tx=0x7f50ac00ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50a6ffd700 1 -- 172.21.15.86:0/2567721906 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac004230 con 0x7f50b01965d0 2024-03-04T12:17:08.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50a6ffd700 1 -- 172.21.15.86:0/2567721906 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50ac0043d0 con 0x7f50b01965d0 2024-03-04T12:17:08.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50b7e35700 1 -- 172.21.15.86:0/2567721906 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b019e790 con 0x7f50b01965d0 2024-03-04T12:17:08.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.834+0000 7f50a6ffd700 1 -- 172.21.15.86:0/2567721906 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac011c90 con 0x7f50b01965d0 2024-03-04T12:17:08.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.835+0000 7f50b7e35700 1 -- 172.21.15.86:0/2567721906 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b019ed50 con 0x7f50b01965d0 2024-03-04T12:17:08.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.836+0000 7f50a6ffd700 1 -- 172.21.15.86:0/2567721906 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7f50ac01a400 con 0x7f50b01965d0 2024-03-04T12:17:08.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.836+0000 7f50b7e35700 1 -- 172.21.15.86:0/2567721906 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b0063450 con 0x7f50b01965d0 2024-03-04T12:17:08.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.836+0000 7f50a6ffd700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f509c0390d0 0x7f509c03b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:08.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.836+0000 7f50a6ffd700 1 -- 172.21.15.86:0/2567721906 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f50ac04b7a0 con 0x7f50b01965d0 2024-03-04T12:17:08.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.836+0000 7f50b5bd1700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f509c0390d0 0x7f509c03b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:08.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.836+0000 7f50b5bd1700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f509c0390d0 0x7f509c03b590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f50a000b5c0 tx=0x7f50a001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:17:08.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:08.841+0000 7f50a6ffd700 1 -- 172.21.15.86:0/2567721906 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f50ac018350 con 0x7f50b01965d0 2024-03-04T12:17:09.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.043+0000 7f50b7e35700 1 -- 172.21.15.86:0/2567721906 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi060=x;smithi063=y;smithi086=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f50b00622f0 con 0x7f509c0390d0 2024-03-04T12:17:09.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:08 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:09.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:08 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:09.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:08 smithi086 ceph-mon[36851]: Reconfiguring mon.b (monmap changed)... 2024-03-04T12:17:09.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:08 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T12:17:09.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:08 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T12:17:09.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:08 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:09.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:08 smithi086 ceph-mon[36851]: Reconfiguring daemon mon.b on smithi063 2024-03-04T12:17:09.048 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:08 smithi086 ceph-mon[36851]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:09.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.050+0000 7f50a6ffd700 1 -- 172.21.15.86:0/2567721906 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f50b00622f0 con 0x7f509c0390d0 2024-03-04T12:17:09.054 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mgr update... 2024-03-04T12:17:09.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.052+0000 7f50b7e35700 1 -- 172.21.15.86:0/2567721906 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f509c0390d0 msgr2=0x7f509c03b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:09.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.052+0000 7f50b7e35700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f509c0390d0 0x7f509c03b590 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f50a000b5c0 tx=0x7f50a001a040 comp rx=0 tx=0).stop 2024-03-04T12:17:09.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.052+0000 7f50b7e35700 1 -- 172.21.15.86:0/2567721906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b01965d0 msgr2=0x7f50b019b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:09.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.052+0000 7f50b7e35700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b01965d0 0x7f50b019b640 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00b780 tx=0x7f50ac00ba90 comp rx=0 tx=0).stop 2024-03-04T12:17:09.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.053+0000 7f50b7e35700 1 -- 172.21.15.86:0/2567721906 shutdown_connections 2024-03-04T12:17:09.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.053+0000 7f50b7e35700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50b0196a50 0x7f50b019dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:09.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.053+0000 7f50b7e35700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f509c0390d0 0x7f509c03b590 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:09.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.053+0000 7f50b7e35700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f50b0100880 0x7f50b0196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:09.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.053+0000 7f50b7e35700 1 --2- 172.21.15.86:0/2567721906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b01965d0 0x7f50b019b640 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:09.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.053+0000 7f50b7e35700 1 -- 172.21.15.86:0/2567721906 >> 172.21.15.86:0/2567721906 conn(0x7f50b00fbe00 msgr2=0x7f50b00fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:17:09.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.053+0000 7f50b7e35700 1 -- 172.21.15.86:0/2567721906 shutdown_connections 2024-03-04T12:17:09.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:17:09.053+0000 7f50b7e35700 1 -- 172.21.15.86:0/2567721906 wait complete. 2024-03-04T12:17:09.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:08 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:09.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:08 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:09.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:08 smithi060 ceph-mon[32745]: Reconfiguring mon.b (monmap changed)... 2024-03-04T12:17:09.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:08 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T12:17:09.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:08 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T12:17:09.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:08 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:09.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:08 smithi060 ceph-mon[32745]: Reconfiguring daemon mon.b on smithi063 2024-03-04T12:17:09.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:08 smithi060 ceph-mon[32745]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:09.658 DEBUG:teuthology.orchestra.run.smithi063:mgr.y> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.y.service 2024-03-04T12:17:09.661 DEBUG:teuthology.orchestra.run.smithi086:mgr.z> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.z.service 2024-03-04T12:17:09.701 INFO:tasks.cephadm:Deploying OSDs... 2024-03-04T12:17:09.701 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:17:09.701 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T12:17:09.720 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T12:17:09.720 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-03-04T12:17:09.778 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T12:17:09.778 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T12:17:09.778 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-04T12:17:09.778 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:17:09.778 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T12:17:09.778 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-04 12:16:16.245584148 +0000 2024-03-04T12:17:09.778 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-04 12:10:25.091771150 +0000 2024-03-04T12:17:09.778 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-04 12:10:25.091771150 +0000 2024-03-04T12:17:09.778 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-04 12:10:25.091771150 +0000 2024-03-04T12:17:09.778 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T12:17:09.849 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-03-04T12:17:09.849 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-03-04T12:17:09.849 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.00012119 s, 4.2 MB/s 2024-03-04T12:17:09.850 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T12:17:09.907 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-03-04T12:17:09.964 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T12:17:09.964 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T12:17:09.965 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-04T12:17:09.965 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:17:09.965 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T12:17:09.965 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-04 12:16:16.282583773 +0000 2024-03-04T12:17:09.965 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-04 12:10:25.440767624 +0000 2024-03-04T12:17:09.965 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-04 12:10:25.440767624 +0000 2024-03-04T12:17:09.965 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-04 12:10:25.440767624 +0000 2024-03-04T12:17:09.965 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T12:17:10.032 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-03-04T12:17:10.032 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-03-04T12:17:10.032 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000113432 s, 4.5 MB/s 2024-03-04T12:17:10.033 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T12:17:10.090 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-03-04T12:17:10.147 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T12:17:10.147 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T12:17:10.147 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-04T12:17:10.147 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:17:10.148 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T12:17:10.148 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-04 12:16:16.314583448 +0000 2024-03-04T12:17:10.148 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-04 12:10:25.780764188 +0000 2024-03-04T12:17:10.148 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-04 12:10:25.780764188 +0000 2024-03-04T12:17:10.148 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-04 12:10:25.780764188 +0000 2024-03-04T12:17:10.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T12:17:10.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:10 smithi060 ceph-mon[32745]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi060=x;smithi063=y;smithi086=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:17:10.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:10 smithi060 ceph-mon[32745]: Saving service mgr spec with placement smithi060=x;smithi063=y;smithi086=z;count:3 2024-03-04T12:17:10.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:10 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:10.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:10 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:10.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:10 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:10.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:10 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T12:17:10.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:10 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T12:17:10.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:10 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-03-04T12:17:10.202 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-03-04T12:17:10.203 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000113658 s, 4.5 MB/s 2024-03-04T12:17:10.204 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T12:17:10.260 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-03-04T12:17:10.316 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T12:17:10.316 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T12:17:10.316 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-04T12:17:10.316 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:17:10.316 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T12:17:10.316 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-04 12:16:16.345583133 +0000 2024-03-04T12:17:10.316 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-04 12:10:26.123760723 +0000 2024-03-04T12:17:10.316 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-04 12:10:26.123760723 +0000 2024-03-04T12:17:10.316 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-04 12:10:26.123760723 +0000 2024-03-04T12:17:10.316 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T12:17:10.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:10 smithi086 ceph-mon[36851]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi060=x;smithi063=y;smithi086=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:17:10.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:10 smithi086 ceph-mon[36851]: Saving service mgr spec with placement smithi060=x;smithi063=y;smithi086=z;count:3 2024-03-04T12:17:10.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:10 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:10.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:10 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:10.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:10 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:10.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:10 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T12:17:10.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:10 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T12:17:10.333 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:10 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:10.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:10 smithi063 ceph-mon[36717]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi060=x;smithi063=y;smithi086=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:17:10.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:10 smithi063 ceph-mon[36717]: Saving service mgr spec with placement smithi060=x;smithi063=y;smithi086=z;count:3 2024-03-04T12:17:10.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:10 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:10.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:10 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:10.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:10 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:10.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:10 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T12:17:10.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:10 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T12:17:10.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:10 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:10.382 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-03-04T12:17:10.382 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-03-04T12:17:10.383 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.6132e-05 s, 5.3 MB/s 2024-03-04T12:17:10.383 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T12:17:10.439 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-03-04T12:17:10.439 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T12:17:10.458 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T12:17:10.459 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-03-04T12:17:10.517 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T12:17:10.518 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T12:17:10.518 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-04T12:17:10.518 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:17:10.518 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T12:17:10.518 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-04 12:16:57.510073947 +0000 2024-03-04T12:17:10.518 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-04 12:11:08.797195319 +0000 2024-03-04T12:17:10.518 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-04 12:11:08.797195319 +0000 2024-03-04T12:17:10.518 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-04 12:11:08.797195319 +0000 2024-03-04T12:17:10.518 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T12:17:10.591 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-04T12:17:10.591 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-04T12:17:10.592 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000271816 s, 1.9 MB/s 2024-03-04T12:17:10.593 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T12:17:10.650 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-03-04T12:17:10.707 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T12:17:10.708 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T12:17:10.708 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-04T12:17:10.708 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:17:10.708 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T12:17:10.708 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-04 12:16:57.511073937 +0000 2024-03-04T12:17:10.708 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-04 12:11:09.168191115 +0000 2024-03-04T12:17:10.708 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-04 12:11:09.168191115 +0000 2024-03-04T12:17:10.708 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-04 12:11:09.168191115 +0000 2024-03-04T12:17:10.708 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T12:17:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-04T12:17:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-04T12:17:10.779 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000229207 s, 2.2 MB/s 2024-03-04T12:17:10.781 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T12:17:10.837 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-03-04T12:17:10.893 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T12:17:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T12:17:10.894 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-04T12:17:10.894 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:17:10.894 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T12:17:10.894 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-04 12:16:57.511073937 +0000 2024-03-04T12:17:10.894 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-04 12:11:09.518187150 +0000 2024-03-04T12:17:10.894 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-04 12:11:09.518187150 +0000 2024-03-04T12:17:10.894 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-04 12:11:09.518187150 +0000 2024-03-04T12:17:10.894 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T12:17:10.964 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-04T12:17:10.964 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-04T12:17:10.964 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000201293 s, 2.5 MB/s 2024-03-04T12:17:10.965 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T12:17:11.022 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-03-04T12:17:11.078 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T12:17:11.078 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T12:17:11.078 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-04T12:17:11.078 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:17:11.079 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T12:17:11.079 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-03-04 12:16:57.512073926 +0000 2024-03-04T12:17:11.079 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-03-04 12:11:09.860183276 +0000 2024-03-04T12:17:11.079 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-03-04 12:11:09.860183276 +0000 2024-03-04T12:17:11.079 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-03-04 12:11:09.860183276 +0000 2024-03-04T12:17:11.079 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T12:17:11.148 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-03-04T12:17:11.149 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-03-04T12:17:11.149 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.00018219 s, 2.8 MB/s 2024-03-04T12:17:11.150 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T12:17:11.206 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-04T12:17:11.206 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T12:17:11.222 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T12:17:11.222 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-03-04T12:17:11.276 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T12:17:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T12:17:11.277 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-04T12:17:11.277 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:17:11.277 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T12:17:11.277 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-04 12:16:47.824782374 +0000 2024-03-04T12:17:11.277 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-04 12:10:41.727846737 +0000 2024-03-04T12:17:11.277 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-04 12:10:41.727846737 +0000 2024-03-04T12:17:11.277 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-04 12:10:41.727846737 +0000 2024-03-04T12:17:11.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T12:17:11.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:11 smithi086 ceph-mon[36851]: Reconfiguring mon.c (monmap changed)... 2024-03-04T12:17:11.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:11 smithi086 ceph-mon[36851]: Reconfiguring daemon mon.c on smithi086 2024-03-04T12:17:11.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:11 smithi086 ceph-mon[36851]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:11.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:11.340 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:11.343 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-04T12:17:11.343 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-04T12:17:11.343 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000290119 s, 1.8 MB/s 2024-03-04T12:17:11.344 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T12:17:11.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:11 smithi063 ceph-mon[36717]: Reconfiguring mon.c (monmap changed)... 2024-03-04T12:17:11.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:11 smithi063 ceph-mon[36717]: Reconfiguring daemon mon.c on smithi086 2024-03-04T12:17:11.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:11 smithi063 ceph-mon[36717]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:11.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:11.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:11.400 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-03-04T12:17:11.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:11 smithi060 ceph-mon[32745]: Reconfiguring mon.c (monmap changed)... 2024-03-04T12:17:11.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:11 smithi060 ceph-mon[32745]: Reconfiguring daemon mon.c on smithi086 2024-03-04T12:17:11.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:11 smithi060 ceph-mon[32745]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:11.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:11.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:11.454 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T12:17:11.454 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T12:17:11.454 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-04T12:17:11.454 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:17:11.454 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T12:17:11.454 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-04 12:16:47.824782374 +0000 2024-03-04T12:17:11.454 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-04 12:10:42.054839408 +0000 2024-03-04T12:17:11.455 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-04 12:10:42.055839386 +0000 2024-03-04T12:17:11.455 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-04 12:10:42.054839408 +0000 2024-03-04T12:17:11.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T12:17:11.520 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-04T12:17:11.520 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-04T12:17:11.520 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000207666 s, 2.5 MB/s 2024-03-04T12:17:11.521 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T12:17:11.577 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-03-04T12:17:11.632 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T12:17:11.633 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T12:17:11.633 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-04T12:17:11.633 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:17:11.633 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T12:17:11.633 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-04 12:16:47.824782374 +0000 2024-03-04T12:17:11.633 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-04 12:10:42.395831766 +0000 2024-03-04T12:17:11.633 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-04 12:10:42.395831766 +0000 2024-03-04T12:17:11.633 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-04 12:10:42.395831766 +0000 2024-03-04T12:17:11.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T12:17:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-04T12:17:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-04T12:17:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000206215 s, 2.5 MB/s 2024-03-04T12:17:11.699 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T12:17:11.754 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-03-04T12:17:11.810 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T12:17:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T12:17:11.811 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-04T12:17:11.811 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:17:11.811 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T12:17:11.811 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-04 12:16:47.825782353 +0000 2024-03-04T12:17:11.811 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-04 12:10:42.735824146 +0000 2024-03-04T12:17:11.811 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-04 12:10:42.735824146 +0000 2024-03-04T12:17:11.811 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-04 12:10:42.735824146 +0000 2024-03-04T12:17:11.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T12:17:11.878 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-04T12:17:11.878 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-04T12:17:11.878 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000204893 s, 2.5 MB/s 2024-03-04T12:17:11.879 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T12:17:11.935 INFO:tasks.cephadm:Deploying osd.0 on smithi060 with /dev/vg_nvme/lv_4... 2024-03-04T12:17:11.936 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-04T12:17:12.173 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:17:12.294 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:12 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:12.294 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:12 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:12.295 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:12 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:12.295 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:12 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:12.295 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:12 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T12:17:12.295 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:12 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T12:17:12.295 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:12 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T12:17:12.295 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:12 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:12.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:12 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:12.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:12 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:12.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:12 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:12.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:12 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:12.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:12 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T12:17:12.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:12 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T12:17:12.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:12 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T12:17:12.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:12 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:12.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:12 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:12.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:12 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:12.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:12 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:12.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:12 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:12.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:12 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T12:17:12.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:12 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T12:17:12.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:12 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T12:17:12.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:12 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:13.282 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:13 smithi086 ceph-mon[36851]: Deploying daemon mgr.z on smithi086 2024-03-04T12:17:13.282 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:13 smithi086 ceph-mon[36851]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:13.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:13 smithi063 ceph-mon[36717]: Deploying daemon mgr.z on smithi086 2024-03-04T12:17:13.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:13 smithi063 ceph-mon[36717]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:13.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:13 smithi060 ceph-mon[32745]: Deploying daemon mgr.z on smithi086 2024-03-04T12:17:13.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:13 smithi060 ceph-mon[32745]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:13.596 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:13 smithi086 systemd[1]: Starting Ceph mgr.z for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:17:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-04T12:17:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-04T12:17:13.900 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10+0 records in 2024-03-04T12:17:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:10+0 records out 2024-03-04T12:17:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00834093 s, 1.3 GB/s 2024-03-04T12:17:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping successful for: 2024-03-04T12:17:13.954 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:13 smithi086 podman[38691]: 2024-03-04 12:17:13.569182932 +0000 UTC m=+0.017738013 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:17:13.955 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:13 smithi086 podman[38691]: 2024-03-04 12:17:13.698912356 +0000 UTC m=+0.147467424 container create d4c9ad43a3c10f148e1e9ebb0e29e8d6a84fe07c6e5985c8068a55c2354d80c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z, ceph=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-04T12:17:13.955 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:13 smithi086 podman[38691]: 2024-03-04 12:17:13.884571958 +0000 UTC m=+0.333127026 container init d4c9ad43a3c10f148e1e9ebb0e29e8d6a84fe07c6e5985c8068a55c2354d80c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-03-04T12:17:13.955 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:13 smithi086 podman[38691]: 2024-03-04 12:17:13.887221951 +0000 UTC m=+0.335777019 container start d4c9ad43a3c10f148e1e9ebb0e29e8d6a84fe07c6e5985c8068a55c2354d80c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:17:13.955 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:13 smithi086 ceph-mgr[38710]: -- 172.21.15.86:0/3202913647 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c36b55e5a0 con 0x55c36b533000 2024-03-04T12:17:13.955 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:13 smithi086 bash[38691]: d4c9ad43a3c10f148e1e9ebb0e29e8d6a84fe07c6e5985c8068a55c2354d80c3 2024-03-04T12:17:14.346 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:13 smithi086 systemd[1]: Started Ceph mgr.z for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:17:14.455 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi060:vg_nvme/lv_4 2024-03-04T12:17:14.710 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:17:14.846 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:14 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:14.428+0000 7fd5b64de200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T12:17:14.846 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:14 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:14.521+0000 7fd5b64de200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T12:17:15.106 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:15.106 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:15.106 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:15.106 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T12:17:15.107 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T12:17:15.107 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T12:17:15.107 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:15.107 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:15 smithi060 ceph-mon[32745]: Deploying daemon mgr.y on smithi063 2024-03-04T12:17:15.107 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:15 smithi060 ceph-mon[32745]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:15.287 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:15.288 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:15.288 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:15.288 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T12:17:15.288 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T12:17:15.288 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T12:17:15.288 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:15.288 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:15 smithi063 ceph-mon[36717]: Deploying daemon mgr.y on smithi063 2024-03-04T12:17:15.289 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:15 smithi063 ceph-mon[36717]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:15.296 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:15.162+0000 7fd5b64de200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T12:17:15.296 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:15.257+0000 7fd5b64de200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T12:17:15.296 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:15.297 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:15.297 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:15.297 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T12:17:15.297 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T12:17:15.297 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T12:17:15.297 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:15.297 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-mon[36851]: Deploying daemon mgr.y on smithi063 2024-03-04T12:17:15.297 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-mon[36851]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:15.596 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:15.437+0000 7fd5b64de200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T12:17:15.596 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:15.538+0000 7fd5b64de200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T12:17:15.915 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:15.624+0000 7fd5b64de200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T12:17:16.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 -- 172.21.15.60:0/4269052276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5610101e50 msgr2=0x7f5610102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:16.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 --2- 172.21.15.60:0/4269052276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5610101e50 0x7f5610102270 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f560c009990 tx=0x7f560c009ca0 comp rx=0 tx=0).stop 2024-03-04T12:17:16.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 -- 172.21.15.60:0/4269052276 shutdown_connections 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 --2- 172.21.15.60:0/4269052276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5610103970 0x7f561010a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 --2- 172.21.15.60:0/4269052276 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5610101e50 0x7f5610102270 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 --2- 172.21.15.60:0/4269052276 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5610102fb0 0x7f5610103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 -- 172.21.15.60:0/4269052276 >> 172.21.15.60:0/4269052276 conn(0x7f56100fd430 msgr2=0x7f56100ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 -- 172.21.15.60:0/4269052276 shutdown_connections 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 -- 172.21.15.60:0/4269052276 wait complete. 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 Processor -- start 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 -- start start 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5610101e50 0x7f5610197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.047+0000 7f56173d8700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5610102fb0 0x7f5610198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f56173d8700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5610103970 0x7f561019f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f56173d8700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561010da40 con 0x7f5610103970 2024-03-04T12:17:16.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f56173d8700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561010d8c0 con 0x7f5610102fb0 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f56173d8700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561010dbc0 con 0x7f5610101e50 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f5615174700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5610101e50 0x7f5610197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f5615975700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5610103970 0x7f561019f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f5615174700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5610101e50 0x7f5610197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:53148/0 (socket says 172.21.15.60:53148) 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f5614973700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5610102fb0 0x7f5610198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f5615174700 1 -- 172.21.15.60:0/826211617 learned_addr learned my addr 172.21.15.60:0/826211617 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f5615975700 1 -- 172.21.15.60:0/826211617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5610101e50 msgr2=0x7f5610197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f5615975700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5610101e50 0x7f5610197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f5615975700 1 -- 172.21.15.60:0/826211617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5610102fb0 msgr2=0x7f5610198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f5615975700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5610102fb0 0x7f5610198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f5615975700 1 -- 172.21.15.60:0/826211617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561019fc50 con 0x7f5610103970 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f5615174700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5610101e50 0x7f5610197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f5615975700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5610103970 0x7f561019f5b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5608009f60 tx=0x7f5608009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f56067fc700 1 -- 172.21.15.60:0/826211617 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56080092e0 con 0x7f5610103970 2024-03-04T12:17:16.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f56067fc700 1 -- 172.21.15.60:0/826211617 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5608004bb0 con 0x7f5610103970 2024-03-04T12:17:16.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f56173d8700 1 -- 172.21.15.60:0/826211617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561019fee0 con 0x7f5610103970 2024-03-04T12:17:16.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f56067fc700 1 -- 172.21.15.60:0/826211617 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5608004280 con 0x7f5610103970 2024-03-04T12:17:16.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.048+0000 7f56173d8700 1 -- 172.21.15.60:0/826211617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56101a0390 con 0x7f5610103970 2024-03-04T12:17:16.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.049+0000 7f56067fc700 1 -- 172.21.15.60:0/826211617 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 13) v1 ==== 45983+0+0 (secure 0 0 0) 0x7f560801b030 con 0x7f5610103970 2024-03-04T12:17:16.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.049+0000 7f56067fc700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f55fc038e80 0x7f55fc03b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:16.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.049+0000 7f56067fc700 1 -- 172.21.15.60:0/826211617 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f560804b710 con 0x7f5610103970 2024-03-04T12:17:16.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.049+0000 7f5615174700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f55fc038e80 0x7f55fc03b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:16.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.049+0000 7f56173d8700 1 -- 172.21.15.60:0/826211617 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f561002d870 con 0x7f5610103970 2024-03-04T12:17:16.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.050+0000 7f5615174700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f55fc038e80 0x7f55fc03b340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f560c00b2e0 tx=0x7f560c00b840 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:17:16.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.055+0000 7f56067fc700 1 -- 172.21.15.60:0/826211617 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5608022020 con 0x7f5610103970 2024-03-04T12:17:16.211 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:16 smithi086 ceph-mon[36851]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:16.212 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:15.910+0000 7fd5b64de200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T12:17:16.212 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:16 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:16.022+0000 7fd5b64de200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T12:17:16.212 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:16 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:16.111+0000 7fd5b64de200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T12:17:16.212 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:16 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:16.207+0000 7fd5b64de200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T12:17:16.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:16.270+0000 7f56173d8700 1 -- 172.21.15.60:0/826211617 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5610062260 con 0x7f55fc038e80 2024-03-04T12:17:16.275 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:16 smithi060 ceph-mon[32745]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:16.285 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:16 smithi063 ceph-mon[36717]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:16.285 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:16 smithi063 systemd[1]: Starting Ceph mgr.y for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:17:16.595 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:16 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:16.386+0000 7fd5b64de200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T12:17:16.599 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:16 smithi063 podman[38084]: 2024-03-04 12:17:16.396104286 +0000 UTC m=+0.121387026 container create 41d77cc52e14d3af7e577bb74731d0c3dbbc6bd2c61e32693da8f5b7c66aa6e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-04T12:17:16.599 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:16 smithi063 podman[38084]: 2024-03-04 12:17:16.300722106 +0000 UTC m=+0.026004860 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:17:17.099 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:16 smithi063 podman[38084]: 2024-03-04 12:17:16.598707396 +0000 UTC m=+0.323990136 container init 41d77cc52e14d3af7e577bb74731d0c3dbbc6bd2c61e32693da8f5b7c66aa6e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-04T12:17:17.099 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:16 smithi063 podman[38084]: 2024-03-04 12:17:16.601323891 +0000 UTC m=+0.326606623 container start 41d77cc52e14d3af7e577bb74731d0c3dbbc6bd2c61e32693da8f5b7c66aa6e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-04T12:17:17.099 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:16 smithi063 ceph-mgr[38103]: -- 172.21.15.63:0/1363539046 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a3ffc585a0 con 0x55a3ffc2d000 2024-03-04T12:17:17.099 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:16 smithi063 bash[38084]: 41d77cc52e14d3af7e577bb74731d0c3dbbc6bd2c61e32693da8f5b7c66aa6e4 2024-03-04T12:17:17.099 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:16 smithi063 systemd[1]: Started Ceph mgr.y for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:17:17.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:17 smithi060 ceph-mon[32745]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:17:17.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:17 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:17:17.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:17 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:17:17.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:17 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:17.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:17 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:17.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:17 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:17.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:17 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:17.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:17 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:17.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:17 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:17.537 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:17.383+0000 7fd5b64de200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T12:17:17.537 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:17.533+0000 7fd5b64de200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T12:17:17.537 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-mon[36851]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:17:17.538 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:17:17.538 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:17:17.538 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:17.538 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:17.538 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:17.538 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:17.538 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:17.538 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:17.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:17 smithi063 ceph-mon[36717]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:17:17.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:17 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:17:17.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:17 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:17:17.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:17 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:17.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:17 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:17.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:17 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:17.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:17 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:17.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:17 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:17.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:17 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:17.600 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:17 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:17.184+0000 7fdfdd95e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T12:17:17.600 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:17 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:17.284+0000 7fdfdd95e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T12:17:18.087 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:17 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:17.986+0000 7fdfdd95e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T12:17:18.087 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:18 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:18.083+0000 7fdfdd95e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T12:17:18.156 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:17.872+0000 7fd5b64de200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T12:17:18.157 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:17.964+0000 7fd5b64de200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T12:17:18.239 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:18 smithi060 ceph-mon[32745]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:18.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:18 smithi063 ceph-mon[36717]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:18.346 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:18 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:18.239+0000 7fdfdd95e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T12:17:18.346 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:18 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:18.342+0000 7fdfdd95e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T12:17:18.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:18 smithi086 ceph-mon[36851]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:18.596 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:18 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:18.161+0000 7fd5b64de200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:17:18.596 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:18 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:18.354+0000 7fd5b64de200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T12:17:18.599 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:18 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:18.429+0000 7fdfdd95e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T12:17:18.899 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:18 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:18.638+0000 7fd5b64de200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:17:18.899 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:18 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:18.895+0000 7fd5b64de200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T12:17:19.009 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:18 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:18.703+0000 7fdfdd95e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T12:17:19.009 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:18 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:18.816+0000 7fdfdd95e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T12:17:19.009 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:18 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:18.907+0000 7fdfdd95e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T12:17:19.009 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:19.005+0000 7fdfdd95e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T12:17:19.200 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:19.131+0000 7fd5b64de200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T12:17:19.349 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:19.188+0000 7fdfdd95e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T12:17:19.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: Reconfiguring mgr.x (unknown last config time)... 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: Reconfiguring daemon mgr.x on smithi060 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2361363963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f"}]: dispatch 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2361363963' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f"}]': finished 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:19.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:19.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:19.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: Reconfiguring mgr.x (unknown last config time)... 2024-03-04T12:17:19.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T12:17:19.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T12:17:19.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:19.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: Reconfiguring daemon mgr.x on smithi060 2024-03-04T12:17:19.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2361363963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f"}]: dispatch 2024-03-04T12:17:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2361363963' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f"}]': finished 2024-03-04T12:17:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T12:17:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: Reconfiguring mgr.x (unknown last config time)... 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: Reconfiguring daemon mgr.x on smithi060 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2361363963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f"}]: dispatch 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2361363963' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f"}]': finished 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:19.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:20.204 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:19 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:19.923+0000 7fd5b64de200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T12:17:20.488 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:20 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3062781782' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:17:20.488 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:20 smithi086 ceph-mon[36851]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:20.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:20 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3062781782' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:17:20.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:20 smithi063 ceph-mon[36717]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:20.599 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:20 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:20.225+0000 7fdfdd95e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T12:17:20.599 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:20 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:20.361+0000 7fdfdd95e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T12:17:20.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:20 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3062781782' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:17:20.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:20 smithi060 ceph-mon[32745]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:20.796 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:20.484+0000 7fd5b64de200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T12:17:20.796 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:20.581+0000 7fd5b64de200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T12:17:20.796 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:17:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:17:20.792+0000 7fd5b64de200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T12:17:21.009 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:20 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:20.707+0000 7fdfdd95e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T12:17:21.010 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:20 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:20.802+0000 7fdfdd95e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T12:17:21.010 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:21 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:21.005+0000 7fdfdd95e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:17:21.215 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:21.210+0000 7f56067fc700 1 -- 172.21.15.60:0/826211617 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 14) v1 ==== 91340+0+0 (secure 0 0 0) 0x7f5608012050 con 0x7f5610103970 2024-03-04T12:17:21.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:21 smithi063 ceph-mon[36717]: Standby manager daemon z started 2024-03-04T12:17:21.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:21 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.86:0/4160723705' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T12:17:21.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:21 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.86:0/4160723705' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T12:17:21.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:21 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.86:0/4160723705' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T12:17:21.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:21 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.86:0/4160723705' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T12:17:21.349 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:21 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:21.196+0000 7fdfdd95e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T12:17:21.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:21 smithi086 ceph-mon[36851]: Standby manager daemon z started 2024-03-04T12:17:21.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:21 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.86:0/4160723705' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T12:17:21.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:21 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.86:0/4160723705' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T12:17:21.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:21 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.86:0/4160723705' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T12:17:21.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:21 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.86:0/4160723705' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T12:17:21.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:21 smithi060 ceph-mon[32745]: Standby manager daemon z started 2024-03-04T12:17:21.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:21 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.86:0/4160723705' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T12:17:21.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:21 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.86:0/4160723705' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T12:17:21.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:21 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.86:0/4160723705' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T12:17:21.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:21 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.86:0/4160723705' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T12:17:21.763 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:21 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:21.489+0000 7fdfdd95e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:17:21.763 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:21 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:21.759+0000 7fdfdd95e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T12:17:22.099 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:22 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:22.006+0000 7fdfdd95e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T12:17:22.559 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:22 smithi060 ceph-mon[32745]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-04T12:17:22.559 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:22 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T12:17:22.559 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:22 smithi060 ceph-mon[32745]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:22.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:22 smithi086 ceph-mon[36851]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-04T12:17:22.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:22 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T12:17:22.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:22 smithi086 ceph-mon[36851]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:22.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:22 smithi063 ceph-mon[36717]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-04T12:17:22.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:22 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T12:17:22.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:22 smithi063 ceph-mon[36717]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:23.099 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:22 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:22.843+0000 7fdfdd95e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T12:17:23.769 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:23.439+0000 7fdfdd95e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T12:17:23.769 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:23.540+0000 7fdfdd95e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T12:17:23.769 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:17:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:17:23.765+0000 7fdfdd95e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T12:17:23.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:23.835+0000 7f56067fc700 1 -- 172.21.15.60:0/826211617 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 15) v1 ==== 136697+0+0 (secure 0 0 0) 0x7f5608007500 con 0x7f5610103970 2024-03-04T12:17:24.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:23 smithi086 ceph-mon[36851]: Standby manager daemon y started 2024-03-04T12:17:24.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:23 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.63:0/1384496245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T12:17:24.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:23 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.63:0/1384496245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T12:17:24.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:23 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.63:0/1384496245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T12:17:24.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:23 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.63:0/1384496245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T12:17:24.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:23 smithi063 ceph-mon[36717]: Standby manager daemon y started 2024-03-04T12:17:24.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:23 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.63:0/1384496245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T12:17:24.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:23 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.63:0/1384496245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T12:17:24.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:23 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.63:0/1384496245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T12:17:24.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:23 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.63:0/1384496245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T12:17:24.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:23 smithi060 ceph-mon[32745]: Standby manager daemon y started 2024-03-04T12:17:24.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:23 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.63:0/1384496245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T12:17:24.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:23 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.63:0/1384496245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T12:17:24.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:23 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.63:0/1384496245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T12:17:24.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:23 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.63:0/1384496245' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T12:17:25.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:24 smithi060 ceph-mon[32745]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-04T12:17:25.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:24 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T12:17:25.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:24 smithi060 ceph-mon[32745]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:25.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:24 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:25.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:24 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:25.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:24 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:25.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:24 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:25.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:24 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:25.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:24 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:25.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:24 smithi086 ceph-mon[36851]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-04T12:17:25.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:24 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T12:17:25.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:24 smithi086 ceph-mon[36851]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:25.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:24 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:25.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:24 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:25.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:24 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:25.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:24 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:25.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:24 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:25.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:24 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:24 smithi063 ceph-mon[36717]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-04T12:17:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:24 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T12:17:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:24 smithi063 ceph-mon[36717]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:24 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:24 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:24 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:24 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:24 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:24 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:26.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:26 smithi060 ceph-mon[32745]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:26.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:26 smithi086 ceph-mon[36851]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:26.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:26 smithi063 ceph-mon[36717]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:27.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T12:17:27.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:27.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:27 smithi060 ceph-mon[32745]: Deploying daemon osd.0 on smithi060 2024-03-04T12:17:27.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T12:17:27.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:27.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:27 smithi086 ceph-mon[36851]: Deploying daemon osd.0 on smithi060 2024-03-04T12:17:27.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T12:17:27.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:27.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:27 smithi063 ceph-mon[36717]: Deploying daemon osd.0 on smithi060 2024-03-04T12:17:28.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:28 smithi060 ceph-mon[32745]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:28.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:28 smithi086 ceph-mon[36851]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:28.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:28 smithi063 ceph-mon[36717]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:30.352 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:30 smithi060 ceph-mon[32745]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:30.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:30 smithi086 ceph-mon[36851]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:30.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:30 smithi063 ceph-mon[36717]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:33.315 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:33.315 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:33.315 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:33.315 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:32 smithi060 ceph-mon[32745]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:33.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:33.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:33.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:33.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:32 smithi086 ceph-mon[36851]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:33.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:33.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:33.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:33.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:32 smithi063 ceph-mon[36717]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:34.415 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:34 smithi060 ceph-mon[32745]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:34.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:34 smithi086 ceph-mon[36851]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:34.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:34 smithi063 ceph-mon[36717]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:35.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.726+0000 7f56067fc700 1 -- 172.21.15.60:0/826211617 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5610062260 con 0x7f55fc038e80 2024-03-04T12:17:35.730 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 0 on host 'smithi060' 2024-03-04T12:17:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.728+0000 7f56173d8700 1 -- 172.21.15.60:0/826211617 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f55fc038e80 msgr2=0x7f55fc03b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.728+0000 7f56173d8700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f55fc038e80 0x7f55fc03b340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f560c00b2e0 tx=0x7f560c00b840 comp rx=0 tx=0).stop 2024-03-04T12:17:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.728+0000 7f56173d8700 1 -- 172.21.15.60:0/826211617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5610103970 msgr2=0x7f561019f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.728+0000 7f56173d8700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5610103970 0x7f561019f5b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5608009f60 tx=0x7f5608009f90 comp rx=0 tx=0).stop 2024-03-04T12:17:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.728+0000 7f56173d8700 1 -- 172.21.15.60:0/826211617 shutdown_connections 2024-03-04T12:17:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.729+0000 7f56173d8700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f55fc038e80 0x7f55fc03b340 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.729+0000 7f56173d8700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5610103970 0x7f561019f5b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.729+0000 7f56173d8700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5610101e50 0x7f5610197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.729+0000 7f56173d8700 1 --2- 172.21.15.60:0/826211617 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5610102fb0 0x7f5610198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.729+0000 7f56173d8700 1 -- 172.21.15.60:0/826211617 >> 172.21.15.60:0/826211617 conn(0x7f56100fd430 msgr2=0x7f56101062e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:17:35.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.729+0000 7f56173d8700 1 -- 172.21.15.60:0/826211617 shutdown_connections 2024-03-04T12:17:35.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:17:35.729+0000 7f56173d8700 1 -- 172.21.15.60:0/826211617 wait complete. 2024-03-04T12:17:35.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:35.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:35.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:35.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:35.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:35.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:35.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:35.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:36.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:36.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:36.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:36.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:36.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:36.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:36.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:36.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:36.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:36.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:36.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:36.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:36.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:36.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:36.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:36.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:36.372 DEBUG:teuthology.orchestra.run.smithi060:osd.0> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.0.service 2024-03-04T12:17:36.375 INFO:tasks.cephadm:Deploying osd.1 on smithi063 with /dev/vg_nvme/lv_4... 2024-03-04T12:17:36.375 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-04T12:17:36.632 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T12:17:36.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:36 smithi060 ceph-mon[32745]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T12:17:36.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:36 smithi060 ceph-mon[32745]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:37.058 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:36 smithi063 ceph-mon[36717]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T12:17:37.058 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:36 smithi063 ceph-mon[36717]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:37.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:36 smithi086 ceph-mon[36851]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T12:17:37.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:36 smithi086 ceph-mon[36851]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:38.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:37 smithi086 ceph-mon[36851]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T12:17:38.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:37 smithi086 ceph-mon[36851]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T12:17:38.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:37 smithi086 ceph-mon[36851]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-03-04T12:17:38.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:37 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:38.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:37 smithi063 ceph-mon[36717]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T12:17:38.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:37 smithi063 ceph-mon[36717]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T12:17:38.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:37 smithi063 ceph-mon[36717]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-03-04T12:17:38.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:37 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:38.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:37 smithi060 ceph-mon[32745]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T12:17:38.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:37 smithi060 ceph-mon[32745]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T12:17:38.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:37 smithi060 ceph-mon[32745]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-03-04T12:17:38.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:37 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:38.490 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-04T12:17:38.491 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-04T12:17:38.491 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-03-04T12:17:38.491 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-03-04T12:17:38.491 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120604 s, 869 MB/s 2024-03-04T12:17:38.491 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-03-04T12:17:38.677 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:17:38 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0[47348]: 2024-03-04T12:17:38.275+0000 7f0cd1701700 -1 osd.0 0 waiting for initial osdmap 2024-03-04T12:17:38.677 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:17:38 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0[47348]: 2024-03-04T12:17:38.279+0000 7f0cc94ec700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T12:17:38.994 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:38 smithi060 ceph-mon[32745]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-03-04T12:17:38.994 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:38 smithi060 ceph-mon[32745]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T12:17:38.994 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:38 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:38.995 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:38 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:38.995 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:38 smithi060 ceph-mon[32745]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:38.995 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:38 smithi060 ceph-mon[32745]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' 2024-03-04T12:17:38.995 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:38 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:39.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:38 smithi086 ceph-mon[36851]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-03-04T12:17:39.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:38 smithi086 ceph-mon[36851]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T12:17:39.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:38 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:39.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:38 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:39.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:38 smithi086 ceph-mon[36851]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:39.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:38 smithi086 ceph-mon[36851]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' 2024-03-04T12:17:39.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:38 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:39.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:38 smithi063 ceph-mon[36717]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-03-04T12:17:39.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:38 smithi063 ceph-mon[36717]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T12:17:39.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:38 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:39.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:38 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:39.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:38 smithi063 ceph-mon[36717]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T12:17:39.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:38 smithi063 ceph-mon[36717]: from='osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647]' entity='osd.0' 2024-03-04T12:17:39.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:38 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:39.237 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi063:vg_nvme/lv_4 2024-03-04T12:17:39.490 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T12:17:39.887 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:39 smithi063 ceph-mon[36717]: purged_snaps scrub starts 2024-03-04T12:17:39.888 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:39 smithi063 ceph-mon[36717]: purged_snaps scrub ok 2024-03-04T12:17:39.888 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:39 smithi063 ceph-mon[36717]: osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647] boot 2024-03-04T12:17:39.888 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:39 smithi063 ceph-mon[36717]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T12:17:39.888 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:39.888 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:39.888 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:39.888 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:17:39.888 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:39.888 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:39.888 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:39.888 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:40.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:39 smithi086 ceph-mon[36851]: purged_snaps scrub starts 2024-03-04T12:17:40.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:39 smithi086 ceph-mon[36851]: purged_snaps scrub ok 2024-03-04T12:17:40.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:39 smithi086 ceph-mon[36851]: osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647] boot 2024-03-04T12:17:40.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:39 smithi086 ceph-mon[36851]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T12:17:40.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:40.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:40.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:40.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:17:40.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:40.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:40.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:40.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:40.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:39 smithi060 ceph-mon[32745]: purged_snaps scrub starts 2024-03-04T12:17:40.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:39 smithi060 ceph-mon[32745]: purged_snaps scrub ok 2024-03-04T12:17:40.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:39 smithi060 ceph-mon[32745]: osd.0 [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647] boot 2024-03-04T12:17:40.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:39 smithi060 ceph-mon[32745]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T12:17:40.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:17:40.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:40.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:40.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:17:40.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:40.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:40.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:17:40.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:40.953 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:40 smithi063 ceph-mon[36717]: Detected new or changed devices on smithi060 2024-03-04T12:17:40.953 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:40 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi060 to 17093M 2024-03-04T12:17:40.953 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:40 smithi063 ceph-mon[36717]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:40.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.981+0000 7fb472f07700 1 -- 172.21.15.63:0/4022480376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb46c100dc0 msgr2=0x7fb46c1011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:40.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.981+0000 7fb472f07700 1 --2- 172.21.15.63:0/4022480376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb46c100dc0 0x7fb46c1011e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb4600099c0 tx=0x7fb460009cd0 comp rx=0 tx=0).stop 2024-03-04T12:17:40.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.981+0000 7fb472f07700 1 -- 172.21.15.63:0/4022480376 shutdown_connections 2024-03-04T12:17:40.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.981+0000 7fb472f07700 1 --2- 172.21.15.63:0/4022480376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb46c102960 0x7fb46c109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:40.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.981+0000 7fb472f07700 1 --2- 172.21.15.63:0/4022480376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb46c100dc0 0x7fb46c1011e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:40.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.981+0000 7fb472f07700 1 --2- 172.21.15.63:0/4022480376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb46c101fc0 0x7fb46c102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:40.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.981+0000 7fb472f07700 1 -- 172.21.15.63:0/4022480376 >> 172.21.15.63:0/4022480376 conn(0x7fb46c0fc380 msgr2=0x7fb46c0fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:17:40.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.981+0000 7fb472f07700 1 -- 172.21.15.63:0/4022480376 shutdown_connections 2024-03-04T12:17:40.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.981+0000 7fb472f07700 1 -- 172.21.15.63:0/4022480376 wait complete. 2024-03-04T12:17:40.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.982+0000 7fb472f07700 1 Processor -- start 2024-03-04T12:17:40.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.982+0000 7fb472f07700 1 -- start start 2024-03-04T12:17:40.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.982+0000 7fb472f07700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb46c100dc0 0x7fb46c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:40.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.982+0000 7fb472f07700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb46c101fc0 0x7fb46c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:40.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.982+0000 7fb472f07700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb46c102960 0x7fb46c19f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:40.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.982+0000 7fb472f07700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c10c990 con 0x7fb46c102960 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.982+0000 7fb472f07700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c10c810 con 0x7fb46c100dc0 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.982+0000 7fb472f07700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c10cb10 con 0x7fb46c101fc0 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.982+0000 7fb470ca3700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb46c100dc0 0x7fb46c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.982+0000 7fb470ca3700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb46c100dc0 0x7fb46c198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39218/0 (socket says 172.21.15.63:39218) 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.982+0000 7fb470ca3700 1 -- 172.21.15.63:0/1991179309 learned_addr learned my addr 172.21.15.63:0/1991179309 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.982+0000 7fb4714a4700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb46c102960 0x7fb46c19f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.983+0000 7fb46bfff700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb46c101fc0 0x7fb46c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.983+0000 7fb470ca3700 1 -- 172.21.15.63:0/1991179309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb46c101fc0 msgr2=0x7fb46c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.983+0000 7fb470ca3700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb46c101fc0 0x7fb46c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.983+0000 7fb470ca3700 1 -- 172.21.15.63:0/1991179309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb46c102960 msgr2=0x7fb46c19f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.983+0000 7fb470ca3700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb46c102960 0x7fb46c19f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.983+0000 7fb470ca3700 1 -- 172.21.15.63:0/1991179309 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c19fcb0 con 0x7fb46c100dc0 2024-03-04T12:17:40.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.983+0000 7fb470ca3700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb46c100dc0 0x7fb46c198050 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb46000c010 tx=0x7fb460005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:17:40.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.983+0000 7fb469ffb700 1 -- 172.21.15.63:0/1991179309 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460027070 con 0x7fb46c100dc0 2024-03-04T12:17:40.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.983+0000 7fb469ffb700 1 -- 172.21.15.63:0/1991179309 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4600041d0 con 0x7fb46c100dc0 2024-03-04T12:17:40.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.983+0000 7fb469ffb700 1 -- 172.21.15.63:0/1991179309 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46000f460 con 0x7fb46c100dc0 2024-03-04T12:17:40.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.983+0000 7fb472f07700 1 -- 172.21.15.63:0/1991179309 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46c19ff40 con 0x7fb46c100dc0 2024-03-04T12:17:40.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.983+0000 7fb472f07700 1 -- 172.21.15.63:0/1991179309 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46c1a0360 con 0x7fb46c100dc0 2024-03-04T12:17:40.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.984+0000 7fb472f07700 1 -- 172.21.15.63:0/1991179309 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46c1a0650 con 0x7fb46c100dc0 2024-03-04T12:17:40.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.985+0000 7fb469ffb700 1 -- 172.21.15.63:0/1991179309 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 15) v1 ==== 136697+0+0 (secure 0 0 0) 0x7fb46002a070 con 0x7fb46c100dc0 2024-03-04T12:17:40.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.986+0000 7fb469ffb700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb45c0a1d80 0x7fb45c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:17:40.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.986+0000 7fb469ffb700 1 -- 172.21.15.63:0/1991179309 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fb4600cda20 con 0x7fb46c100dc0 2024-03-04T12:17:40.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.986+0000 7fb46bfff700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb45c0a1d80 0x7fb45c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:17:40.991 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.987+0000 7fb46bfff700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb45c0a1d80 0x7fb45c0a4240 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb458005ea0 tx=0x7fb458005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:17:40.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:40.990+0000 7fb469ffb700 1 -- 172.21.15.63:0/1991179309 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb460031070 con 0x7fb46c100dc0 2024-03-04T12:17:41.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:40 smithi086 ceph-mon[36851]: Detected new or changed devices on smithi060 2024-03-04T12:17:41.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:40 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi060 to 17093M 2024-03-04T12:17:41.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:40 smithi086 ceph-mon[36851]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:41.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:40 smithi060 ceph-mon[32745]: Detected new or changed devices on smithi060 2024-03-04T12:17:41.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:40 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi060 to 17093M 2024-03-04T12:17:41.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:40 smithi060 ceph-mon[32745]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:41.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:41.204+0000 7fb472f07700 1 -- 172.21.15.63:0/1991179309 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb46c1a0820 con 0x7fb45c0a1d80 2024-03-04T12:17:42.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:41 smithi086 ceph-mon[36851]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T12:17:42.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:41 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:17:42.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:41 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:17:42.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:41 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:42.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:41 smithi063 ceph-mon[36717]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T12:17:42.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:41 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:17:42.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:41 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:17:42.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:41 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:42.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:41 smithi060 ceph-mon[32745]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T12:17:42.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:41 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:17:42.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:41 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:17:42.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:41 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:42.920 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:42 smithi063 ceph-mon[36717]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:17:42.920 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:42 smithi063 ceph-mon[36717]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:43.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:42 smithi086 ceph-mon[36851]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:17:43.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:42 smithi086 ceph-mon[36851]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:43.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:42 smithi060 ceph-mon[32745]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:17:43.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:42 smithi060 ceph-mon[32745]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:44.090 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:43 smithi063 ceph-mon[36717]: from='client.? 172.21.15.63:0/2335676874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59fe4466-e274-4543-a2a4-f95588613bf4"}]: dispatch 2024-03-04T12:17:44.090 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:43 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59fe4466-e274-4543-a2a4-f95588613bf4"}]: dispatch 2024-03-04T12:17:44.091 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:43 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59fe4466-e274-4543-a2a4-f95588613bf4"}]': finished 2024-03-04T12:17:44.091 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:43 smithi063 ceph-mon[36717]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T12:17:44.091 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:17:44.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:43 smithi086 ceph-mon[36851]: from='client.? 172.21.15.63:0/2335676874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59fe4466-e274-4543-a2a4-f95588613bf4"}]: dispatch 2024-03-04T12:17:44.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:43 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59fe4466-e274-4543-a2a4-f95588613bf4"}]: dispatch 2024-03-04T12:17:44.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:43 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59fe4466-e274-4543-a2a4-f95588613bf4"}]': finished 2024-03-04T12:17:44.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:43 smithi086 ceph-mon[36851]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T12:17:44.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:17:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:43 smithi060 ceph-mon[32745]: from='client.? 172.21.15.63:0/2335676874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59fe4466-e274-4543-a2a4-f95588613bf4"}]: dispatch 2024-03-04T12:17:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:43 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59fe4466-e274-4543-a2a4-f95588613bf4"}]: dispatch 2024-03-04T12:17:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:43 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59fe4466-e274-4543-a2a4-f95588613bf4"}]': finished 2024-03-04T12:17:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:43 smithi060 ceph-mon[32745]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T12:17:44.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:17:45.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:44 smithi086 ceph-mon[36851]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:45.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:44 smithi086 ceph-mon[36851]: from='client.? 172.21.15.63:0/492481768' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:17:45.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:44 smithi063 ceph-mon[36717]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:45.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:44 smithi063 ceph-mon[36717]: from='client.? 172.21.15.63:0/492481768' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:17:45.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:44 smithi060 ceph-mon[32745]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:45.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:44 smithi060 ceph-mon[32745]: from='client.? 172.21.15.63:0/492481768' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:17:46.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:46 smithi060 ceph-mon[32745]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:46.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:46 smithi086 ceph-mon[36851]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:46.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:46 smithi063 ceph-mon[36717]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:48.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:48 smithi063 ceph-mon[36717]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:48.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:48 smithi060 ceph-mon[32745]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:48.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:48 smithi086 ceph-mon[36851]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:50.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:50 smithi060 ceph-mon[32745]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:50.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:50 smithi086 ceph-mon[36851]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:50.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:50 smithi063 ceph-mon[36717]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:51.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:51 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T12:17:51.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:51 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:51.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:51 smithi060 ceph-mon[32745]: Deploying daemon osd.1 on smithi063 2024-03-04T12:17:51.582 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:51 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T12:17:51.583 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:51 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:51.583 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:51 smithi063 ceph-mon[36717]: Deploying daemon osd.1 on smithi063 2024-03-04T12:17:51.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:51 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T12:17:51.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:51 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:17:51.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:51 smithi086 ceph-mon[36851]: Deploying daemon osd.1 on smithi063 2024-03-04T12:17:52.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:52 smithi060 ceph-mon[32745]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:52.513 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:52 smithi063 ceph-mon[36717]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:52.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:52 smithi086 ceph-mon[36851]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:54.309 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:54 smithi063 ceph-mon[36717]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:54.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:54 smithi060 ceph-mon[32745]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:54.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:54 smithi086 ceph-mon[36851]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:56.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:56 smithi060 ceph-mon[32745]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:56.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:56 smithi086 ceph-mon[36851]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:56.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:56 smithi063 ceph-mon[36717]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:57.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:57 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:57.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:57 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:57.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:57 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:57.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:57 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:57.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:57 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:57.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:57 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:57.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:57 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:57.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:57 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:17:57.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:57 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:17:58.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:17:58 smithi063 ceph-mon[36717]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:58.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:17:58 smithi060 ceph-mon[32745]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:58.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:17:58 smithi086 ceph-mon[36851]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:17:59.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.775+0000 7fb469ffb700 1 -- 172.21.15.63:0/1991179309 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb46c1a0820 con 0x7fb45c0a1d80 2024-03-04T12:17:59.780 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 1 on host 'smithi063' 2024-03-04T12:17:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.777+0000 7fb472f07700 1 -- 172.21.15.63:0/1991179309 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb45c0a1d80 msgr2=0x7fb45c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.777+0000 7fb472f07700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb45c0a1d80 0x7fb45c0a4240 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb458005ea0 tx=0x7fb458005e30 comp rx=0 tx=0).stop 2024-03-04T12:17:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.777+0000 7fb472f07700 1 -- 172.21.15.63:0/1991179309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb46c100dc0 msgr2=0x7fb46c198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:17:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.777+0000 7fb472f07700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb46c100dc0 0x7fb46c198050 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb46000c010 tx=0x7fb460005e80 comp rx=0 tx=0).stop 2024-03-04T12:17:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.778+0000 7fb472f07700 1 -- 172.21.15.63:0/1991179309 shutdown_connections 2024-03-04T12:17:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.778+0000 7fb472f07700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb45c0a1d80 0x7fb45c0a4240 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.778+0000 7fb472f07700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb46c102960 0x7fb46c19f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.778+0000 7fb472f07700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb46c100dc0 0x7fb46c198050 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.778+0000 7fb472f07700 1 --2- 172.21.15.63:0/1991179309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb46c101fc0 0x7fb46c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:17:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.778+0000 7fb472f07700 1 -- 172.21.15.63:0/1991179309 >> 172.21.15.63:0/1991179309 conn(0x7fb46c0fc380 msgr2=0x7fb46c0fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:17:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.778+0000 7fb472f07700 1 -- 172.21.15.63:0/1991179309 shutdown_connections 2024-03-04T12:17:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:17:59.778+0000 7fb472f07700 1 -- 172.21.15.63:0/1991179309 wait complete. 2024-03-04T12:18:00.403 DEBUG:teuthology.orchestra.run.smithi063:osd.1> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.1.service 2024-03-04T12:18:00.405 INFO:tasks.cephadm:Deploying osd.2 on smithi086 with /dev/vg_nvme/lv_4... 2024-03-04T12:18:00.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-04T12:18:00.649 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:18:01.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:01.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:01.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:01.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:01.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:00 smithi086 ceph-mon[36851]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:18:01.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:00 smithi086 ceph-mon[36851]: from='osd.1 [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T12:18:01.034 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:00 smithi086 ceph-mon[36851]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T12:18:01.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:01.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:01.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:01.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:01.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:00 smithi063 ceph-mon[36717]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:18:01.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:00 smithi063 ceph-mon[36717]: from='osd.1 [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T12:18:01.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:00 smithi063 ceph-mon[36717]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T12:18:01.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:01.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:01.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:01.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:01.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:00 smithi060 ceph-mon[32745]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:18:01.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:00 smithi060 ceph-mon[32745]: from='osd.1 [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T12:18:01.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:00 smithi060 ceph-mon[32745]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T12:18:01.973 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:01 smithi086 ceph-mon[36851]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T12:18:01.973 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:01 smithi086 ceph-mon[36851]: from='osd.1 [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:18:01.973 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:01 smithi086 ceph-mon[36851]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T12:18:01.973 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:01 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:01.973 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:01 smithi086 ceph-mon[36851]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:18:02.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:01 smithi063 ceph-mon[36717]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T12:18:02.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:01 smithi063 ceph-mon[36717]: from='osd.1 [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:18:02.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:01 smithi063 ceph-mon[36717]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T12:18:02.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:01 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:02.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:01 smithi063 ceph-mon[36717]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:18:02.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:01 smithi060 ceph-mon[32745]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T12:18:02.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:01 smithi060 ceph-mon[32745]: from='osd.1 [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:18:02.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:01 smithi060 ceph-mon[32745]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T12:18:02.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:01 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:02.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:01 smithi060 ceph-mon[32745]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:18:02.535 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-04T12:18:02.535 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-04T12:18:02.535 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-03-04T12:18:02.535 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-03-04T12:18:02.535 INFO:teuthology.orchestra.run.smithi086.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125522 s, 835 MB/s 2024-03-04T12:18:02.535 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 2024-03-04T12:18:02.535 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-03-04T12:18:02.594 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:18:02 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1[41293]: 2024-03-04T12:18:02.333+0000 7f47f8352700 -1 osd.1 0 waiting for initial osdmap 2024-03-04T12:18:02.594 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:18:02 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1[41293]: 2024-03-04T12:18:02.337+0000 7f47f013d700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T12:18:03.143 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:02 smithi086 ceph-mon[36851]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-04T12:18:03.143 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:02 smithi086 ceph-mon[36851]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T12:18:03.143 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:02 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:03.144 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:02 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:03.144 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:02 smithi086 ceph-mon[36851]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:18:03.144 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:02 smithi086 ceph-mon[36851]: from='osd.1 ' entity='osd.1' 2024-03-04T12:18:03.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_4 2024-03-04T12:18:03.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:02 smithi060 ceph-mon[32745]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-04T12:18:03.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:02 smithi060 ceph-mon[32745]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T12:18:03.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:02 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:03.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:02 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:03.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:02 smithi060 ceph-mon[32745]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:18:03.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:02 smithi060 ceph-mon[32745]: from='osd.1 ' entity='osd.1' 2024-03-04T12:18:03.182 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:02 smithi063 ceph-mon[36717]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-04T12:18:03.182 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:02 smithi063 ceph-mon[36717]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T12:18:03.182 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:02 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:03.182 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:02 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:03.182 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:02 smithi063 ceph-mon[36717]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T12:18:03.182 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:02 smithi063 ceph-mon[36717]: from='osd.1 ' entity='osd.1' 2024-03-04T12:18:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:18:04.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: purged_snaps scrub starts 2024-03-04T12:18:04.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: purged_snaps scrub ok 2024-03-04T12:18:04.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: osd.1 [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350] boot 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:04.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: purged_snaps scrub starts 2024-03-04T12:18:04.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: purged_snaps scrub ok 2024-03-04T12:18:04.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:04.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:18:04.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:04.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:04.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: osd.1 [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350] boot 2024-03-04T12:18:04.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T12:18:04.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:04.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:04.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:04.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:04.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: purged_snaps scrub starts 2024-03-04T12:18:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: purged_snaps scrub ok 2024-03-04T12:18:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:18:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: osd.1 [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350] boot 2024-03-04T12:18:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T12:18:04.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:18:04.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:04.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:04.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:04.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:04.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.883+0000 7fe8aac74700 1 -- 172.21.15.86:0/1251985778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8a4101fc0 msgr2=0x7fe8a4102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:04.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.883+0000 7fe8aac74700 1 --2- 172.21.15.86:0/1251985778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8a4101fc0 0x7fe8a4102420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe890009990 tx=0x7fe890009ca0 comp rx=0 tx=0).stop 2024-03-04T12:18:04.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.883+0000 7fe8aac74700 1 -- 172.21.15.86:0/1251985778 shutdown_connections 2024-03-04T12:18:04.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.883+0000 7fe8aac74700 1 --2- 172.21.15.86:0/1251985778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a4102960 0x7fe8a4109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:04.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.883+0000 7fe8aac74700 1 --2- 172.21.15.86:0/1251985778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8a4100dc0 0x7fe8a41011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:04.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.883+0000 7fe8aac74700 1 --2- 172.21.15.86:0/1251985778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8a4101fc0 0x7fe8a4102420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:04.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.883+0000 7fe8aac74700 1 -- 172.21.15.86:0/1251985778 >> 172.21.15.86:0/1251985778 conn(0x7fe8a40fc380 msgr2=0x7fe8a40fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:18:04.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.883+0000 7fe8aac74700 1 -- 172.21.15.86:0/1251985778 shutdown_connections 2024-03-04T12:18:04.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.883+0000 7fe8aac74700 1 -- 172.21.15.86:0/1251985778 wait complete. 2024-03-04T12:18:04.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8aac74700 1 Processor -- start 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8aac74700 1 -- start start 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8aac74700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a4100dc0 0x7fe8a41980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8aac74700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8a4101fc0 0x7fe8a4198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8aac74700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8a4102960 0x7fe8a419f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8aac74700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a410c9b0 con 0x7fe8a4100dc0 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8aac74700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a410c830 con 0x7fe8a4101fc0 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8aac74700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a410cb30 con 0x7fe8a4102960 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8a9211700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8a4102960 0x7fe8a419f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8a9211700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8a4102960 0x7fe8a419f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37154/0 (socket says 172.21.15.86:37154) 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8a9211700 1 -- 172.21.15.86:0/2891415273 learned_addr learned my addr 172.21.15.86:0/2891415273 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8a8a10700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a4100dc0 0x7fe8a41980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.884+0000 7fe8a3fff700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8a4101fc0 0x7fe8a4198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.885+0000 7fe8a9211700 1 -- 172.21.15.86:0/2891415273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8a4101fc0 msgr2=0x7fe8a4198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:04.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.885+0000 7fe8a9211700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8a4101fc0 0x7fe8a4198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:04.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.885+0000 7fe8a9211700 1 -- 172.21.15.86:0/2891415273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a4100dc0 msgr2=0x7fe8a41980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:04.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.885+0000 7fe8a9211700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a4100dc0 0x7fe8a41980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:04.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.885+0000 7fe8a9211700 1 -- 172.21.15.86:0/2891415273 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a419fd40 con 0x7fe8a4102960 2024-03-04T12:18:04.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.885+0000 7fe8a9211700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8a4102960 0x7fe8a419f6a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe89c00add0 tx=0x7fe89c00c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:18:04.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.885+0000 7fe8a1ffb700 1 -- 172.21.15.86:0/2891415273 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c017020 con 0x7fe8a4102960 2024-03-04T12:18:04.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.885+0000 7fe8aac74700 1 -- 172.21.15.86:0/2891415273 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a419ffd0 con 0x7fe8a4102960 2024-03-04T12:18:04.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.885+0000 7fe8a1ffb700 1 -- 172.21.15.86:0/2891415273 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe89c00ce20 con 0x7fe8a4102960 2024-03-04T12:18:04.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.885+0000 7fe8a1ffb700 1 -- 172.21.15.86:0/2891415273 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89c0053e0 con 0x7fe8a4102960 2024-03-04T12:18:04.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.885+0000 7fe8aac74700 1 -- 172.21.15.86:0/2891415273 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a41a0560 con 0x7fe8a4102960 2024-03-04T12:18:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.886+0000 7fe8aac74700 1 -- 172.21.15.86:0/2891415273 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a4068b10 con 0x7fe8a4102960 2024-03-04T12:18:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.887+0000 7fe8a1ffb700 1 -- 172.21.15.86:0/2891415273 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 15) v1 ==== 136697+0+0 (secure 0 0 0) 0x7fe89c00c940 con 0x7fe8a4102960 2024-03-04T12:18:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.887+0000 7fe8a1ffb700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe8940a1d80 0x7fe8940a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:18:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.888+0000 7fe8a1ffb700 1 -- 172.21.15.86:0/2891415273 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fe89c0cbcb0 con 0x7fe8a4102960 2024-03-04T12:18:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.888+0000 7fe8a8a10700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe8940a1d80 0x7fe8940a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:18:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.888+0000 7fe8a8a10700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe8940a1d80 0x7fe8940a4240 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4101e20 tx=0x7fe898009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:18:04.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:04.892+0000 7fe8a1ffb700 1 -- 172.21.15.86:0/2891415273 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe8a4068b10 con 0x7fe8a4102960 2024-03-04T12:18:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:05.099+0000 7fe8aac74700 1 -- 172.21.15.86:0/2891415273 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8a40fe530 con 0x7fe8940a1d80 2024-03-04T12:18:05.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:04 smithi086 ceph-mon[36851]: Detected new or changed devices on smithi063 2024-03-04T12:18:05.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:04 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi063 to 17080M 2024-03-04T12:18:05.103 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:04 smithi086 ceph-mon[36851]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:05.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:04 smithi060 ceph-mon[32745]: Detected new or changed devices on smithi063 2024-03-04T12:18:05.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:04 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi063 to 17080M 2024-03-04T12:18:05.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:04 smithi060 ceph-mon[32745]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:05.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:04 smithi063 ceph-mon[36717]: Detected new or changed devices on smithi063 2024-03-04T12:18:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:04 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi063 to 17080M 2024-03-04T12:18:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:04 smithi063 ceph-mon[36717]: pgmap v63: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:05.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:05 smithi086 ceph-mon[36851]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T12:18:05.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:05 smithi086 ceph-mon[36851]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:18:05.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:18:05.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:18:05.996 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:06.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:05 smithi060 ceph-mon[32745]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T12:18:06.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:05 smithi060 ceph-mon[32745]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:18:06.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:18:06.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:18:06.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:06.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:05 smithi063 ceph-mon[36717]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T12:18:06.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:05 smithi063 ceph-mon[36717]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:18:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:18:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:18:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:07.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:06 smithi060 ceph-mon[32745]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:07.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:06 smithi086 ceph-mon[36851]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:07.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:06 smithi063 ceph-mon[36717]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:08.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:07 smithi060 ceph-mon[32745]: from='client.? 172.21.15.86:0/358949713' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d05a35c6-2586-47fa-8172-6597e7925861"}]: dispatch 2024-03-04T12:18:08.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:07 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d05a35c6-2586-47fa-8172-6597e7925861"}]: dispatch 2024-03-04T12:18:08.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:07 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d05a35c6-2586-47fa-8172-6597e7925861"}]': finished 2024-03-04T12:18:08.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:07 smithi060 ceph-mon[32745]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T12:18:08.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:07 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:08.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:07 smithi086 ceph-mon[36851]: from='client.? 172.21.15.86:0/358949713' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d05a35c6-2586-47fa-8172-6597e7925861"}]: dispatch 2024-03-04T12:18:08.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:07 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d05a35c6-2586-47fa-8172-6597e7925861"}]: dispatch 2024-03-04T12:18:08.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:07 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d05a35c6-2586-47fa-8172-6597e7925861"}]': finished 2024-03-04T12:18:08.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:07 smithi086 ceph-mon[36851]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T12:18:08.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:07 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:08.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:07 smithi063 ceph-mon[36717]: from='client.? 172.21.15.86:0/358949713' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d05a35c6-2586-47fa-8172-6597e7925861"}]: dispatch 2024-03-04T12:18:08.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:07 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d05a35c6-2586-47fa-8172-6597e7925861"}]: dispatch 2024-03-04T12:18:08.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:07 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d05a35c6-2586-47fa-8172-6597e7925861"}]': finished 2024-03-04T12:18:08.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:07 smithi063 ceph-mon[36717]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T12:18:08.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:07 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:09.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:08 smithi060 ceph-mon[32745]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:09.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:08 smithi060 ceph-mon[32745]: from='client.? 172.21.15.86:0/3255900403' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:18:09.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:08 smithi086 ceph-mon[36851]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:09.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:08 smithi086 ceph-mon[36851]: from='client.? 172.21.15.86:0/3255900403' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:18:09.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:08 smithi063 ceph-mon[36717]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:09.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:08 smithi063 ceph-mon[36717]: from='client.? 172.21.15.86:0/3255900403' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:18:10.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:10 smithi060 ceph-mon[32745]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:10.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:10 smithi086 ceph-mon[36851]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:10.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:10 smithi063 ceph-mon[36717]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:12.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:12 smithi086 ceph-mon[36851]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:12.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:12 smithi060 ceph-mon[32745]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:12.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:12 smithi063 ceph-mon[36717]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:14.327 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:14 smithi086 ceph-mon[36851]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:14.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:14 smithi060 ceph-mon[32745]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:14.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:14 smithi063 ceph-mon[36717]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:15.413 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T12:18:15.413 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:15.413 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:15 smithi086 ceph-mon[36851]: Deploying daemon osd.2 on smithi086 2024-03-04T12:18:15.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T12:18:15.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:15.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:15 smithi060 ceph-mon[32745]: Deploying daemon osd.2 on smithi086 2024-03-04T12:18:15.501 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T12:18:15.501 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:15.501 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:15 smithi063 ceph-mon[36717]: Deploying daemon osd.2 on smithi086 2024-03-04T12:18:16.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:16 smithi060 ceph-mon[32745]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:16.427 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:16 smithi086 ceph-mon[36851]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:16.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:16 smithi063 ceph-mon[36717]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:18.293 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:18 smithi086 ceph-mon[36851]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:18.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:18 smithi060 ceph-mon[32745]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:18.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:18 smithi063 ceph-mon[36717]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:21.100 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:20 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:21.100 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:20 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:21.100 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:20 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:21.100 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:20 smithi086 ceph-mon[36851]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:21.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:20 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:21.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:20 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:21.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:20 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:21.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:20 smithi063 ceph-mon[36717]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:21.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:21.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:21.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:21.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:20 smithi060 ceph-mon[32745]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:22.270 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:22 smithi086 ceph-mon[36851]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:22.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:22 smithi060 ceph-mon[32745]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:22.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:22 smithi063 ceph-mon[36717]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:23.269 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 2 on host 'smithi086' 2024-03-04T12:18:23.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.264+0000 7fe8a1ffb700 1 -- 172.21.15.86:0/2891415273 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe8a40fe530 con 0x7fe8940a1d80 2024-03-04T12:18:23.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.267+0000 7fe8aac74700 1 -- 172.21.15.86:0/2891415273 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe8940a1d80 msgr2=0x7fe8940a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:23.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.267+0000 7fe8aac74700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe8940a1d80 0x7fe8940a4240 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4101e20 tx=0x7fe898009190 comp rx=0 tx=0).stop 2024-03-04T12:18:23.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.267+0000 7fe8aac74700 1 -- 172.21.15.86:0/2891415273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8a4102960 msgr2=0x7fe8a419f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:23.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.267+0000 7fe8aac74700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8a4102960 0x7fe8a419f6a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe89c00add0 tx=0x7fe89c00c330 comp rx=0 tx=0).stop 2024-03-04T12:18:23.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.267+0000 7fe8aac74700 1 -- 172.21.15.86:0/2891415273 shutdown_connections 2024-03-04T12:18:23.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.267+0000 7fe8aac74700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe8a4102960 0x7fe8a419f6a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:23.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.267+0000 7fe8aac74700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe8940a1d80 0x7fe8940a4240 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:23.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.267+0000 7fe8aac74700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe8a4100dc0 0x7fe8a41980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:23.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.267+0000 7fe8aac74700 1 --2- 172.21.15.86:0/2891415273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8a4101fc0 0x7fe8a4198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:23.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.267+0000 7fe8aac74700 1 -- 172.21.15.86:0/2891415273 >> 172.21.15.86:0/2891415273 conn(0x7fe8a40fc380 msgr2=0x7fe8a40fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:18:23.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.267+0000 7fe8aac74700 1 -- 172.21.15.86:0/2891415273 shutdown_connections 2024-03-04T12:18:23.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:18:23.267+0000 7fe8aac74700 1 -- 172.21.15.86:0/2891415273 wait complete. 2024-03-04T12:18:23.568 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:23 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.568 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:23 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.568 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:23 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:23.568 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:23 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:23.568 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:23 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.568 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:23 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:23.568 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:23 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.568 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:23 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:23 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:23 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:23 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:23.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:23 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:23.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:23 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:23 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:23.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:23 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:23 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.851 DEBUG:teuthology.orchestra.run.smithi086:osd.2> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.2.service 2024-03-04T12:18:23.854 INFO:tasks.cephadm:Deploying osd.3 on smithi060 with /dev/vg_nvme/lv_3... 2024-03-04T12:18:23.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-04T12:18:23.884 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:23 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.884 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:23 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.884 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:23 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:23.884 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:23 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:23.884 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:23 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.884 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:23 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:23.884 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:23 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:23.884 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:23 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:24.109 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:18:24.563 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:24 smithi060 ceph-mon[32745]: from='osd.2 [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T12:18:24.563 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:24 smithi060 ceph-mon[32745]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T12:18:24.563 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:24 smithi060 ceph-mon[32745]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:24.699 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:24 smithi086 ceph-mon[36851]: from='osd.2 [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T12:18:24.699 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:24 smithi086 ceph-mon[36851]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T12:18:24.699 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:24 smithi086 ceph-mon[36851]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:24.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:24 smithi063 ceph-mon[36717]: from='osd.2 [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T12:18:24.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:24 smithi063 ceph-mon[36717]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T12:18:24.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:24 smithi063 ceph-mon[36717]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:25.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:25 smithi086 ceph-mon[36851]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T12:18:25.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:25 smithi086 ceph-mon[36851]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T12:18:25.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:25 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:25.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:25 smithi086 ceph-mon[36851]: from='osd.2 [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:18:25.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:25 smithi086 ceph-mon[36851]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:18:25.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:25 smithi063 ceph-mon[36717]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T12:18:25.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:25 smithi063 ceph-mon[36717]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T12:18:25.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:25 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:25.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:25 smithi063 ceph-mon[36717]: from='osd.2 [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:18:25.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:25 smithi063 ceph-mon[36717]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:18:25.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:25 smithi060 ceph-mon[32745]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T12:18:25.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:25 smithi060 ceph-mon[32745]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T12:18:25.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:25 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:25.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:25 smithi060 ceph-mon[32745]: from='osd.2 [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:18:25.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:25 smithi060 ceph-mon[32745]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:18:26.025 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-04T12:18:26.025 INFO:teuthology.orchestra.run.smithi060.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-04T12:18:26.025 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10+0 records in 2024-03-04T12:18:26.025 INFO:teuthology.orchestra.run.smithi060.stderr:10+0 records out 2024-03-04T12:18:26.025 INFO:teuthology.orchestra.run.smithi060.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0083706 s, 1.3 GB/s 2024-03-04T12:18:26.025 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping successful for: 2024-03-04T12:18:26.146 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:18:25 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2[41969]: 2024-03-04T12:18:25.883+0000 7f941485f700 -1 osd.2 0 waiting for initial osdmap 2024-03-04T12:18:26.146 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:18:25 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2[41969]: 2024-03-04T12:18:25.887+0000 7f940be49700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T12:18:26.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:26 smithi063 ceph-mon[36717]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T12:18:26.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:26 smithi063 ceph-mon[36717]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T12:18:26.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:26 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:26.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:26 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:26.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:26 smithi063 ceph-mon[36717]: from='osd.2 ' entity='osd.2' 2024-03-04T12:18:26.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:26 smithi063 ceph-mon[36717]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:26.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:26 smithi060 ceph-mon[32745]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T12:18:26.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:26 smithi060 ceph-mon[32745]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T12:18:26.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:26 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:26.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:26 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:26.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:26 smithi060 ceph-mon[32745]: from='osd.2 ' entity='osd.2' 2024-03-04T12:18:26.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:26 smithi060 ceph-mon[32745]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:26.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi060:vg_nvme/lv_3 2024-03-04T12:18:26.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:26 smithi086 ceph-mon[36851]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T12:18:26.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:26 smithi086 ceph-mon[36851]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T12:18:26.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:26 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:26.774 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:26 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:26.774 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:26 smithi086 ceph-mon[36851]: from='osd.2 ' entity='osd.2' 2024-03-04T12:18:26.774 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:26 smithi086 ceph-mon[36851]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-04T12:18:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:18:27.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: purged_snaps scrub starts 2024-03-04T12:18:27.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: purged_snaps scrub ok 2024-03-04T12:18:27.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:27.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: Detected new or changed devices on smithi086 2024-03-04T12:18:27.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:27.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:27.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:18:27.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi086 to 17080M 2024-03-04T12:18:27.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:27.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: osd.2 [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222] boot 2024-03-04T12:18:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T12:18:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:27.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: purged_snaps scrub starts 2024-03-04T12:18:27.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: purged_snaps scrub ok 2024-03-04T12:18:27.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:27.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: Detected new or changed devices on smithi086 2024-03-04T12:18:27.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:27.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:27.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:18:27.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi086 to 17080M 2024-03-04T12:18:27.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:27.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:27.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:27.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:27.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: osd.2 [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222] boot 2024-03-04T12:18:27.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T12:18:27.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: purged_snaps scrub starts 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: purged_snaps scrub ok 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: Detected new or changed devices on smithi086 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi086 to 17080M 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:27.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: osd.2 [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222] boot 2024-03-04T12:18:27.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T12:18:27.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:18:28.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.661+0000 7f78e2279700 1 -- 172.21.15.60:0/1940289709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc102fd0 msgr2=0x7f78dc103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:28.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.661+0000 7f78e2279700 1 --2- 172.21.15.60:0/1940289709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc102fd0 0x7f78dc103450 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009990 tx=0x7f78c4009ca0 comp rx=0 tx=0).stop 2024-03-04T12:18:28.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.661+0000 7f78e2279700 1 -- 172.21.15.60:0/1940289709 shutdown_connections 2024-03-04T12:18:28.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.661+0000 7f78e2279700 1 --2- 172.21.15.60:0/1940289709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78dc103990 0x7f78dc10a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:28.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.661+0000 7f78e2279700 1 --2- 172.21.15.60:0/1940289709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78dc101e70 0x7f78dc102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:28.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.661+0000 7f78e2279700 1 --2- 172.21.15.60:0/1940289709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc102fd0 0x7f78dc103450 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:28.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.661+0000 7f78e2279700 1 -- 172.21.15.60:0/1940289709 >> 172.21.15.60:0/1940289709 conn(0x7f78dc0fd430 msgr2=0x7f78dc0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:18:28.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.661+0000 7f78e2279700 1 -- 172.21.15.60:0/1940289709 shutdown_connections 2024-03-04T12:18:28.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.661+0000 7f78e2279700 1 -- 172.21.15.60:0/1940289709 wait complete. 2024-03-04T12:18:28.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.661+0000 7f78e2279700 1 Processor -- start 2024-03-04T12:18:28.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78e2279700 1 -- start start 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78e2279700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc101e70 0x7f78dc198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78e2279700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78dc102fd0 0x7f78dc1985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78e2279700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78dc103990 0x7f78dc19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78e2279700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc10da40 con 0x7f78dc101e70 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78e2279700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc10d8c0 con 0x7f78dc102fd0 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78e2279700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78dc10dbc0 con 0x7f78dc103990 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78dbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc101e70 0x7f78dc198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78dbfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc101e70 0x7f78dc198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53826/0 (socket says 172.21.15.60:53826) 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78dbfff700 1 -- 172.21.15.60:0/878307330 learned_addr learned my addr 172.21.15.60:0/878307330 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78db7fe700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78dc102fd0 0x7f78dc1985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78e0816700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78dc103990 0x7f78dc19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78dbfff700 1 -- 172.21.15.60:0/878307330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78dc103990 msgr2=0x7f78dc19f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78dbfff700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78dc103990 0x7f78dc19f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:28.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78dbfff700 1 -- 172.21.15.60:0/878307330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78dc102fd0 msgr2=0x7f78dc1985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:28.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78dbfff700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78dc102fd0 0x7f78dc1985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:28.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.662+0000 7f78dbfff700 1 -- 172.21.15.60:0/878307330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78dc19fcd0 con 0x7f78dc101e70 2024-03-04T12:18:28.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.663+0000 7f78dbfff700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc101e70 0x7f78dc198070 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00c950 tx=0x7f78cc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:18:28.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.663+0000 7f78d97fa700 1 -- 172.21.15.60:0/878307330 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78cc007ac0 con 0x7f78dc101e70 2024-03-04T12:18:28.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.663+0000 7f78d97fa700 1 -- 172.21.15.60:0/878307330 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78cc004d10 con 0x7f78dc101e70 2024-03-04T12:18:28.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.663+0000 7f78e2279700 1 -- 172.21.15.60:0/878307330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78dc19ffc0 con 0x7f78dc101e70 2024-03-04T12:18:28.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.663+0000 7f78d97fa700 1 -- 172.21.15.60:0/878307330 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78cc00f450 con 0x7f78dc101e70 2024-03-04T12:18:28.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.663+0000 7f78e2279700 1 -- 172.21.15.60:0/878307330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78dc1a0410 con 0x7f78dc101e70 2024-03-04T12:18:28.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.664+0000 7f78e2279700 1 -- 172.21.15.60:0/878307330 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78dc068ae0 con 0x7f78dc101e70 2024-03-04T12:18:28.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.665+0000 7f78d97fa700 1 -- 172.21.15.60:0/878307330 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 15) v1 ==== 136697+0+0 (secure 0 0 0) 0x7f78cc004750 con 0x7f78dc101e70 2024-03-04T12:18:28.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.665+0000 7f78d97fa700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f78c80a1d80 0x7f78c80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:18:28.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.666+0000 7f78db7fe700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f78c80a1d80 0x7f78c80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:18:28.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.666+0000 7f78d97fa700 1 -- 172.21.15.60:0/878307330 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7f78cc0cc780 con 0x7f78dc101e70 2024-03-04T12:18:28.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.666+0000 7f78db7fe700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f78c80a1d80 0x7f78c80a4240 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78c400b5c0 tx=0x7f78c40038f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:18:28.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.669+0000 7f78d97fa700 1 -- 172.21.15.60:0/878307330 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f78cc014030 con 0x7f78dc101e70 2024-03-04T12:18:28.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:28 smithi086 ceph-mon[36851]: pgmap v80: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:28.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:28 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T12:18:28.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:28 smithi063 ceph-mon[36717]: pgmap v80: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:28.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:28 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T12:18:28.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:28.883+0000 7f78e2279700 1 -- 172.21.15.60:0/878307330 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f78dc062260 con 0x7f78c80a1d80 2024-03-04T12:18:28.888 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:28 smithi060 ceph-mon[32745]: pgmap v80: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:28.888 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:28 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T12:18:29.755 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:29 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T12:18:29.755 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:29 smithi060 ceph-mon[32745]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T12:18:29.755 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:29 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T12:18:29.755 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:29 smithi060 ceph-mon[32745]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:18:29.755 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:29 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:18:29.755 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:29 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:18:29.755 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:29 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:29.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:29 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T12:18:29.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:29 smithi086 ceph-mon[36851]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T12:18:29.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:29 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T12:18:29.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:29 smithi086 ceph-mon[36851]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:18:29.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:29 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:18:29.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:29 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:18:29.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:29 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:29.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:29 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T12:18:29.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:29 smithi063 ceph-mon[36717]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T12:18:29.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:29 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T12:18:29.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:29 smithi063 ceph-mon[36717]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:18:29.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:29 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:18:29.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:29 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:18:29.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:29 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:30.097 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:18:29 smithi060 sudo[52192]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T12:18:30.097 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:18:29 smithi060 sudo[52192]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T12:18:30.097 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:18:29 smithi060 sudo[52192]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T12:18:30.097 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:18:29 smithi060 sudo[52192]: pam_unix(sudo:session): session closed for user root 2024-03-04T12:18:30.426 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:18:30 smithi060 sudo[52195]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T12:18:30.426 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:18:30 smithi060 sudo[52195]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T12:18:30.427 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:18:30 smithi060 sudo[52195]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T12:18:30.427 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:18:30 smithi060 sudo[52195]: pam_unix(sudo:session): session closed for user root 2024-03-04T12:18:30.727 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:30 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T12:18:30.727 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:30 smithi060 ceph-mon[32745]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T12:18:30.727 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:30 smithi060 ceph-mon[32745]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:30.727 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:30 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:30.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:30 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T12:18:30.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:30 smithi086 ceph-mon[36851]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T12:18:30.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:30 smithi086 ceph-mon[36851]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:30.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:30 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:30.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:30 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T12:18:30.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:30 smithi063 ceph-mon[36717]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T12:18:30.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:30 smithi063 ceph-mon[36717]: pgmap v83: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:30.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:30 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:30.850 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:18:30 smithi063 sudo[45295]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T12:18:30.850 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:18:30 smithi063 sudo[45295]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T12:18:30.850 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:18:30 smithi063 sudo[45295]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T12:18:30.850 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:18:30 smithi063 sudo[45295]: pam_unix(sudo:session): session closed for user root 2024-03-04T12:18:31.350 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:18:31 smithi063 sudo[45298]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T12:18:31.350 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:18:31 smithi063 sudo[45298]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T12:18:31.350 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:18:31 smithi063 sudo[45298]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T12:18:31.350 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:18:31 smithi063 sudo[45298]: pam_unix(sudo:session): session closed for user root 2024-03-04T12:18:31.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:31.473+0000 7f78d97fa700 1 -- 172.21.15.60:0/878307330 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f78cc096fe0 con 0x7f78dc101e70 2024-03-04T12:18:31.543 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:31 smithi086 ceph-mon[36851]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T12:18:31.543 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:31 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:31.543 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:18:31 smithi086 sudo[45788]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T12:18:31.543 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:18:31 smithi086 sudo[45788]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T12:18:31.543 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:18:31 smithi086 sudo[45788]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T12:18:31.543 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:18:31 smithi086 sudo[45788]: pam_unix(sudo:session): session closed for user root 2024-03-04T12:18:31.819 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:31 smithi060 ceph-mon[32745]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T12:18:31.819 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:31 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:31.846 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:18:31 smithi086 sudo[45791]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T12:18:31.846 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:18:31 smithi086 sudo[45791]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T12:18:31.847 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:18:31 smithi086 sudo[45791]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T12:18:31.847 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:18:31 smithi086 sudo[45791]: pam_unix(sudo:session): session closed for user root 2024-03-04T12:18:31.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:31 smithi063 ceph-mon[36717]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T12:18:31.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:31 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:32.083 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:31 smithi060 sudo[52274]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T12:18:32.083 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:31 smithi060 sudo[52274]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T12:18:32.083 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:31 smithi060 sudo[52274]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T12:18:32.083 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:31 smithi060 sudo[52274]: pam_unix(sudo:session): session closed for user root 2024-03-04T12:18:32.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 sudo[52277]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T12:18:32.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 sudo[52277]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T12:18:32.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 sudo[52277]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T12:18:32.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 sudo[52277]: pam_unix(sudo:session): session closed for user root 2024-03-04T12:18:32.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 sudo[45794]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T12:18:32.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 sudo[45794]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T12:18:32.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 sudo[45794]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T12:18:32.579 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 sudo[45794]: pam_unix(sudo:session): session closed for user root 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 sudo[45797]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 sudo[45797]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 sudo[45797]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 sudo[45797]: pam_unix(sudo:session): session closed for user root 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3215870589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a823b593-a7b5-45ff-8765-cac26276dedd"}]: dispatch 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a823b593-a7b5-45ff-8765-cac26276dedd"}]: dispatch 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:18:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:18:32.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a823b593-a7b5-45ff-8765-cac26276dedd"}]': finished 2024-03-04T12:18:32.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: osdmap e22: 4 total, 3 up, 4 in 2024-03-04T12:18:32.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:32.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:18:32.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:18:32.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:18:32.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:32 smithi086 ceph-mon[36851]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3215870589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a823b593-a7b5-45ff-8765-cac26276dedd"}]: dispatch 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a823b593-a7b5-45ff-8765-cac26276dedd"}]: dispatch 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a823b593-a7b5-45ff-8765-cac26276dedd"}]': finished 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: osdmap e22: 4 total, 3 up, 4 in 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:18:32.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:18:32.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:32 smithi063 ceph-mon[36717]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:32.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-04T12:18:32.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:32.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T12:18:32.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3215870589' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a823b593-a7b5-45ff-8765-cac26276dedd"}]: dispatch 2024-03-04T12:18:32.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T12:18:32.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:18:32.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a823b593-a7b5-45ff-8765-cac26276dedd"}]: dispatch 2024-03-04T12:18:32.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:18:32.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:18:32.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a823b593-a7b5-45ff-8765-cac26276dedd"}]': finished 2024-03-04T12:18:32.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: osdmap e22: 4 total, 3 up, 4 in 2024-03-04T12:18:32.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:32.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:18:32.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:18:32.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:18:32.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:32 smithi060 ceph-mon[32745]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:33.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 sudo[45301]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T12:18:33.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 sudo[45301]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T12:18:33.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 sudo[45301]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T12:18:33.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 sudo[45301]: pam_unix(sudo:session): session closed for user root 2024-03-04T12:18:33.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:33 smithi086 ceph-mon[36851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T12:18:33.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:33 smithi086 ceph-mon[36851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T12:18:33.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:33 smithi086 ceph-mon[36851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T12:18:33.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:33 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:18:33.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:33 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:18:33.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:33 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:18:33.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:33 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2395631485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:18:33.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:33 smithi086 ceph-mon[36851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T12:18:33.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 sudo[45304]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T12:18:33.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 sudo[45304]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T12:18:33.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 sudo[45304]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T12:18:33.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 sudo[45304]: pam_unix(sudo:session): session closed for user root 2024-03-04T12:18:33.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 ceph-mon[36717]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T12:18:33.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 ceph-mon[36717]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T12:18:33.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 ceph-mon[36717]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T12:18:33.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:18:33.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:18:33.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:18:33.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2395631485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:18:33.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:33 smithi063 ceph-mon[36717]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T12:18:33.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:33 smithi060 ceph-mon[32745]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T12:18:33.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:33 smithi060 ceph-mon[32745]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T12:18:33.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:33 smithi060 ceph-mon[32745]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T12:18:33.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:33 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:18:33.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:33 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:18:33.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:33 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:18:33.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:33 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2395631485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:18:33.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:33 smithi060 ceph-mon[32745]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T12:18:34.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:34 smithi086 ceph-mon[36851]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:34.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:34 smithi063 ceph-mon[36717]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:34.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:34 smithi060 ceph-mon[32745]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:36.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:36 smithi060 ceph-mon[32745]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:36.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:36 smithi086 ceph-mon[36851]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:36.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:36 smithi063 ceph-mon[36717]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:38.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:38 smithi060 ceph-mon[32745]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:38.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:38 smithi086 ceph-mon[36851]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:38.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:38 smithi063 ceph-mon[36717]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:39.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T12:18:39.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:39.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T12:18:39.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:39.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T12:18:39.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:40.831 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:40 smithi060 ceph-mon[32745]: Deploying daemon osd.3 on smithi060 2024-03-04T12:18:40.832 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:40 smithi060 ceph-mon[32745]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:40.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:40 smithi086 ceph-mon[36851]: Deploying daemon osd.3 on smithi060 2024-03-04T12:18:40.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:40 smithi086 ceph-mon[36851]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:40.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:40 smithi063 ceph-mon[36717]: Deploying daemon osd.3 on smithi060 2024-03-04T12:18:40.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:40 smithi063 ceph-mon[36717]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:42.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:42 smithi060 ceph-mon[32745]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:42.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:42 smithi086 ceph-mon[36851]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:42.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:42 smithi063 ceph-mon[36717]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:44.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:44 smithi060 ceph-mon[32745]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:44.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:44 smithi086 ceph-mon[36851]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:44.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:44 smithi063 ceph-mon[36717]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:46.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:46 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:46.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:46 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:46.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:46 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:46.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:46 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:46.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:46 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:46.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:46 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:46.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:46 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:46.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:46 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:46.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:46 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:47.345 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:47 smithi060 ceph-mon[32745]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:47.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:47 smithi086 ceph-mon[36851]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:47.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:47 smithi063 ceph-mon[36717]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:48.329 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:48 smithi060 ceph-mon[32745]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:48.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.388+0000 7f78d97fa700 1 -- 172.21.15.60:0/878307330 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f78dc062260 con 0x7f78c80a1d80 2024-03-04T12:18:48.393 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 3 on host 'smithi060' 2024-03-04T12:18:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.390+0000 7f78e2279700 1 -- 172.21.15.60:0/878307330 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f78c80a1d80 msgr2=0x7f78c80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.390+0000 7f78e2279700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f78c80a1d80 0x7f78c80a4240 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78c400b5c0 tx=0x7f78c40038f0 comp rx=0 tx=0).stop 2024-03-04T12:18:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.390+0000 7f78e2279700 1 -- 172.21.15.60:0/878307330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc101e70 msgr2=0x7f78dc198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.390+0000 7f78e2279700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc101e70 0x7f78dc198070 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00c950 tx=0x7f78cc00cc60 comp rx=0 tx=0).stop 2024-03-04T12:18:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.390+0000 7f78e2279700 1 -- 172.21.15.60:0/878307330 shutdown_connections 2024-03-04T12:18:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.390+0000 7f78e2279700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78dc103990 0x7f78dc19f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.390+0000 7f78e2279700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f78c80a1d80 0x7f78c80a4240 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.390+0000 7f78e2279700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f78dc101e70 0x7f78dc198070 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.390+0000 7f78e2279700 1 --2- 172.21.15.60:0/878307330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78dc102fd0 0x7f78dc1985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.390+0000 7f78e2279700 1 -- 172.21.15.60:0/878307330 >> 172.21.15.60:0/878307330 conn(0x7f78dc0fd430 msgr2=0x7f78dc106300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:18:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.391+0000 7f78e2279700 1 -- 172.21.15.60:0/878307330 shutdown_connections 2024-03-04T12:18:48.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:18:48.391+0000 7f78e2279700 1 -- 172.21.15.60:0/878307330 wait complete. 2024-03-04T12:18:48.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:48 smithi086 ceph-mon[36851]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:48.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:48 smithi063 ceph-mon[36717]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:49.016 DEBUG:teuthology.orchestra.run.smithi060:osd.3> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.3.service 2024-03-04T12:18:49.019 INFO:tasks.cephadm:Deploying osd.4 on smithi063 with /dev/vg_nvme/lv_3... 2024-03-04T12:18:49.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-04T12:18:49.277 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T12:18:49.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:49.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:49.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:49.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:49.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:49 smithi060 ceph-mon[32745]: from='osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T12:18:49.692 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:49.692 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:49.693 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:49.693 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:49.693 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:49 smithi063 ceph-mon[36717]: from='osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T12:18:49.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:49.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:49.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:49.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:49.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:49 smithi086 ceph-mon[36851]: from='osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T12:18:50.759 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:50 smithi063 ceph-mon[36717]: from='osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T12:18:50.759 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:50 smithi063 ceph-mon[36717]: osdmap e23: 4 total, 3 up, 4 in 2024-03-04T12:18:50.759 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:50 smithi063 ceph-mon[36717]: from='osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-03-04T12:18:50.760 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:50 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:50.760 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:50 smithi063 ceph-mon[36717]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:50.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:50 smithi086 ceph-mon[36851]: from='osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T12:18:50.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:50 smithi086 ceph-mon[36851]: osdmap e23: 4 total, 3 up, 4 in 2024-03-04T12:18:50.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:50 smithi086 ceph-mon[36851]: from='osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-03-04T12:18:50.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:50 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:50.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:50 smithi086 ceph-mon[36851]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:50.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:50 smithi060 ceph-mon[32745]: from='osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T12:18:50.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:50 smithi060 ceph-mon[32745]: osdmap e23: 4 total, 3 up, 4 in 2024-03-04T12:18:50.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:50 smithi060 ceph-mon[32745]: from='osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-03-04T12:18:50.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:50 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:50.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:50 smithi060 ceph-mon[32745]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:50.927 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:18:50 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3[54334]: 2024-03-04T12:18:50.444+0000 7fd2260ff700 -1 osd.3 0 waiting for initial osdmap 2024-03-04T12:18:50.927 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:18:50 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3[54334]: 2024-03-04T12:18:50.449+0000 7fd221ef2700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T12:18:51.345 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-04T12:18:51.345 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-04T12:18:51.345 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-03-04T12:18:51.345 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-03-04T12:18:51.345 INFO:teuthology.orchestra.run.smithi063.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125933 s, 833 MB/s 2024-03-04T12:18:51.345 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-03-04T12:18:51.623 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:51 smithi060 ceph-mon[32745]: from='osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-03-04T12:18:51.623 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:51 smithi060 ceph-mon[32745]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T12:18:51.623 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:51 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:51.623 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:51 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:51.623 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:51 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:51.655 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:51 smithi063 ceph-mon[36717]: from='osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-03-04T12:18:51.655 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:51 smithi063 ceph-mon[36717]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T12:18:51.655 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:51 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:51.655 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:51 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:51.655 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:51 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:51.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:51 smithi086 ceph-mon[36851]: from='osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-03-04T12:18:51.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:51 smithi086 ceph-mon[36851]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T12:18:51.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:51 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:51.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:51 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:51.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:51 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:52.158 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi063:vg_nvme/lv_3 2024-03-04T12:18:52.407 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T12:18:52.776 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: purged_snaps scrub starts 2024-03-04T12:18:52.776 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: purged_snaps scrub ok 2024-03-04T12:18:52.776 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157] boot 2024-03-04T12:18:52.776 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: osdmap e25: 4 total, 4 up, 4 in 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: Detected new or changed devices on smithi060 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi060 to 8546M 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:52.777 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:52.778 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: purged_snaps scrub starts 2024-03-04T12:18:52.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: purged_snaps scrub ok 2024-03-04T12:18:52.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157] boot 2024-03-04T12:18:52.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: osdmap e25: 4 total, 4 up, 4 in 2024-03-04T12:18:52.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:52.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: Detected new or changed devices on smithi060 2024-03-04T12:18:52.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:18:52.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:18:52.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi060 to 8546M 2024-03-04T12:18:52.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:52.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:52.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:52.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:52.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:52.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:52.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: purged_snaps scrub starts 2024-03-04T12:18:52.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: purged_snaps scrub ok 2024-03-04T12:18:52.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: osd.3 [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157] boot 2024-03-04T12:18:52.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: osdmap e25: 4 total, 4 up, 4 in 2024-03-04T12:18:52.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:18:52.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: Detected new or changed devices on smithi060 2024-03-04T12:18:52.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:18:52.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:18:52.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi060 to 8546M 2024-03-04T12:18:52.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:52.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:52.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:52.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 96 MiB used, 268 GiB / 268 GiB avail 2024-03-04T12:18:52.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:18:52.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:52.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:18:52.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:18:53.529 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:53 smithi063 ceph-mon[36717]: osdmap e26: 4 total, 4 up, 4 in 2024-03-04T12:18:53.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:53 smithi086 ceph-mon[36851]: osdmap e26: 4 total, 4 up, 4 in 2024-03-04T12:18:53.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:53 smithi060 ceph-mon[32745]: osdmap e26: 4 total, 4 up, 4 in 2024-03-04T12:18:54.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.096+0000 7fd414345700 1 -- 172.21.15.63:0/1988084512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c1028f0 msgr2=0x7fd40c1091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:54.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.096+0000 7fd414345700 1 --2- 172.21.15.63:0/1988084512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c1028f0 0x7fd40c1091a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd40400b0d0 tx=0x7fd40400b3e0 comp rx=0 tx=0).stop 2024-03-04T12:18:54.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.096+0000 7fd414345700 1 -- 172.21.15.63:0/1988084512 shutdown_connections 2024-03-04T12:18:54.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.096+0000 7fd414345700 1 --2- 172.21.15.63:0/1988084512 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c1028f0 0x7fd40c1091a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:54.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.096+0000 7fd414345700 1 --2- 172.21.15.63:0/1988084512 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd40c100d50 0x7fd40c101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.096+0000 7fd414345700 1 --2- 172.21.15.63:0/1988084512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd40c101f50 0x7fd40c1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.096+0000 7fd414345700 1 -- 172.21.15.63:0/1988084512 >> 172.21.15.63:0/1988084512 conn(0x7fd40c0fc2f0 msgr2=0x7fd40c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.096+0000 7fd414345700 1 -- 172.21.15.63:0/1988084512 shutdown_connections 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.096+0000 7fd414345700 1 -- 172.21.15.63:0/1988084512 wait complete. 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd414345700 1 Processor -- start 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd414345700 1 -- start start 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd414345700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd40c100d50 0x7fd40c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd414345700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c101f50 0x7fd40c1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd414345700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd40c1028f0 0x7fd40c19f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd414345700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c10c940 con 0x7fd40c101f50 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd414345700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c10c7c0 con 0x7fd40c1028f0 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd414345700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c10cac0 con 0x7fd40c100d50 2024-03-04T12:18:54.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd4128e2700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd40c1028f0 0x7fd40c19f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd4128e2700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd40c1028f0 0x7fd40c19f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42008/0 (socket says 172.21.15.63:42008) 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd4128e2700 1 -- 172.21.15.63:0/3665933838 learned_addr learned my addr 172.21.15.63:0/3665933838 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd4118e0700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c101f50 0x7fd40c1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.097+0000 7fd4120e1700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd40c100d50 0x7fd40c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd4128e2700 1 -- 172.21.15.63:0/3665933838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd40c100d50 msgr2=0x7fd40c1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd4128e2700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd40c100d50 0x7fd40c1980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd4128e2700 1 -- 172.21.15.63:0/3665933838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c101f50 msgr2=0x7fd40c1985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd4128e2700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c101f50 0x7fd40c1985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd4128e2700 1 -- 172.21.15.63:0/3665933838 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c19fd00 con 0x7fd40c1028f0 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd4120e1700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd40c100d50 0x7fd40c1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd4118e0700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c101f50 0x7fd40c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd4128e2700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd40c1028f0 0x7fd40c19f660 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd404007a20 tx=0x7fd404007a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd4037fe700 1 -- 172.21.15.63:0/3665933838 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd404021020 con 0x7fd40c1028f0 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd414345700 1 -- 172.21.15.63:0/3665933838 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40c19ff90 con 0x7fd40c1028f0 2024-03-04T12:18:54.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd4037fe700 1 -- 172.21.15.63:0/3665933838 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd40401d950 con 0x7fd40c1028f0 2024-03-04T12:18:54.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd4037fe700 1 -- 172.21.15.63:0/3665933838 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd404005430 con 0x7fd40c1028f0 2024-03-04T12:18:54.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.098+0000 7fd414345700 1 -- 172.21.15.63:0/3665933838 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40c1a03e0 con 0x7fd40c1028f0 2024-03-04T12:18:54.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.099+0000 7fd414345700 1 -- 172.21.15.63:0/3665933838 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40c068ae0 con 0x7fd40c1028f0 2024-03-04T12:18:54.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.100+0000 7fd4037fe700 1 -- 172.21.15.63:0/3665933838 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fd404024020 con 0x7fd40c1028f0 2024-03-04T12:18:54.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.101+0000 7fd4037fe700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd3f80a1e10 0x7fd3f80a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:18:54.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.101+0000 7fd4037fe700 1 -- 172.21.15.63:0/3665933838 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3698+0+0 (secure 0 0 0) 0x7fd4040cde40 con 0x7fd40c1028f0 2024-03-04T12:18:54.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.101+0000 7fd4120e1700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd3f80a1e10 0x7fd3f80a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:18:54.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.102+0000 7fd4120e1700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd3f80a1e10 0x7fd3f80a42d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd40c101db0 tx=0x7fd408009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:18:54.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.106+0000 7fd4037fe700 1 -- 172.21.15.63:0/3665933838 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd40402b020 con 0x7fd40c1028f0 2024-03-04T12:18:54.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:18:54.316+0000 7fd414345700 1 -- 172.21.15.63:0/3665933838 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd40c0fe4a0 con 0x7fd3f80a1e10 2024-03-04T12:18:54.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:54 smithi086 ceph-mon[36851]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T12:18:54.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:54 smithi086 ceph-mon[36851]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T12:18:54.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:54 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:18:54.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:54 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:18:54.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:54 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:54.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:54 smithi063 ceph-mon[36717]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T12:18:54.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:54 smithi063 ceph-mon[36717]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T12:18:54.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:54 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:18:54.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:54 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:18:54.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:54 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:54.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:54 smithi060 ceph-mon[32745]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T12:18:54.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:54 smithi060 ceph-mon[32745]: pgmap v102: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T12:18:54.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:54 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:18:54.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:54 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:18:54.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:54 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:18:55.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:55 smithi063 ceph-mon[36717]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:18:55.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:55 smithi063 ceph-mon[36717]: osdmap e28: 4 total, 4 up, 4 in 2024-03-04T12:18:55.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:55 smithi086 ceph-mon[36851]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:18:55.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:55 smithi086 ceph-mon[36851]: osdmap e28: 4 total, 4 up, 4 in 2024-03-04T12:18:55.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:55 smithi060 ceph-mon[32745]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:18:55.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:55 smithi060 ceph-mon[32745]: osdmap e28: 4 total, 4 up, 4 in 2024-03-04T12:18:56.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:56 smithi086 ceph-mon[36851]: osdmap e29: 4 total, 4 up, 4 in 2024-03-04T12:18:56.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:56 smithi086 ceph-mon[36851]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T12:18:56.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:56 smithi063 ceph-mon[36717]: osdmap e29: 4 total, 4 up, 4 in 2024-03-04T12:18:56.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:56 smithi063 ceph-mon[36717]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T12:18:56.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:56 smithi060 ceph-mon[32745]: osdmap e29: 4 total, 4 up, 4 in 2024-03-04T12:18:56.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:56 smithi060 ceph-mon[32745]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T12:18:57.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:57 smithi086 ceph-mon[36851]: from='client.? 172.21.15.63:0/2929781966' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75da5356-1007-4b12-89dc-5e91ec454df8"}]: dispatch 2024-03-04T12:18:57.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:57 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75da5356-1007-4b12-89dc-5e91ec454df8"}]: dispatch 2024-03-04T12:18:57.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:57 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75da5356-1007-4b12-89dc-5e91ec454df8"}]': finished 2024-03-04T12:18:57.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:57 smithi086 ceph-mon[36851]: osdmap e30: 5 total, 4 up, 5 in 2024-03-04T12:18:57.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:57 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:18:57.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:57 smithi063 ceph-mon[36717]: from='client.? 172.21.15.63:0/2929781966' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75da5356-1007-4b12-89dc-5e91ec454df8"}]: dispatch 2024-03-04T12:18:57.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:57 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75da5356-1007-4b12-89dc-5e91ec454df8"}]: dispatch 2024-03-04T12:18:57.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:57 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75da5356-1007-4b12-89dc-5e91ec454df8"}]': finished 2024-03-04T12:18:57.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:57 smithi063 ceph-mon[36717]: osdmap e30: 5 total, 4 up, 5 in 2024-03-04T12:18:57.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:57 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:18:57.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:57 smithi060 ceph-mon[32745]: from='client.? 172.21.15.63:0/2929781966' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75da5356-1007-4b12-89dc-5e91ec454df8"}]: dispatch 2024-03-04T12:18:57.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:57 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75da5356-1007-4b12-89dc-5e91ec454df8"}]: dispatch 2024-03-04T12:18:57.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:57 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75da5356-1007-4b12-89dc-5e91ec454df8"}]': finished 2024-03-04T12:18:57.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:57 smithi060 ceph-mon[32745]: osdmap e30: 5 total, 4 up, 5 in 2024-03-04T12:18:57.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:57 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:18:58.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:58 smithi086 ceph-mon[36851]: from='client.? 172.21.15.63:0/3202134884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:18:58.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:18:58 smithi086 ceph-mon[36851]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-04T12:18:58.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:58 smithi063 ceph-mon[36717]: from='client.? 172.21.15.63:0/3202134884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:18:58.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:18:58 smithi063 ceph-mon[36717]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-04T12:18:58.926 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:58 smithi060 ceph-mon[32745]: from='client.? 172.21.15.63:0/3202134884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:18:58.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:18:58 smithi060 ceph-mon[32745]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-03-04T12:19:00.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:00 smithi060 ceph-mon[32745]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:00.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:00 smithi086 ceph-mon[36851]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:00.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:00 smithi063 ceph-mon[36717]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:02.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:02 smithi063 ceph-mon[36717]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:02.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:02 smithi060 ceph-mon[32745]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:02.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:02 smithi086 ceph-mon[36851]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:04.410 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:04 smithi063 ceph-mon[36717]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:04.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:04 smithi060 ceph-mon[32745]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:04.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:04 smithi086 ceph-mon[36851]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T12:19:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:05.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:05 smithi063 ceph-mon[36717]: Deploying daemon osd.4 on smithi063 2024-03-04T12:19:05.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T12:19:05.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:05.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:05 smithi060 ceph-mon[32745]: Deploying daemon osd.4 on smithi063 2024-03-04T12:19:05.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T12:19:05.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:05.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:05 smithi086 ceph-mon[36851]: Deploying daemon osd.4 on smithi063 2024-03-04T12:19:06.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:06 smithi063 ceph-mon[36717]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:06.426 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:06 smithi060 ceph-mon[32745]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:06.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:06 smithi086 ceph-mon[36851]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:08.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:08 smithi063 ceph-mon[36717]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:08.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:08 smithi060 ceph-mon[32745]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:08.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:08 smithi086 ceph-mon[36851]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:10.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:10 smithi060 ceph-mon[32745]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:10.571 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:10 smithi063 ceph-mon[36717]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:10.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:10 smithi086 ceph-mon[36851]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:11.477 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:11.477 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:11.477 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:19:11.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:11.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:11.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:19:11.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:11.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:11.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:19:12.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:12 smithi063 ceph-mon[36717]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:12.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:12 smithi086 ceph-mon[36851]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:12.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:12 smithi060 ceph-mon[32745]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:14.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.087+0000 7fd4037fe700 1 -- 172.21.15.63:0/3665933838 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd40c0fe4a0 con 0x7fd3f80a1e10 2024-03-04T12:19:14.092 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 4 on host 'smithi063' 2024-03-04T12:19:14.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.089+0000 7fd414345700 1 -- 172.21.15.63:0/3665933838 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd3f80a1e10 msgr2=0x7fd3f80a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:19:14.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.089+0000 7fd414345700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd3f80a1e10 0x7fd3f80a42d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd40c101db0 tx=0x7fd408009260 comp rx=0 tx=0).stop 2024-03-04T12:19:14.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.089+0000 7fd414345700 1 -- 172.21.15.63:0/3665933838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd40c1028f0 msgr2=0x7fd40c19f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:19:14.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.089+0000 7fd414345700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd40c1028f0 0x7fd40c19f660 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd404007a20 tx=0x7fd404007a50 comp rx=0 tx=0).stop 2024-03-04T12:19:14.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.089+0000 7fd414345700 1 -- 172.21.15.63:0/3665933838 shutdown_connections 2024-03-04T12:19:14.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.089+0000 7fd414345700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd40c1028f0 0x7fd40c19f660 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:14.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.089+0000 7fd414345700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd40c100d50 0x7fd40c1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:14.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.089+0000 7fd414345700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd3f80a1e10 0x7fd3f80a42d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:14.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.089+0000 7fd414345700 1 --2- 172.21.15.63:0/3665933838 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd40c101f50 0x7fd40c1985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:14.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.089+0000 7fd414345700 1 -- 172.21.15.63:0/3665933838 >> 172.21.15.63:0/3665933838 conn(0x7fd40c0fc2f0 msgr2=0x7fd40c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:19:14.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.089+0000 7fd414345700 1 -- 172.21.15.63:0/3665933838 shutdown_connections 2024-03-04T12:19:14.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:19:14.090+0000 7fd414345700 1 -- 172.21.15.63:0/3665933838 wait complete. 2024-03-04T12:19:14.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:14 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:14 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:14 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:14.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:14 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:19:14.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:14 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:14 smithi063 ceph-mon[36717]: from='osd.4 [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T12:19:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:14 smithi063 ceph-mon[36717]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T12:19:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:14 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:14 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:14 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:19:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:14 smithi063 ceph-mon[36717]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:14.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:14 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:14 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:14 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:14.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:14 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:19:14.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:14 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:14 smithi086 ceph-mon[36851]: from='osd.4 [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T12:19:14.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:14 smithi086 ceph-mon[36851]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T12:19:14.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:14 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:14 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:14 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:19:14.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:14 smithi086 ceph-mon[36851]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:14.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:14 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:14 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:14 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:14.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:14 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:19:14.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:14 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:14 smithi060 ceph-mon[32745]: from='osd.4 [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T12:19:14.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:14 smithi060 ceph-mon[32745]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T12:19:14.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:14 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:14 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:14.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:14 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:19:14.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:14 smithi060 ceph-mon[32745]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:14.793 DEBUG:teuthology.orchestra.run.smithi063:osd.4> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.4.service 2024-03-04T12:19:14.796 INFO:tasks.cephadm:Deploying osd.5 on smithi086 with /dev/vg_nvme/lv_3... 2024-03-04T12:19:14.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-04T12:19:15.041 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:19:15.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:15 smithi063 ceph-mon[36717]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T12:19:15.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:15 smithi063 ceph-mon[36717]: osdmap e31: 5 total, 4 up, 5 in 2024-03-04T12:19:15.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:15.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:15 smithi063 ceph-mon[36717]: from='osd.4 [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:19:15.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:15 smithi063 ceph-mon[36717]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:19:15.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:15 smithi086 ceph-mon[36851]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T12:19:15.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:15 smithi086 ceph-mon[36851]: osdmap e31: 5 total, 4 up, 5 in 2024-03-04T12:19:15.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:15.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:15 smithi086 ceph-mon[36851]: from='osd.4 [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:19:15.416 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:15 smithi086 ceph-mon[36851]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:19:15.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:15 smithi060 ceph-mon[32745]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T12:19:15.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:15 smithi060 ceph-mon[32745]: osdmap e31: 5 total, 4 up, 5 in 2024-03-04T12:19:15.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:15.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:15 smithi060 ceph-mon[32745]: from='osd.4 [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:19:15.681 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:15 smithi060 ceph-mon[32745]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:19:15.969 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:19:15 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4[48088]: 2024-03-04T12:19:15.660+0000 7f632a70b700 -1 osd.4 0 waiting for initial osdmap 2024-03-04T12:19:15.969 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:19:15 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4[48088]: 2024-03-04T12:19:15.666+0000 7f6321cf5700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T12:19:16.240 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:16 smithi063 ceph-mon[36717]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-04T12:19:16.240 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:16 smithi063 ceph-mon[36717]: osdmap e32: 5 total, 4 up, 5 in 2024-03-04T12:19:16.240 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:16 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:16.241 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:16 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:16.241 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:16 smithi063 ceph-mon[36717]: from='osd.4 ' entity='osd.4' 2024-03-04T12:19:16.241 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:16 smithi063 ceph-mon[36717]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:16.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:16 smithi086 ceph-mon[36851]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-04T12:19:16.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:16 smithi086 ceph-mon[36851]: osdmap e32: 5 total, 4 up, 5 in 2024-03-04T12:19:16.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:16 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:16.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:16 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:16.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:16 smithi086 ceph-mon[36851]: from='osd.4 ' entity='osd.4' 2024-03-04T12:19:16.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:16 smithi086 ceph-mon[36851]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:16.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:16 smithi060 ceph-mon[32745]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-04T12:19:16.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:16 smithi060 ceph-mon[32745]: osdmap e32: 5 total, 4 up, 5 in 2024-03-04T12:19:16.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:16 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:16.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:16 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:16.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:16 smithi060 ceph-mon[32745]: from='osd.4 ' entity='osd.4' 2024-03-04T12:19:16.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:16 smithi060 ceph-mon[32745]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 123 MiB used, 357 GiB / 358 GiB avail 2024-03-04T12:19:16.941 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-04T12:19:16.942 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-04T12:19:16.942 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-03-04T12:19:16.942 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-03-04T12:19:16.942 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113444 s, 924 MB/s 2024-03-04T12:19:16.942 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-03-04T12:19:17.442 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:17 smithi063 ceph-mon[36717]: purged_snaps scrub starts 2024-03-04T12:19:17.443 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:17 smithi063 ceph-mon[36717]: purged_snaps scrub ok 2024-03-04T12:19:17.443 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:17 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:17.443 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:17 smithi063 ceph-mon[36717]: osd.4 [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955] boot 2024-03-04T12:19:17.443 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:17 smithi063 ceph-mon[36717]: osdmap e33: 5 total, 5 up, 5 in 2024-03-04T12:19:17.443 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:17 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:17.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:17 smithi086 ceph-mon[36851]: purged_snaps scrub starts 2024-03-04T12:19:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:17 smithi086 ceph-mon[36851]: purged_snaps scrub ok 2024-03-04T12:19:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:17 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:17 smithi086 ceph-mon[36851]: osd.4 [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955] boot 2024-03-04T12:19:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:17 smithi086 ceph-mon[36851]: osdmap e33: 5 total, 5 up, 5 in 2024-03-04T12:19:17.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:17 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:17.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_3 2024-03-04T12:19:17.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:17 smithi060 ceph-mon[32745]: purged_snaps scrub starts 2024-03-04T12:19:17.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:17 smithi060 ceph-mon[32745]: purged_snaps scrub ok 2024-03-04T12:19:17.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:17 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:17.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:17 smithi060 ceph-mon[32745]: osd.4 [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955] boot 2024-03-04T12:19:17.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:17 smithi060 ceph-mon[32745]: osdmap e33: 5 total, 5 up, 5 in 2024-03-04T12:19:17.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:17 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:19:17.888 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:19:18.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:18 smithi060 ceph-mon[32745]: osdmap e34: 5 total, 5 up, 5 in 2024-03-04T12:19:18.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:18 smithi060 ceph-mon[32745]: Detected new or changed devices on smithi063 2024-03-04T12:19:18.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:18.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:18.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:19:18.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:19:18.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:18 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi063 to 8540M 2024-03-04T12:19:18.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:18.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:18.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:19:18.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:18.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:18 smithi060 ceph-mon[32745]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:18.991 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:18 smithi086 ceph-mon[36851]: osdmap e34: 5 total, 5 up, 5 in 2024-03-04T12:19:18.991 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:18 smithi086 ceph-mon[36851]: Detected new or changed devices on smithi063 2024-03-04T12:19:18.991 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:18 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:18.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:18 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:18.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:18 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:19:18.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:18 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:19:18.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:18 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi063 to 8540M 2024-03-04T12:19:18.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:18 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:18.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:18 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:18.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:18 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:19:18.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:18 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:18.992 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:18 smithi086 ceph-mon[36851]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:19.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:18 smithi063 ceph-mon[36717]: osdmap e34: 5 total, 5 up, 5 in 2024-03-04T12:19:19.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:18 smithi063 ceph-mon[36717]: Detected new or changed devices on smithi063 2024-03-04T12:19:19.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:18 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:19.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:18 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:19.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:18 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:19:19.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:18 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:19:19.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:18 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi063 to 8540M 2024-03-04T12:19:19.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:18 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:19.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:18 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:19.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:18 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:19:19.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:18 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:19.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:18 smithi063 ceph-mon[36717]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.542+0000 7f6935486700 1 -- 172.21.15.86:0/3756733509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6930103970 msgr2=0x7f693010a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:19:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.542+0000 7f6935486700 1 --2- 172.21.15.86:0/3756733509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6930103970 0x7f693010a290 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f692800b0a0 tx=0x7f692800b3b0 comp rx=0 tx=0).stop 2024-03-04T12:19:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.543+0000 7f6935486700 1 -- 172.21.15.86:0/3756733509 shutdown_connections 2024-03-04T12:19:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.543+0000 7f6935486700 1 --2- 172.21.15.86:0/3756733509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6930103970 0x7f693010a290 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.543+0000 7f6935486700 1 --2- 172.21.15.86:0/3756733509 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6930101e50 0x7f6930102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.543+0000 7f6935486700 1 --2- 172.21.15.86:0/3756733509 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6930102fb0 0x7f6930103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.543+0000 7f6935486700 1 -- 172.21.15.86:0/3756733509 >> 172.21.15.86:0/3756733509 conn(0x7f69300fd430 msgr2=0x7f69300ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:19:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.543+0000 7f6935486700 1 -- 172.21.15.86:0/3756733509 shutdown_connections 2024-03-04T12:19:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.543+0000 7f6935486700 1 -- 172.21.15.86:0/3756733509 wait complete. 2024-03-04T12:19:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.543+0000 7f6935486700 1 Processor -- start 2024-03-04T12:19:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.543+0000 7f6935486700 1 -- start start 2024-03-04T12:19:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.543+0000 7f6935486700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6930101e50 0x7f6930197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:19:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f6935486700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6930102fb0 0x7f6930198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:19:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f6935486700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6930103970 0x7f693019f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:19:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f6935486700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693010d980 con 0x7f6930102fb0 2024-03-04T12:19:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f6935486700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693010d800 con 0x7f6930101e50 2024-03-04T12:19:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f6935486700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693010db00 con 0x7f6930103970 2024-03-04T12:19:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f692f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6930103970 0x7f693019f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:19:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f692f7fe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6930103970 0x7f693019f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34252/0 (socket says 172.21.15.86:34252) 2024-03-04T12:19:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f692f7fe700 1 -- 172.21.15.86:0/4107641307 learned_addr learned my addr 172.21.15.86:0/4107641307 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-03-04T12:19:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f692effd700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6930101e50 0x7f6930197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:19:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f692e7fc700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6930102fb0 0x7f6930198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:19:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f692f7fe700 1 -- 172.21.15.86:0/4107641307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6930101e50 msgr2=0x7f6930197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:19:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f692f7fe700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6930101e50 0x7f6930197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f692f7fe700 1 -- 172.21.15.86:0/4107641307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6930102fb0 msgr2=0x7f6930198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:19:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f692f7fe700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6930102fb0 0x7f6930198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f692f7fe700 1 -- 172.21.15.86:0/4107641307 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693019fb30 con 0x7f6930103970 2024-03-04T12:19:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f692effd700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6930101e50 0x7f6930197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:19:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f692f7fe700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6930103970 0x7f693019f490 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6928009cd0 tx=0x7f692800bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:19:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f6917fff700 1 -- 172.21.15.86:0/4107641307 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6928021070 con 0x7f6930103970 2024-03-04T12:19:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f6935486700 1 -- 172.21.15.86:0/4107641307 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693019fdc0 con 0x7f6930103970 2024-03-04T12:19:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f6917fff700 1 -- 172.21.15.86:0/4107641307 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f692801d930 con 0x7f6930103970 2024-03-04T12:19:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f6917fff700 1 -- 172.21.15.86:0/4107641307 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6928005630 con 0x7f6930103970 2024-03-04T12:19:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.544+0000 7f6935486700 1 -- 172.21.15.86:0/4107641307 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69301a0260 con 0x7f6930103970 2024-03-04T12:19:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.545+0000 7f6935486700 1 -- 172.21.15.86:0/4107641307 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6930068ae0 con 0x7f6930103970 2024-03-04T12:19:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.546+0000 7f6917fff700 1 -- 172.21.15.86:0/4107641307 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f6928024070 con 0x7f6930103970 2024-03-04T12:19:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.547+0000 7f6917fff700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f69180a1e10 0x7f69180a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:19:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.547+0000 7f6917fff700 1 -- 172.21.15.86:0/4107641307 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f69280ce160 con 0x7f6930103970 2024-03-04T12:19:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.547+0000 7f692effd700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f69180a1e10 0x7f69180a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:19:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.547+0000 7f692effd700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f69180a1e10 0x7f69180a42d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f692000bcf0 tx=0x7f692000b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:19:19.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.552+0000 7f6917fff700 1 -- 172.21.15.86:0/4107641307 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f69280253b0 con 0x7f6930103970 2024-03-04T12:19:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:19.753+0000 7f6935486700 1 -- 172.21.15.86:0/4107641307 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6930062260 con 0x7f69180a1e10 2024-03-04T12:19:20.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:19:20.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:19:20.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:20.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:19:20.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:19:20.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:20.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:19:20.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:19:20.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:21.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:20 smithi060 ceph-mon[32745]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:19:21.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:20 smithi060 ceph-mon[32745]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:21.276 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:20 smithi086 ceph-mon[36851]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:19:21.276 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:20 smithi086 ceph-mon[36851]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:21.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:20 smithi063 ceph-mon[36717]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:19:21.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:20 smithi063 ceph-mon[36717]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:22.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:22 smithi060 ceph-mon[32745]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:22.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:22 smithi086 ceph-mon[36851]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:22.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:22 smithi063 ceph-mon[36717]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:23.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:23 smithi060 ceph-mon[32745]: from='client.? 172.21.15.86:0/3814471032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d29eae4-5669-4e1e-a94f-ddb37bcc1a72"}]: dispatch 2024-03-04T12:19:23.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:23 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d29eae4-5669-4e1e-a94f-ddb37bcc1a72"}]: dispatch 2024-03-04T12:19:23.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:23 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d29eae4-5669-4e1e-a94f-ddb37bcc1a72"}]': finished 2024-03-04T12:19:23.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:23 smithi060 ceph-mon[32745]: osdmap e35: 6 total, 5 up, 6 in 2024-03-04T12:19:23.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:23 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:23.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:23 smithi060 ceph-mon[32745]: from='client.? 172.21.15.86:0/2313168133' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:19:23.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:23 smithi086 ceph-mon[36851]: from='client.? 172.21.15.86:0/3814471032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d29eae4-5669-4e1e-a94f-ddb37bcc1a72"}]: dispatch 2024-03-04T12:19:23.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:23 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d29eae4-5669-4e1e-a94f-ddb37bcc1a72"}]: dispatch 2024-03-04T12:19:23.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:23 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d29eae4-5669-4e1e-a94f-ddb37bcc1a72"}]': finished 2024-03-04T12:19:23.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:23 smithi086 ceph-mon[36851]: osdmap e35: 6 total, 5 up, 6 in 2024-03-04T12:19:23.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:23 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:23.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:23 smithi086 ceph-mon[36851]: from='client.? 172.21.15.86:0/2313168133' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:19:23.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:23 smithi063 ceph-mon[36717]: from='client.? 172.21.15.86:0/3814471032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d29eae4-5669-4e1e-a94f-ddb37bcc1a72"}]: dispatch 2024-03-04T12:19:23.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:23 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d29eae4-5669-4e1e-a94f-ddb37bcc1a72"}]: dispatch 2024-03-04T12:19:23.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:23 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d29eae4-5669-4e1e-a94f-ddb37bcc1a72"}]': finished 2024-03-04T12:19:23.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:23 smithi063 ceph-mon[36717]: osdmap e35: 6 total, 5 up, 6 in 2024-03-04T12:19:23.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:23 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:23.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:23 smithi063 ceph-mon[36717]: from='client.? 172.21.15.86:0/2313168133' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:19:24.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:24 smithi060 ceph-mon[32745]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:24.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:24 smithi086 ceph-mon[36851]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:24.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:24 smithi063 ceph-mon[36717]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:26.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:26 smithi086 ceph-mon[36851]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:26.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:26 smithi060 ceph-mon[32745]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:26.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:26 smithi063 ceph-mon[36717]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:28.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:28 smithi060 ceph-mon[32745]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:28.475 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:28 smithi086 ceph-mon[36851]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:28.489 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:28 smithi063 ceph-mon[36717]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:30.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:29 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T12:19:30.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:29 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:30.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:29 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T12:19:30.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:29 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:30.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:29 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T12:19:30.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:29 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:30.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:30 smithi086 ceph-mon[36851]: Deploying daemon osd.5 on smithi086 2024-03-04T12:19:30.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:30 smithi086 ceph-mon[36851]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:31.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:30 smithi063 ceph-mon[36717]: Deploying daemon osd.5 on smithi086 2024-03-04T12:19:31.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:30 smithi063 ceph-mon[36717]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:31.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:30 smithi060 ceph-mon[32745]: Deploying daemon osd.5 on smithi086 2024-03-04T12:19:31.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:30 smithi060 ceph-mon[32745]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:32.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:32 smithi060 ceph-mon[32745]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:32.488 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:32 smithi086 ceph-mon[36851]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:32.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:32 smithi063 ceph-mon[36717]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:34.213 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:34 smithi086 ceph-mon[36851]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:34.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:34 smithi060 ceph-mon[32745]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:34.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:34 smithi063 ceph-mon[36717]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:36.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:36 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:36.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:36 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:36.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:36 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:19:36.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:36 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:36.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:36 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:36.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:36 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:19:36.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:36 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:36.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:36 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:36.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:36 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:19:37.346 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:37 smithi086 ceph-mon[36851]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:37.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:37 smithi063 ceph-mon[36717]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:37.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:37 smithi060 ceph-mon[32745]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:38.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:38 smithi060 ceph-mon[32745]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:38.549 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:38 smithi086 ceph-mon[36851]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:38.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:38 smithi063 ceph-mon[36717]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:38.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.619+0000 7f6917fff700 1 -- 172.21.15.86:0/4107641307 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6930062260 con 0x7f69180a1e10 2024-03-04T12:19:38.625 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 5 on host 'smithi086' 2024-03-04T12:19:38.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.621+0000 7f6935486700 1 -- 172.21.15.86:0/4107641307 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f69180a1e10 msgr2=0x7f69180a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:19:38.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.621+0000 7f6935486700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f69180a1e10 0x7f69180a42d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f692000bcf0 tx=0x7f692000b2c0 comp rx=0 tx=0).stop 2024-03-04T12:19:38.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.622+0000 7f6935486700 1 -- 172.21.15.86:0/4107641307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6930103970 msgr2=0x7f693019f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:19:38.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.622+0000 7f6935486700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6930103970 0x7f693019f490 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6928009cd0 tx=0x7f692800bf10 comp rx=0 tx=0).stop 2024-03-04T12:19:38.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.622+0000 7f6935486700 1 -- 172.21.15.86:0/4107641307 shutdown_connections 2024-03-04T12:19:38.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.622+0000 7f6935486700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6930103970 0x7f693019f490 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:38.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.622+0000 7f6935486700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6930101e50 0x7f6930197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:38.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.622+0000 7f6935486700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f69180a1e10 0x7f69180a42d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:38.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.622+0000 7f6935486700 1 --2- 172.21.15.86:0/4107641307 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6930102fb0 0x7f6930198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:38.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.622+0000 7f6935486700 1 -- 172.21.15.86:0/4107641307 >> 172.21.15.86:0/4107641307 conn(0x7f69300fd430 msgr2=0x7f69301062e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:19:38.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.622+0000 7f6935486700 1 -- 172.21.15.86:0/4107641307 shutdown_connections 2024-03-04T12:19:38.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:19:38.622+0000 7f6935486700 1 -- 172.21.15.86:0/4107641307 wait complete. 2024-03-04T12:19:39.246 DEBUG:teuthology.orchestra.run.smithi086:osd.5> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.5.service 2024-03-04T12:19:39.248 INFO:tasks.cephadm:Deploying osd.6 on smithi060 with /dev/vg_nvme/lv_2... 2024-03-04T12:19:39.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-04T12:19:39.502 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:19:39.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:39.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:39.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:39 smithi086 ceph-mon[36851]: from='osd.5 [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T12:19:39.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:39 smithi086 ceph-mon[36851]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T12:19:39.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:39.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:39.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:39.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:39.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:39 smithi063 ceph-mon[36717]: from='osd.5 [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T12:19:39.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:39 smithi063 ceph-mon[36717]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T12:19:39.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:39.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:39.909 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:39.909 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:39.909 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:39 smithi060 ceph-mon[32745]: from='osd.5 [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T12:19:39.910 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:39 smithi060 ceph-mon[32745]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T12:19:39.910 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:39.910 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:40.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:40 smithi086 ceph-mon[36851]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T12:19:40.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:40 smithi086 ceph-mon[36851]: osdmap e36: 6 total, 5 up, 6 in 2024-03-04T12:19:40.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:40 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:40.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:40 smithi086 ceph-mon[36851]: from='osd.5 [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:19:40.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:40 smithi086 ceph-mon[36851]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:19:40.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:40 smithi086 ceph-mon[36851]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:40.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:40 smithi063 ceph-mon[36717]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T12:19:40.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:40 smithi063 ceph-mon[36717]: osdmap e36: 6 total, 5 up, 6 in 2024-03-04T12:19:40.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:40 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:40.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:40 smithi063 ceph-mon[36717]: from='osd.5 [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:19:40.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:40 smithi063 ceph-mon[36717]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:19:40.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:40 smithi063 ceph-mon[36717]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:40.915 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:40 smithi060 ceph-mon[32745]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T12:19:40.915 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:40 smithi060 ceph-mon[32745]: osdmap e36: 6 total, 5 up, 6 in 2024-03-04T12:19:40.915 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:40 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:40.915 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:40 smithi060 ceph-mon[32745]: from='osd.5 [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:19:40.915 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:40 smithi060 ceph-mon[32745]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:19:40.915 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:40 smithi060 ceph-mon[32745]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:41.213 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:19:41 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:19:41.029+0000 7f291c85a700 -1 osd.5 0 waiting for initial osdmap 2024-03-04T12:19:41.213 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:19:41 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:19:41.035+0000 7f2914645700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T12:19:41.822 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-04T12:19:41.822 INFO:teuthology.orchestra.run.smithi060.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-04T12:19:41.822 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10+0 records in 2024-03-04T12:19:41.822 INFO:teuthology.orchestra.run.smithi060.stderr:10+0 records out 2024-03-04T12:19:41.822 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00852978 s, 1.2 GB/s 2024-03-04T12:19:41.822 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping successful for: 2024-03-04T12:19:41.822 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:41 smithi060 ceph-mon[32745]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T12:19:41.822 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:41 smithi060 ceph-mon[32745]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T12:19:41.822 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:41 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:41.823 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:41 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:41.823 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:41 smithi060 ceph-mon[32745]: from='osd.5 ' entity='osd.5' 2024-03-04T12:19:41.846 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:41 smithi086 ceph-mon[36851]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T12:19:41.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:41 smithi086 ceph-mon[36851]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T12:19:41.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:41 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:41.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:41 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:41.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:41 smithi086 ceph-mon[36851]: from='osd.5 ' entity='osd.5' 2024-03-04T12:19:41.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:41 smithi063 ceph-mon[36717]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T12:19:41.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:41 smithi063 ceph-mon[36717]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T12:19:41.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:41 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:41.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:41 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:41.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:41 smithi063 ceph-mon[36717]: from='osd.5 ' entity='osd.5' 2024-03-04T12:19:42.602 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi060:vg_nvme/lv_2 2024-03-04T12:19:42.726 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:42 smithi086 ceph-mon[36851]: purged_snaps scrub starts 2024-03-04T12:19:42.726 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:42 smithi086 ceph-mon[36851]: purged_snaps scrub ok 2024-03-04T12:19:42.726 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:42 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:42.726 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:42 smithi086 ceph-mon[36851]: osd.5 [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157] boot 2024-03-04T12:19:42.726 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:42 smithi086 ceph-mon[36851]: osdmap e38: 6 total, 6 up, 6 in 2024-03-04T12:19:42.726 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:42 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:42.726 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:42 smithi086 ceph-mon[36851]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:42.846 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:19:42.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:42 smithi063 ceph-mon[36717]: purged_snaps scrub starts 2024-03-04T12:19:42.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:42 smithi063 ceph-mon[36717]: purged_snaps scrub ok 2024-03-04T12:19:42.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:42 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:42.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:42 smithi063 ceph-mon[36717]: osd.5 [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157] boot 2024-03-04T12:19:42.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:42 smithi063 ceph-mon[36717]: osdmap e38: 6 total, 6 up, 6 in 2024-03-04T12:19:42.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:42 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:42.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:42 smithi063 ceph-mon[36717]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:42.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:42 smithi060 ceph-mon[32745]: purged_snaps scrub starts 2024-03-04T12:19:42.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:42 smithi060 ceph-mon[32745]: purged_snaps scrub ok 2024-03-04T12:19:42.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:42 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:42.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:42 smithi060 ceph-mon[32745]: osd.5 [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157] boot 2024-03-04T12:19:42.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:42 smithi060 ceph-mon[32745]: osdmap e38: 6 total, 6 up, 6 in 2024-03-04T12:19:42.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:42 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:19:42.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:42 smithi060 ceph-mon[32745]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 154 MiB used, 447 GiB / 447 GiB avail 2024-03-04T12:19:43.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: osdmap e39: 6 total, 6 up, 6 in 2024-03-04T12:19:43.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: Detected new or changed devices on smithi086 2024-03-04T12:19:43.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:43.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:43.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:19:43.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:19:43.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi086 to 8540M 2024-03-04T12:19:43.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:43.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:43.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:19:43.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:43.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:19:43.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:43.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:19:43.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:44.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: osdmap e39: 6 total, 6 up, 6 in 2024-03-04T12:19:44.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: Detected new or changed devices on smithi086 2024-03-04T12:19:44.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:44.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:44.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:19:44.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:19:44.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi086 to 8540M 2024-03-04T12:19:44.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:19:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:19:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:19:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:44.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: osdmap e39: 6 total, 6 up, 6 in 2024-03-04T12:19:44.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: Detected new or changed devices on smithi086 2024-03-04T12:19:44.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:44.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:44.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:19:44.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:19:44.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi086 to 8540M 2024-03-04T12:19:44.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:44.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:19:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:19:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:19:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:19:44.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.496+0000 7fcec5654700 1 -- 172.21.15.60:0/139000005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcec0107ba0 msgr2=0x7fcec0109fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:19:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.496+0000 7fcec5654700 1 --2- 172.21.15.60:0/139000005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcec0107ba0 0x7fcec0109fb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fceb800b0a0 tx=0x7fceb800b3b0 comp rx=0 tx=0).stop 2024-03-04T12:19:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.496+0000 7fcec5654700 1 -- 172.21.15.60:0/139000005 shutdown_connections 2024-03-04T12:19:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.496+0000 7fcec5654700 1 --2- 172.21.15.60:0/139000005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcec0107ba0 0x7fcec0109fb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.496+0000 7fcec5654700 1 --2- 172.21.15.60:0/139000005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcec0102940 0x7fcec0104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.496+0000 7fcec5654700 1 --2- 172.21.15.60:0/139000005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec0105270 0x7fcec0107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.496+0000 7fcec5654700 1 -- 172.21.15.60:0/139000005 >> 172.21.15.60:0/139000005 conn(0x7fcec00fc2f0 msgr2=0x7fcec00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:19:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.496+0000 7fcec5654700 1 -- 172.21.15.60:0/139000005 shutdown_connections 2024-03-04T12:19:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.496+0000 7fcec5654700 1 -- 172.21.15.60:0/139000005 wait complete. 2024-03-04T12:19:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.497+0000 7fcec5654700 1 Processor -- start 2024-03-04T12:19:44.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.497+0000 7fcec5654700 1 -- start start 2024-03-04T12:19:44.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.497+0000 7fcec5654700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcec0102940 0x7fcec0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:19:44.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.497+0000 7fcec5654700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcec0105270 0x7fcec01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:19:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.497+0000 7fcec5654700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec0107ba0 0x7fcec019f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:19:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcec5654700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec010da80 con 0x7fcec0105270 2024-03-04T12:19:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcec5654700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec010d900 con 0x7fcec0102940 2024-03-04T12:19:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcec5654700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec010dc00 con 0x7fcec0107ba0 2024-03-04T12:19:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcebe7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcec0105270 0x7fcec01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:19:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcebe7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcec0105270 0x7fcec01985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38872/0 (socket says 172.21.15.60:38872) 2024-03-04T12:19:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcebe7fc700 1 -- 172.21.15.60:0/4262028590 learned_addr learned my addr 172.21.15.60:0/4262028590 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:19:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcebeffd700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcec0102940 0x7fcec0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:19:44.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcebe7fc700 1 -- 172.21.15.60:0/4262028590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec0107ba0 msgr2=0x7fcec019f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:19:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcebf7fe700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec0107ba0 0x7fcec019f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:19:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcebe7fc700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec0107ba0 0x7fcec019f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcebe7fc700 1 -- 172.21.15.60:0/4262028590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcec0102940 msgr2=0x7fcec0198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:19:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcebe7fc700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcec0102940 0x7fcec0198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:19:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcebe7fc700 1 -- 172.21.15.60:0/4262028590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec019fcd0 con 0x7fcec0105270 2024-03-04T12:19:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcebeffd700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcec0102940 0x7fcec0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:19:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcebe7fc700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcec0105270 0x7fcec01985b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fceb400b560 tx=0x7fceb400b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:19:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcea7fff700 1 -- 172.21.15.60:0/4262028590 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4015020 con 0x7fcec0105270 2024-03-04T12:19:44.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcec5654700 1 -- 172.21.15.60:0/4262028590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec019ffc0 con 0x7fcec0105270 2024-03-04T12:19:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcea7fff700 1 -- 172.21.15.60:0/4262028590 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fceb4011d10 con 0x7fcec0105270 2024-03-04T12:19:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcea7fff700 1 -- 172.21.15.60:0/4262028590 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb400f2e0 con 0x7fcec0105270 2024-03-04T12:19:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.498+0000 7fcec5654700 1 -- 172.21.15.60:0/4262028590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec01a0420 con 0x7fcec0105270 2024-03-04T12:19:44.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.499+0000 7fcec5654700 1 -- 172.21.15.60:0/4262028590 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec0068ae0 con 0x7fcec0105270 2024-03-04T12:19:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.500+0000 7fcea7fff700 1 -- 172.21.15.60:0/4262028590 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fceb4018020 con 0x7fcec0105270 2024-03-04T12:19:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.501+0000 7fcea7fff700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcea80a1e10 0x7fcea80a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:19:44.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.501+0000 7fcebeffd700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcea80a1e10 0x7fcea80a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:19:44.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.501+0000 7fcea7fff700 1 -- 172.21.15.60:0/4262028590 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fceb40cc780 con 0x7fcec0105270 2024-03-04T12:19:44.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.502+0000 7fcebeffd700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcea80a1e10 0x7fcea80a42d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fceb0005fd0 tx=0x7fceb0005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:19:44.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.506+0000 7fcea7fff700 1 -- 172.21.15.60:0/4262028590 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fceb401f020 con 0x7fcec0105270 2024-03-04T12:19:44.725 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:19:44.720+0000 7fcec5654700 1 -- 172.21.15.60:0/4262028590 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fcec0062260 con 0x7fcea80a1e10 2024-03-04T12:19:44.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:44 smithi060 ceph-mon[32745]: osdmap e40: 6 total, 6 up, 6 in 2024-03-04T12:19:44.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:44 smithi060 ceph-mon[32745]: pgmap v140: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-04T12:19:44.998 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:44 smithi086 ceph-mon[36851]: osdmap e40: 6 total, 6 up, 6 in 2024-03-04T12:19:44.998 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:44 smithi086 ceph-mon[36851]: pgmap v140: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-04T12:19:45.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:44 smithi063 ceph-mon[36717]: osdmap e40: 6 total, 6 up, 6 in 2024-03-04T12:19:45.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:44 smithi063 ceph-mon[36717]: pgmap v140: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-04T12:19:45.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:45 smithi060 ceph-mon[32745]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-04T12:19:45.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:45 smithi060 ceph-mon[32745]: osdmap e41: 6 total, 6 up, 6 in 2024-03-04T12:19:45.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:45 smithi060 ceph-mon[32745]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:19:45.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:45 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:19:45.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:45 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:19:45.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:45 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:46.011 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:45 smithi086 ceph-mon[36851]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-04T12:19:46.011 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:45 smithi086 ceph-mon[36851]: osdmap e41: 6 total, 6 up, 6 in 2024-03-04T12:19:46.011 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:45 smithi086 ceph-mon[36851]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:19:46.011 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:45 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:19:46.011 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:45 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:19:46.012 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:45 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:46.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:45 smithi063 ceph-mon[36717]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-04T12:19:46.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:45 smithi063 ceph-mon[36717]: osdmap e41: 6 total, 6 up, 6 in 2024-03-04T12:19:46.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:45 smithi063 ceph-mon[36717]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:19:46.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:45 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:19:46.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:45 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:19:46.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:45 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:46.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:46 smithi060 ceph-mon[32745]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T12:19:46.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:46 smithi060 ceph-mon[32745]: pgmap v143: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-04T12:19:47.001 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:46 smithi063 ceph-mon[36717]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T12:19:47.002 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:46 smithi063 ceph-mon[36717]: pgmap v143: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-04T12:19:47.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:46 smithi086 ceph-mon[36851]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T12:19:47.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:46 smithi086 ceph-mon[36851]: pgmap v143: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-04T12:19:47.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:47 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1442237683' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4794189a-f45f-4859-99de-d7578ff20141"}]: dispatch 2024-03-04T12:19:47.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:47 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1442237683' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4794189a-f45f-4859-99de-d7578ff20141"}]': finished 2024-03-04T12:19:47.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:47 smithi060 ceph-mon[32745]: osdmap e43: 7 total, 6 up, 7 in 2024-03-04T12:19:47.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:47 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:19:48.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:47 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1442237683' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4794189a-f45f-4859-99de-d7578ff20141"}]: dispatch 2024-03-04T12:19:48.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:47 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1442237683' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4794189a-f45f-4859-99de-d7578ff20141"}]': finished 2024-03-04T12:19:48.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:47 smithi086 ceph-mon[36851]: osdmap e43: 7 total, 6 up, 7 in 2024-03-04T12:19:48.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:47 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:19:48.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:47 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1442237683' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4794189a-f45f-4859-99de-d7578ff20141"}]: dispatch 2024-03-04T12:19:48.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:47 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1442237683' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4794189a-f45f-4859-99de-d7578ff20141"}]': finished 2024-03-04T12:19:48.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:47 smithi063 ceph-mon[36717]: osdmap e43: 7 total, 6 up, 7 in 2024-03-04T12:19:48.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:47 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:19:48.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:48 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2036662438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:19:48.965 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:48 smithi060 ceph-mon[32745]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-04T12:19:49.096 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:48 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2036662438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:19:49.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:48 smithi086 ceph-mon[36851]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-04T12:19:49.099 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:48 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2036662438' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:19:49.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:48 smithi063 ceph-mon[36717]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-04T12:19:50.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:50 smithi060 ceph-mon[32745]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T12:19:50.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:50 smithi060 ceph-mon[32745]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-04T12:19:50.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:50 smithi060 ceph-mon[32745]: Cluster is now healthy 2024-03-04T12:19:50.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:50 smithi086 ceph-mon[36851]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T12:19:50.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:50 smithi086 ceph-mon[36851]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-04T12:19:50.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:50 smithi086 ceph-mon[36851]: Cluster is now healthy 2024-03-04T12:19:50.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:50 smithi063 ceph-mon[36717]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T12:19:50.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:50 smithi063 ceph-mon[36717]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-04T12:19:50.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:50 smithi063 ceph-mon[36717]: Cluster is now healthy 2024-03-04T12:19:52.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:52 smithi060 ceph-mon[32745]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-04T12:19:52.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:52 smithi086 ceph-mon[36851]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-04T12:19:52.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:52 smithi063 ceph-mon[36717]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-04T12:19:54.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:54 smithi060 ceph-mon[32745]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-04T12:19:54.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:54 smithi086 ceph-mon[36851]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-04T12:19:54.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:54 smithi063 ceph-mon[36717]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-04T12:19:55.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:55 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T12:19:55.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:55 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:55.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:55 smithi060 ceph-mon[32745]: Deploying daemon osd.6 on smithi060 2024-03-04T12:19:55.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:55 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T12:19:55.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:55 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:55.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:55 smithi086 ceph-mon[36851]: Deploying daemon osd.6 on smithi060 2024-03-04T12:19:55.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:55 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T12:19:55.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:55 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:19:55.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:55 smithi063 ceph-mon[36717]: Deploying daemon osd.6 on smithi060 2024-03-04T12:19:56.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:56 smithi060 ceph-mon[32745]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T12:19:56.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:56 smithi086 ceph-mon[36851]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T12:19:56.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:56 smithi063 ceph-mon[36717]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T12:19:58.371 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:19:58 smithi060 ceph-mon[32745]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-04T12:19:58.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:19:58 smithi086 ceph-mon[36851]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-04T12:19:58.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:19:58 smithi063 ceph-mon[36717]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-04T12:20:00.280 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:00 smithi060 ceph-mon[32745]: overall HEALTH_OK 2024-03-04T12:20:00.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:00 smithi086 ceph-mon[36851]: overall HEALTH_OK 2024-03-04T12:20:00.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:00 smithi063 ceph-mon[36717]: overall HEALTH_OK 2024-03-04T12:20:01.266 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:01 smithi060 ceph-mon[32745]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-04T12:20:01.266 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:01 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:01.267 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:01 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:01.267 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:01 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:01.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:01 smithi086 ceph-mon[36851]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-04T12:20:01.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:01 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:01.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:01 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:01.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:01 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:01.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:01 smithi063 ceph-mon[36717]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-04T12:20:01.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:01 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:01.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:01 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:01.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:01 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:02.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:02 smithi060 ceph-mon[32745]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-04T12:20:02.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:02 smithi086 ceph-mon[36851]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-04T12:20:02.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:02 smithi063 ceph-mon[36717]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-04T12:20:04.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.382+0000 7fcea7fff700 1 -- 172.21.15.60:0/4262028590 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcec0062260 con 0x7fcea80a1e10 2024-03-04T12:20:04.387 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 6 on host 'smithi060' 2024-03-04T12:20:04.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.384+0000 7fcec5654700 1 -- 172.21.15.60:0/4262028590 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcea80a1e10 msgr2=0x7fcea80a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:20:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.384+0000 7fcec5654700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcea80a1e10 0x7fcea80a42d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fceb0005fd0 tx=0x7fceb0005f60 comp rx=0 tx=0).stop 2024-03-04T12:20:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.384+0000 7fcec5654700 1 -- 172.21.15.60:0/4262028590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcec0105270 msgr2=0x7fcec01985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:20:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.384+0000 7fcec5654700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcec0105270 0x7fcec01985b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fceb400b560 tx=0x7fceb400b920 comp rx=0 tx=0).stop 2024-03-04T12:20:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.384+0000 7fcec5654700 1 -- 172.21.15.60:0/4262028590 shutdown_connections 2024-03-04T12:20:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.384+0000 7fcec5654700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec0107ba0 0x7fcec019f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.384+0000 7fcec5654700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcec0102940 0x7fcec0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.384+0000 7fcec5654700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcea80a1e10 0x7fcea80a42d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.384+0000 7fcec5654700 1 --2- 172.21.15.60:0/4262028590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcec0105270 0x7fcec01985b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.384+0000 7fcec5654700 1 -- 172.21.15.60:0/4262028590 >> 172.21.15.60:0/4262028590 conn(0x7fcec00fc2f0 msgr2=0x7fcec0103280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:20:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.384+0000 7fcec5654700 1 -- 172.21.15.60:0/4262028590 shutdown_connections 2024-03-04T12:20:04.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:20:04.384+0000 7fcec5654700 1 -- 172.21.15.60:0/4262028590 wait complete. 2024-03-04T12:20:04.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:04.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:04.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:04 smithi060 ceph-mon[32745]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-04T12:20:04.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:04.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:04 smithi060 ceph-mon[32745]: from='osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T12:20:04.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:04.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:04.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:04 smithi086 ceph-mon[36851]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-04T12:20:04.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:04.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:04 smithi086 ceph-mon[36851]: from='osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T12:20:04.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:04.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:04.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:04 smithi063 ceph-mon[36717]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-04T12:20:04.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:04.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:04.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:04 smithi063 ceph-mon[36717]: from='osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T12:20:05.075 DEBUG:teuthology.orchestra.run.smithi060:osd.6> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.6.service 2024-03-04T12:20:05.078 INFO:tasks.cephadm:Deploying osd.7 on smithi063 with /dev/vg_nvme/lv_2... 2024-03-04T12:20:05.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-04T12:20:05.354 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T12:20:06.583 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:06 smithi063 ceph-mon[36717]: from='osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T12:20:06.583 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:06 smithi063 ceph-mon[36717]: osdmap e44: 7 total, 6 up, 7 in 2024-03-04T12:20:06.583 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:06 smithi063 ceph-mon[36717]: from='osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-03-04T12:20:06.583 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:06 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:20:06.583 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:06 smithi063 ceph-mon[36717]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-04T12:20:06.626 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:06 smithi060 ceph-mon[32745]: from='osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T12:20:06.626 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:06 smithi060 ceph-mon[32745]: osdmap e44: 7 total, 6 up, 7 in 2024-03-04T12:20:06.627 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:06 smithi060 ceph-mon[32745]: from='osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-03-04T12:20:06.627 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:06 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:20:06.627 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:06 smithi060 ceph-mon[32745]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-04T12:20:06.627 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:20:06 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:20:06.370+0000 7ffaa294f700 -1 osd.6 0 waiting for initial osdmap 2024-03-04T12:20:06.627 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:20:06 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:20:06.382+0000 7ffa9e742700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T12:20:06.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:06 smithi086 ceph-mon[36851]: from='osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T12:20:06.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:06 smithi086 ceph-mon[36851]: osdmap e44: 7 total, 6 up, 7 in 2024-03-04T12:20:06.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:06 smithi086 ceph-mon[36851]: from='osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-03-04T12:20:06.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:06 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:20:06.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:06 smithi086 ceph-mon[36851]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-04T12:20:07.432 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:07 smithi063 ceph-mon[36717]: from='osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-03-04T12:20:07.432 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:07 smithi063 ceph-mon[36717]: osdmap e45: 7 total, 6 up, 7 in 2024-03-04T12:20:07.432 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:07 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:20:07.432 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:07 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:20:07.524 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-04T12:20:07.524 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-04T12:20:07.524 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-03-04T12:20:07.524 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-03-04T12:20:07.525 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011374 s, 922 MB/s 2024-03-04T12:20:07.525 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-03-04T12:20:07.644 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:07 smithi060 ceph-mon[32745]: from='osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-03-04T12:20:07.644 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:07 smithi060 ceph-mon[32745]: osdmap e45: 7 total, 6 up, 7 in 2024-03-04T12:20:07.644 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:07 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:20:07.645 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:07 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:20:07.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:07 smithi086 ceph-mon[36851]: from='osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-03-04T12:20:07.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:07 smithi086 ceph-mon[36851]: osdmap e45: 7 total, 6 up, 7 in 2024-03-04T12:20:07.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:07 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:20:07.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:07 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:20:08.200 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi063:vg_nvme/lv_2 2024-03-04T12:20:08.458 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T12:20:08.580 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:08 smithi063 ceph-mon[36717]: purged_snaps scrub starts 2024-03-04T12:20:08.580 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:08 smithi063 ceph-mon[36717]: purged_snaps scrub ok 2024-03-04T12:20:08.580 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:08 smithi063 ceph-mon[36717]: osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459] boot 2024-03-04T12:20:08.580 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:08 smithi063 ceph-mon[36717]: osdmap e46: 7 total, 7 up, 7 in 2024-03-04T12:20:08.580 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:08 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:20:08.580 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:08 smithi063 ceph-mon[36717]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-04T12:20:08.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:08 smithi060 ceph-mon[32745]: purged_snaps scrub starts 2024-03-04T12:20:08.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:08 smithi060 ceph-mon[32745]: purged_snaps scrub ok 2024-03-04T12:20:08.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:08 smithi060 ceph-mon[32745]: osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459] boot 2024-03-04T12:20:08.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:08 smithi060 ceph-mon[32745]: osdmap e46: 7 total, 7 up, 7 in 2024-03-04T12:20:08.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:08 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:20:08.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:08 smithi060 ceph-mon[32745]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-04T12:20:08.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:08 smithi086 ceph-mon[36851]: purged_snaps scrub starts 2024-03-04T12:20:08.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:08 smithi086 ceph-mon[36851]: purged_snaps scrub ok 2024-03-04T12:20:08.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:08 smithi086 ceph-mon[36851]: osd.6 [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459] boot 2024-03-04T12:20:08.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:08 smithi086 ceph-mon[36851]: osdmap e46: 7 total, 7 up, 7 in 2024-03-04T12:20:08.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:08 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:20:08.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:08 smithi086 ceph-mon[36851]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-04T12:20:09.597 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:09 smithi063 ceph-mon[36717]: osdmap e47: 7 total, 7 up, 7 in 2024-03-04T12:20:09.597 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:09 smithi063 ceph-mon[36717]: Detected new or changed devices on smithi060 2024-03-04T12:20:09.598 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:09.598 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:09.598 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:09.598 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:09.598 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:09.598 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:09 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi060 to 5697M 2024-03-04T12:20:09.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:09.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:09.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:09.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:09.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:09 smithi060 ceph-mon[32745]: osdmap e47: 7 total, 7 up, 7 in 2024-03-04T12:20:09.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:09 smithi060 ceph-mon[32745]: Detected new or changed devices on smithi060 2024-03-04T12:20:09.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:09.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:09.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:09.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:09.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:09.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:09 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi060 to 5697M 2024-03-04T12:20:09.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:09.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:09.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:09.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:09.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:09 smithi086 ceph-mon[36851]: osdmap e47: 7 total, 7 up, 7 in 2024-03-04T12:20:09.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:09 smithi086 ceph-mon[36851]: Detected new or changed devices on smithi060 2024-03-04T12:20:09.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:09.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:09.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:09.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:09.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:09.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:09 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi060 to 5697M 2024-03-04T12:20:09.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:09.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:09.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:09.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.168+0000 7f4ab56f4700 1 -- 172.21.15.63:0/1619546160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ab0102fb0 msgr2=0x7f4ab0103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.168+0000 7f4ab56f4700 1 --2- 172.21.15.63:0/1619546160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ab0102fb0 0x7f4ab0103430 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4009990 tx=0x7f4aa4009ca0 comp rx=0 tx=0).stop 2024-03-04T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.168+0000 7f4ab56f4700 1 -- 172.21.15.63:0/1619546160 shutdown_connections 2024-03-04T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.169+0000 7f4ab56f4700 1 --2- 172.21.15.63:0/1619546160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ab0103970 0x7f4ab010a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.169+0000 7f4ab56f4700 1 --2- 172.21.15.63:0/1619546160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ab0101e50 0x7f4ab0102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.169+0000 7f4ab56f4700 1 --2- 172.21.15.63:0/1619546160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ab0102fb0 0x7f4ab0103430 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:10.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.169+0000 7f4ab56f4700 1 -- 172.21.15.63:0/1619546160 >> 172.21.15.63:0/1619546160 conn(0x7f4ab00fd430 msgr2=0x7f4ab00ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:20:10.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.169+0000 7f4ab56f4700 1 -- 172.21.15.63:0/1619546160 shutdown_connections 2024-03-04T12:20:10.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.169+0000 7f4ab56f4700 1 -- 172.21.15.63:0/1619546160 wait complete. 2024-03-04T12:20:10.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.169+0000 7f4ab56f4700 1 Processor -- start 2024-03-04T12:20:10.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4ab56f4700 1 -- start start 2024-03-04T12:20:10.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4ab56f4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ab0101e50 0x7f4ab01980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:20:10.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4ab56f4700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ab0102fb0 0x7f4ab01985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:20:10.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4ab56f4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ab0103970 0x7f4ab019f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:20:10.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4ab56f4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab010daa0 con 0x7f4ab0103970 2024-03-04T12:20:10.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4ab56f4700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab010d920 con 0x7f4ab0102fb0 2024-03-04T12:20:10.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4ab56f4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab010dc20 con 0x7f4ab0101e50 2024-03-04T12:20:10.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4aae7fc700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ab0102fb0 0x7f4ab01985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:20:10.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4aae7fc700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ab0102fb0 0x7f4ab01985f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34584/0 (socket says 172.21.15.63:34584) 2024-03-04T12:20:10.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4aae7fc700 1 -- 172.21.15.63:0/2654463596 learned_addr learned my addr 172.21.15.63:0/2654463596 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-04T12:20:10.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4aaeffd700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ab0101e50 0x7f4ab01980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:20:10.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4aaf7fe700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ab0103970 0x7f4ab019f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:20:10.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4aae7fc700 1 -- 172.21.15.63:0/2654463596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ab0101e50 msgr2=0x7f4ab01980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:20:10.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4aae7fc700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ab0101e50 0x7f4ab01980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:10.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4aae7fc700 1 -- 172.21.15.63:0/2654463596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ab0103970 msgr2=0x7f4ab019f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:20:10.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4aae7fc700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ab0103970 0x7f4ab019f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:10.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4aae7fc700 1 -- 172.21.15.63:0/2654463596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab019fd10 con 0x7f4ab0102fb0 2024-03-04T12:20:10.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4aaeffd700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ab0101e50 0x7f4ab01980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:20:10.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.170+0000 7f4aae7fc700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ab0102fb0 0x7f4ab01985f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400bff0 tx=0x7f4aa4003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:20:10.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.171+0000 7f4a97fff700 1 -- 172.21.15.63:0/2654463596 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4003e40 con 0x7f4ab0102fb0 2024-03-04T12:20:10.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.171+0000 7f4ab56f4700 1 -- 172.21.15.63:0/2654463596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab019ffa0 con 0x7f4ab0102fb0 2024-03-04T12:20:10.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.171+0000 7f4ab56f4700 1 -- 172.21.15.63:0/2654463596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab01a03c0 con 0x7f4ab0102fb0 2024-03-04T12:20:10.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.171+0000 7f4a97fff700 1 -- 172.21.15.63:0/2654463596 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4aa4004d10 con 0x7f4ab0102fb0 2024-03-04T12:20:10.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.171+0000 7f4a97fff700 1 -- 172.21.15.63:0/2654463596 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa40056c0 con 0x7f4ab0102fb0 2024-03-04T12:20:10.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.172+0000 7f4ab56f4700 1 -- 172.21.15.63:0/2654463596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab0068ae0 con 0x7f4ab0102fb0 2024-03-04T12:20:10.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.173+0000 7f4a97fff700 1 -- 172.21.15.63:0/2654463596 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f4aa4004150 con 0x7f4ab0102fb0 2024-03-04T12:20:10.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.173+0000 7f4a97fff700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4a980a1e90 0x7f4a980a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:20:10.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.174+0000 7f4a97fff700 1 -- 172.21.15.63:0/2654463596 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f4aa40ce650 con 0x7f4ab0102fb0 2024-03-04T12:20:10.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.174+0000 7f4aaeffd700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4a980a1e90 0x7f4a980a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:20:10.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.174+0000 7f4aaeffd700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4a980a1e90 0x7f4a980a4350 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0005950 tx=0x7f4aa0005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:20:10.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.177+0000 7f4a97fff700 1 -- 172.21.15.63:0/2654463596 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4aa4013070 con 0x7f4ab0102fb0 2024-03-04T12:20:10.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:10.388+0000 7f4ab56f4700 1 -- 172.21.15.63:0/2654463596 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ab0062320 con 0x7f4a980a1e90 2024-03-04T12:20:10.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:10 smithi063 ceph-mon[36717]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:10.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:10 smithi060 ceph-mon[32745]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:10.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:10 smithi086 ceph-mon[36851]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:11.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:11 smithi063 ceph-mon[36717]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:20:11.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:20:11.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:20:11.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:11.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:11 smithi060 ceph-mon[32745]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:20:11.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:20:11.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:20:11.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:11.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:11 smithi086 ceph-mon[36851]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:20:11.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:20:11.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:20:11.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:12.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:12 smithi060 ceph-mon[32745]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:12.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:12 smithi086 ceph-mon[36851]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:12.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:12 smithi063 ceph-mon[36717]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:13.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:13 smithi060 ceph-mon[32745]: from='client.? 172.21.15.63:0/882385685' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86f9c999-69b0-4c33-b39c-5c307ec78e58"}]: dispatch 2024-03-04T12:20:13.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:13 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86f9c999-69b0-4c33-b39c-5c307ec78e58"}]: dispatch 2024-03-04T12:20:13.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:13 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86f9c999-69b0-4c33-b39c-5c307ec78e58"}]': finished 2024-03-04T12:20:13.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:13 smithi060 ceph-mon[32745]: osdmap e48: 8 total, 7 up, 8 in 2024-03-04T12:20:13.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:13 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:13.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:13 smithi086 ceph-mon[36851]: from='client.? 172.21.15.63:0/882385685' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86f9c999-69b0-4c33-b39c-5c307ec78e58"}]: dispatch 2024-03-04T12:20:13.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:13 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86f9c999-69b0-4c33-b39c-5c307ec78e58"}]: dispatch 2024-03-04T12:20:13.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:13 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86f9c999-69b0-4c33-b39c-5c307ec78e58"}]': finished 2024-03-04T12:20:13.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:13 smithi086 ceph-mon[36851]: osdmap e48: 8 total, 7 up, 8 in 2024-03-04T12:20:13.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:13 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:13.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:13 smithi063 ceph-mon[36717]: from='client.? 172.21.15.63:0/882385685' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86f9c999-69b0-4c33-b39c-5c307ec78e58"}]: dispatch 2024-03-04T12:20:13.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:13 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "86f9c999-69b0-4c33-b39c-5c307ec78e58"}]: dispatch 2024-03-04T12:20:13.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:13 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "86f9c999-69b0-4c33-b39c-5c307ec78e58"}]': finished 2024-03-04T12:20:13.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:13 smithi063 ceph-mon[36717]: osdmap e48: 8 total, 7 up, 8 in 2024-03-04T12:20:13.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:13 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:14.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:14 smithi060 ceph-mon[32745]: from='client.? 172.21.15.63:0/1555933503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:20:14.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:14 smithi060 ceph-mon[32745]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:14.748 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:14 smithi086 ceph-mon[36851]: from='client.? 172.21.15.63:0/1555933503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:20:14.748 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:14 smithi086 ceph-mon[36851]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:14.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:14 smithi063 ceph-mon[36717]: from='client.? 172.21.15.63:0/1555933503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:20:14.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:14 smithi063 ceph-mon[36717]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:16.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:16 smithi086 ceph-mon[36851]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:16.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:16 smithi063 ceph-mon[36717]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:16.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:16 smithi060 ceph-mon[32745]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:18.499 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:18 smithi063 ceph-mon[36717]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:18.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:18 smithi086 ceph-mon[36851]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:18.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:18 smithi060 ceph-mon[32745]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:20.567 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:20 smithi063 ceph-mon[36717]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:20.596 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:20 smithi086 ceph-mon[36851]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:20.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:20 smithi060 ceph-mon[32745]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:21.551 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:21 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T12:20:21.551 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:21 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:21.551 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:21 smithi063 ceph-mon[36717]: Deploying daemon osd.7 on smithi063 2024-03-04T12:20:21.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:21 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T12:20:21.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:21 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:21.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:21 smithi086 ceph-mon[36851]: Deploying daemon osd.7 on smithi063 2024-03-04T12:20:21.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:21 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T12:20:21.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:21 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:21.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:21 smithi060 ceph-mon[32745]: Deploying daemon osd.7 on smithi063 2024-03-04T12:20:22.553 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:22 smithi063 ceph-mon[36717]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:22.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:22 smithi086 ceph-mon[36851]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:22.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:22 smithi060 ceph-mon[32745]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:24.539 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:24 smithi063 ceph-mon[36717]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:24.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:24 smithi086 ceph-mon[36851]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:24.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:24 smithi060 ceph-mon[32745]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:26.303 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:26 smithi063 ceph-mon[36717]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:26.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:26 smithi086 ceph-mon[36851]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:26.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:26 smithi060 ceph-mon[32745]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:28.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:28.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:28.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:28.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:28.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:28.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:28.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:28.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:28.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:29.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:28 smithi060 ceph-mon[32745]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:29.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:28 smithi086 ceph-mon[36851]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:29.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:28 smithi063 ceph-mon[36717]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:30.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.361+0000 7f4a97fff700 1 -- 172.21.15.63:0/2654463596 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4ab0062320 con 0x7f4a980a1e90 2024-03-04T12:20:30.367 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 7 on host 'smithi063' 2024-03-04T12:20:30.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.363+0000 7f4ab56f4700 1 -- 172.21.15.63:0/2654463596 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4a980a1e90 msgr2=0x7f4a980a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:20:30.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.363+0000 7f4ab56f4700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4a980a1e90 0x7f4a980a4350 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0005950 tx=0x7f4aa0005f50 comp rx=0 tx=0).stop 2024-03-04T12:20:30.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.363+0000 7f4ab56f4700 1 -- 172.21.15.63:0/2654463596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ab0102fb0 msgr2=0x7f4ab01985f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:20:30.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.363+0000 7f4ab56f4700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ab0102fb0 0x7f4ab01985f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400bff0 tx=0x7f4aa4003680 comp rx=0 tx=0).stop 2024-03-04T12:20:30.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.363+0000 7f4ab56f4700 1 -- 172.21.15.63:0/2654463596 shutdown_connections 2024-03-04T12:20:30.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.363+0000 7f4ab56f4700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4a980a1e90 0x7f4a980a4350 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:30.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.363+0000 7f4ab56f4700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ab0103970 0x7f4ab019f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:30.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.363+0000 7f4ab56f4700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ab0101e50 0x7f4ab01980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:30.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.363+0000 7f4ab56f4700 1 --2- 172.21.15.63:0/2654463596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ab0102fb0 0x7f4ab01985f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:30.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.363+0000 7f4ab56f4700 1 -- 172.21.15.63:0/2654463596 >> 172.21.15.63:0/2654463596 conn(0x7f4ab00fd430 msgr2=0x7f4ab0106340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:20:30.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.363+0000 7f4ab56f4700 1 -- 172.21.15.63:0/2654463596 shutdown_connections 2024-03-04T12:20:30.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:20:30.363+0000 7f4ab56f4700 1 -- 172.21.15.63:0/2654463596 wait complete. 2024-03-04T12:20:30.558 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:30 smithi063 ceph-mon[36717]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:30.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:30 smithi086 ceph-mon[36851]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:30.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:30 smithi060 ceph-mon[32745]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:31.308 DEBUG:teuthology.orchestra.run.smithi063:osd.7> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.7.service 2024-03-04T12:20:31.310 INFO:tasks.cephadm:Deploying osd.8 on smithi086 with /dev/vg_nvme/lv_2... 2024-03-04T12:20:31.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-04T12:20:31.557 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:20:31.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:31 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:31.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:31 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:31.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:31 smithi086 ceph-mon[36851]: from='osd.7 [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T12:20:31.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:31 smithi086 ceph-mon[36851]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T12:20:31.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:31 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:31.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:31 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:31.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:31 smithi063 ceph-mon[36717]: from='osd.7 [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T12:20:31.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:31 smithi063 ceph-mon[36717]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T12:20:31.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:31 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:31.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:31 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:31.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:31 smithi060 ceph-mon[32745]: from='osd.7 [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T12:20:31.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:31 smithi060 ceph-mon[32745]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T12:20:32.373 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:32 smithi063 ceph-mon[36717]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T12:20:32.373 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:32 smithi063 ceph-mon[36717]: from='osd.7 [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:20:32.374 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:32 smithi063 ceph-mon[36717]: osdmap e49: 8 total, 7 up, 8 in 2024-03-04T12:20:32.374 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:32.374 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:32 smithi063 ceph-mon[36717]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:20:32.374 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:32.374 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:32.374 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:32 smithi063 ceph-mon[36717]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:32.374 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:20:32 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:20:32.360+0000 7f79d65b1700 -1 osd.7 0 waiting for initial osdmap 2024-03-04T12:20:32.374 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:20:32 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:20:32.368+0000 7f79d23a4700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T12:20:32.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:32 smithi060 ceph-mon[32745]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T12:20:32.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:32 smithi060 ceph-mon[32745]: from='osd.7 [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:20:32.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:32 smithi060 ceph-mon[32745]: osdmap e49: 8 total, 7 up, 8 in 2024-03-04T12:20:32.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:32.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:32 smithi060 ceph-mon[32745]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:20:32.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:32.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:32.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:32 smithi060 ceph-mon[32745]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:32.686 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:32 smithi086 ceph-mon[36851]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T12:20:32.686 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:32 smithi086 ceph-mon[36851]: from='osd.7 [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:20:32.687 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:32 smithi086 ceph-mon[36851]: osdmap e49: 8 total, 7 up, 8 in 2024-03-04T12:20:32.687 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:32.687 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:32 smithi086 ceph-mon[36851]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:20:32.687 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:32.687 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:32.687 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:32 smithi086 ceph-mon[36851]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-04T12:20:33.440 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:33 smithi063 ceph-mon[36717]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-04T12:20:33.440 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:33 smithi063 ceph-mon[36717]: osdmap e50: 8 total, 7 up, 8 in 2024-03-04T12:20:33.440 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:33 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:33.440 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:33 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:33.628 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:33 smithi086 ceph-mon[36851]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-04T12:20:33.628 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:33 smithi086 ceph-mon[36851]: osdmap e50: 8 total, 7 up, 8 in 2024-03-04T12:20:33.628 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:33 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:33.628 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:33 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-04T12:20:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-04T12:20:33.628 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-03-04T12:20:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-03-04T12:20:33.629 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114518 s, 916 MB/s 2024-03-04T12:20:33.629 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-03-04T12:20:33.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:33 smithi060 ceph-mon[32745]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-04T12:20:33.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:33 smithi060 ceph-mon[32745]: osdmap e50: 8 total, 7 up, 8 in 2024-03-04T12:20:33.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:33 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:33.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:33 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:34.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_2 2024-03-04T12:20:34.535 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:20:34.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:34 smithi086 ceph-mon[36851]: purged_snaps scrub starts 2024-03-04T12:20:34.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:34 smithi086 ceph-mon[36851]: purged_snaps scrub ok 2024-03-04T12:20:34.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:34 smithi086 ceph-mon[36851]: osd.7 [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290] boot 2024-03-04T12:20:34.584 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:34 smithi086 ceph-mon[36851]: osdmap e51: 8 total, 8 up, 8 in 2024-03-04T12:20:34.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:34 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:34.585 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:34 smithi086 ceph-mon[36851]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:34.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:34 smithi063 ceph-mon[36717]: purged_snaps scrub starts 2024-03-04T12:20:34.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:34 smithi063 ceph-mon[36717]: purged_snaps scrub ok 2024-03-04T12:20:34.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:34 smithi063 ceph-mon[36717]: osd.7 [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290] boot 2024-03-04T12:20:34.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:34 smithi063 ceph-mon[36717]: osdmap e51: 8 total, 8 up, 8 in 2024-03-04T12:20:34.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:34 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:34.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:34 smithi063 ceph-mon[36717]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:34.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:34 smithi060 ceph-mon[32745]: purged_snaps scrub starts 2024-03-04T12:20:34.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:34 smithi060 ceph-mon[32745]: purged_snaps scrub ok 2024-03-04T12:20:34.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:34 smithi060 ceph-mon[32745]: osd.7 [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290] boot 2024-03-04T12:20:34.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:34 smithi060 ceph-mon[32745]: osdmap e51: 8 total, 8 up, 8 in 2024-03-04T12:20:34.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:34 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:20:34.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:34 smithi060 ceph-mon[32745]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:35.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: osdmap e52: 8 total, 8 up, 8 in 2024-03-04T12:20:35.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: Detected new or changed devices on smithi063 2024-03-04T12:20:35.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:35.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:35.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:35.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi063 to 5693M 2024-03-04T12:20:35.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:35.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:35.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:35.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:35.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:35.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.679 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: osdmap e52: 8 total, 8 up, 8 in 2024-03-04T12:20:35.679 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: Detected new or changed devices on smithi063 2024-03-04T12:20:35.679 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.679 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.679 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:35.680 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:35.680 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:35.680 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi063 to 5693M 2024-03-04T12:20:35.680 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.680 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:35.680 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:35.680 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.680 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:35.680 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:35.680 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:35.680 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: osdmap e52: 8 total, 8 up, 8 in 2024-03-04T12:20:35.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: Detected new or changed devices on smithi063 2024-03-04T12:20:35.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:35.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:35.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:20:35.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi063 to 5693M 2024-03-04T12:20:35.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:35.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:35.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:35.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:35.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:35.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:35.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:36.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.246+0000 7f132c2d9700 1 -- 172.21.15.86:0/1319104866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1324101f50 msgr2=0x7f13241023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.246+0000 7f132c2d9700 1 --2- 172.21.15.86:0/1319104866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1324101f50 0x7f13241023b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1314009990 tx=0x7f1314009ca0 comp rx=0 tx=0).stop 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.246+0000 7f132c2d9700 1 -- 172.21.15.86:0/1319104866 shutdown_connections 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.246+0000 7f132c2d9700 1 --2- 172.21.15.86:0/1319104866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f13241028f0 0x7f13241091a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.246+0000 7f132c2d9700 1 --2- 172.21.15.86:0/1319104866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1324100d50 0x7f1324101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.246+0000 7f132c2d9700 1 --2- 172.21.15.86:0/1319104866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1324101f50 0x7f13241023b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.246+0000 7f132c2d9700 1 -- 172.21.15.86:0/1319104866 >> 172.21.15.86:0/1319104866 conn(0x7f13240fc310 msgr2=0x7f13240fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.246+0000 7f132c2d9700 1 -- 172.21.15.86:0/1319104866 shutdown_connections 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.246+0000 7f132c2d9700 1 -- 172.21.15.86:0/1319104866 wait complete. 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.246+0000 7f132c2d9700 1 Processor -- start 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.246+0000 7f132c2d9700 1 -- start start 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132c2d9700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1324100d50 0x7f132419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132c2d9700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1324101f50 0x7f132419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:20:36.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132c2d9700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13241028f0 0x7f13241a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132c2d9700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132410c930 con 0x7f1324100d50 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132c2d9700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132410c7b0 con 0x7f1324101f50 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132c2d9700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132410cab0 con 0x7f13241028f0 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132a876700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13241028f0 0x7f13241a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132a876700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13241028f0 0x7f13241a1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:42682/0 (socket says 172.21.15.86:42682) 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132a876700 1 -- 172.21.15.86:0/3660367892 learned_addr learned my addr 172.21.15.86:0/3660367892 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f1329874700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1324101f50 0x7f132419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132a075700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1324100d50 0x7f132419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132a876700 1 -- 172.21.15.86:0/3660367892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1324101f50 msgr2=0x7f132419a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132a876700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1324101f50 0x7f132419a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132a876700 1 -- 172.21.15.86:0/3660367892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1324100d50 msgr2=0x7f132419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132a876700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1324100d50 0x7f132419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132a876700 1 -- 172.21.15.86:0/3660367892 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13241a1ec0 con 0x7f13241028f0 2024-03-04T12:20:36.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f1329874700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1324101f50 0x7f132419a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:20:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132a876700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13241028f0 0x7f13241a1820 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f131c007f10 tx=0x7f131c00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:20:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f131b7fe700 1 -- 172.21.15.86:0/3660367892 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c004bb0 con 0x7f13241028f0 2024-03-04T12:20:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f131b7fe700 1 -- 172.21.15.86:0/3660367892 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f131c004d50 con 0x7f13241028f0 2024-03-04T12:20:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132c2d9700 1 -- 172.21.15.86:0/3660367892 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13241a21b0 con 0x7f13241028f0 2024-03-04T12:20:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f131b7fe700 1 -- 172.21.15.86:0/3660367892 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c005a60 con 0x7f13241028f0 2024-03-04T12:20:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.247+0000 7f132c2d9700 1 -- 172.21.15.86:0/3660367892 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13241a26a0 con 0x7f13241028f0 2024-03-04T12:20:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.248+0000 7f132c2d9700 1 -- 172.21.15.86:0/3660367892 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1324068ae0 con 0x7f13241028f0 2024-03-04T12:20:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.249+0000 7f131b7fe700 1 -- 172.21.15.86:0/3660367892 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f131c01a020 con 0x7f13241028f0 2024-03-04T12:20:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.250+0000 7f131b7fe700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f13100a1e10 0x7f13100a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:20:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.250+0000 7f131b7fe700 1 -- 172.21.15.86:0/3660367892 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f131c0cd0b0 con 0x7f13241028f0 2024-03-04T12:20:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.250+0000 7f132a075700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f13100a1e10 0x7f13100a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:20:36.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.251+0000 7f132a075700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f13100a1e10 0x7f13100a42d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1320005950 tx=0x7f132000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:20:36.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.253+0000 7f131b7fe700 1 -- 172.21.15.86:0/3660367892 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f131c010070 con 0x7f13241028f0 2024-03-04T12:20:36.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:36.459+0000 7f132c2d9700 1 -- 172.21.15.86:0/3660367892 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f13240fe490 con 0x7f13100a1e10 2024-03-04T12:20:36.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:36 smithi060 ceph-mon[32745]: osdmap e53: 8 total, 8 up, 8 in 2024-03-04T12:20:36.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:36 smithi060 ceph-mon[32745]: pgmap v179: 1 pgs: 1 unknown; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:36.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:36 smithi086 ceph-mon[36851]: osdmap e53: 8 total, 8 up, 8 in 2024-03-04T12:20:36.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:36 smithi086 ceph-mon[36851]: pgmap v179: 1 pgs: 1 unknown; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:36.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:36 smithi063 ceph-mon[36717]: osdmap e53: 8 total, 8 up, 8 in 2024-03-04T12:20:36.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:36 smithi063 ceph-mon[36717]: pgmap v179: 1 pgs: 1 unknown; 577 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:37.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:37 smithi060 ceph-mon[32745]: osdmap e54: 8 total, 8 up, 8 in 2024-03-04T12:20:37.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:37 smithi060 ceph-mon[32745]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:20:37.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:37 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:20:37.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:37 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:20:37.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:37 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:37.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:37 smithi086 ceph-mon[36851]: osdmap e54: 8 total, 8 up, 8 in 2024-03-04T12:20:37.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:37 smithi086 ceph-mon[36851]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:20:37.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:37 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:20:37.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:37 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:20:37.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:37 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:37.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:37 smithi063 ceph-mon[36717]: osdmap e54: 8 total, 8 up, 8 in 2024-03-04T12:20:37.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:37 smithi063 ceph-mon[36717]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:20:37.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:37 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:20:37.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:37 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:20:37.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:37 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:38.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:38 smithi060 ceph-mon[32745]: osdmap e55: 8 total, 8 up, 8 in 2024-03-04T12:20:38.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:38 smithi060 ceph-mon[32745]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:38.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:38 smithi086 ceph-mon[36851]: osdmap e55: 8 total, 8 up, 8 in 2024-03-04T12:20:38.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:38 smithi086 ceph-mon[36851]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:38.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:38 smithi063 ceph-mon[36717]: osdmap e55: 8 total, 8 up, 8 in 2024-03-04T12:20:38.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:38 smithi063 ceph-mon[36717]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:39.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:39 smithi060 ceph-mon[32745]: from='client.? 172.21.15.86:0/731983691' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5848de7e-e01b-4ae8-a060-ead59bf03d90"}]: dispatch 2024-03-04T12:20:39.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:39 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5848de7e-e01b-4ae8-a060-ead59bf03d90"}]: dispatch 2024-03-04T12:20:39.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:39 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5848de7e-e01b-4ae8-a060-ead59bf03d90"}]': finished 2024-03-04T12:20:39.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:39 smithi060 ceph-mon[32745]: osdmap e56: 9 total, 8 up, 9 in 2024-03-04T12:20:39.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:39 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:39.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:39 smithi086 ceph-mon[36851]: from='client.? 172.21.15.86:0/731983691' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5848de7e-e01b-4ae8-a060-ead59bf03d90"}]: dispatch 2024-03-04T12:20:39.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:39 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5848de7e-e01b-4ae8-a060-ead59bf03d90"}]: dispatch 2024-03-04T12:20:39.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:39 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5848de7e-e01b-4ae8-a060-ead59bf03d90"}]': finished 2024-03-04T12:20:39.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:39 smithi086 ceph-mon[36851]: osdmap e56: 9 total, 8 up, 9 in 2024-03-04T12:20:39.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:39 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:39.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:39 smithi063 ceph-mon[36717]: from='client.? 172.21.15.86:0/731983691' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5848de7e-e01b-4ae8-a060-ead59bf03d90"}]: dispatch 2024-03-04T12:20:39.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:39 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5848de7e-e01b-4ae8-a060-ead59bf03d90"}]: dispatch 2024-03-04T12:20:39.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:39 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5848de7e-e01b-4ae8-a060-ead59bf03d90"}]': finished 2024-03-04T12:20:39.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:39 smithi063 ceph-mon[36717]: osdmap e56: 9 total, 8 up, 9 in 2024-03-04T12:20:39.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:39 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:40.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:40 smithi060 ceph-mon[32745]: from='client.? 172.21.15.86:0/3907417788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:20:40.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:40 smithi060 ceph-mon[32745]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:40.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:40 smithi086 ceph-mon[36851]: from='client.? 172.21.15.86:0/3907417788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:20:40.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:40 smithi086 ceph-mon[36851]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:40.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:40 smithi063 ceph-mon[36717]: from='client.? 172.21.15.86:0/3907417788' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:20:40.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:40 smithi063 ceph-mon[36717]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:42.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:42 smithi086 ceph-mon[36851]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:42.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:42 smithi063 ceph-mon[36717]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:42.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:42 smithi060 ceph-mon[32745]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:44.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:44 smithi086 ceph-mon[36851]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:44.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:44 smithi063 ceph-mon[36717]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:44.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:44 smithi060 ceph-mon[32745]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:46.514 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:46 smithi086 ceph-mon[36851]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:46.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:46 smithi063 ceph-mon[36717]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:46.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:46 smithi060 ceph-mon[32745]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:47.425 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:47 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-04T12:20:47.425 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:47 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:47.425 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:47 smithi086 ceph-mon[36851]: Deploying daemon osd.8 on smithi086 2024-03-04T12:20:47.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:47 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-04T12:20:47.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:47 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:47.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:47 smithi063 ceph-mon[36717]: Deploying daemon osd.8 on smithi086 2024-03-04T12:20:47.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:47 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-04T12:20:47.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:47 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:47.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:47 smithi060 ceph-mon[32745]: Deploying daemon osd.8 on smithi086 2024-03-04T12:20:48.517 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:48 smithi086 ceph-mon[36851]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:48.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:48 smithi063 ceph-mon[36717]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:48.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:48 smithi060 ceph-mon[32745]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:50.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:50 smithi086 ceph-mon[36851]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:50.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:50 smithi063 ceph-mon[36717]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:50.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:50 smithi060 ceph-mon[32745]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:52.580 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:52 smithi086 ceph-mon[36851]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:52.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:52 smithi063 ceph-mon[36717]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:52.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:52 smithi060 ceph-mon[32745]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:53.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:53.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:53.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:53.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:53.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:53.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:53.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:53.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:53.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:55.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:55 smithi086 ceph-mon[36851]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:55.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:55 smithi063 ceph-mon[36717]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:55.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:55 smithi060 ceph-mon[32745]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:56.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.123+0000 7f131b7fe700 1 -- 172.21.15.86:0/3660367892 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f13240fe490 con 0x7f13100a1e10 2024-03-04T12:20:56.129 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 8 on host 'smithi086' 2024-03-04T12:20:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.125+0000 7f132c2d9700 1 -- 172.21.15.86:0/3660367892 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f13100a1e10 msgr2=0x7f13100a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:20:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.125+0000 7f132c2d9700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f13100a1e10 0x7f13100a42d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1320005950 tx=0x7f132000a300 comp rx=0 tx=0).stop 2024-03-04T12:20:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.125+0000 7f132c2d9700 1 -- 172.21.15.86:0/3660367892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13241028f0 msgr2=0x7f13241a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:20:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.125+0000 7f132c2d9700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13241028f0 0x7f13241a1820 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f131c007f10 tx=0x7f131c00d3b0 comp rx=0 tx=0).stop 2024-03-04T12:20:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.126+0000 7f132c2d9700 1 -- 172.21.15.86:0/3660367892 shutdown_connections 2024-03-04T12:20:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.126+0000 7f132c2d9700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13241028f0 0x7f13241a1820 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.126+0000 7f132c2d9700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f13100a1e10 0x7f13100a42d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.126+0000 7f132c2d9700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1324100d50 0x7f132419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.126+0000 7f132c2d9700 1 --2- 172.21.15.86:0/3660367892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1324101f50 0x7f132419a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:20:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.126+0000 7f132c2d9700 1 -- 172.21.15.86:0/3660367892 >> 172.21.15.86:0/3660367892 conn(0x7f13240fc310 msgr2=0x7f13240fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:20:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.126+0000 7f132c2d9700 1 -- 172.21.15.86:0/3660367892 shutdown_connections 2024-03-04T12:20:56.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:20:56.126+0000 7f132c2d9700 1 -- 172.21.15.86:0/3660367892 wait complete. 2024-03-04T12:20:56.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:56.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:56.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:56 smithi060 ceph-mon[32745]: from='osd.8 [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-04T12:20:56.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:56 smithi060 ceph-mon[32745]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-04T12:20:56.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:56.772 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:56.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:56.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:56 smithi086 ceph-mon[36851]: from='osd.8 [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-04T12:20:56.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:56 smithi086 ceph-mon[36851]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-04T12:20:56.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.773 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:56.799 DEBUG:teuthology.orchestra.run.smithi086:osd.8> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.8.service 2024-03-04T12:20:56.802 INFO:tasks.cephadm:Deploying osd.9 on smithi060 with /dev/vg_nvme/lv_1... 2024-03-04T12:20:56.802 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-04T12:20:56.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:20:56.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:20:56.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:56 smithi063 ceph-mon[36717]: from='osd.8 [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-04T12:20:56.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:56 smithi063 ceph-mon[36717]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-04T12:20:56.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:20:56.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:20:57.057 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:20:57.512 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:57 smithi060 ceph-mon[32745]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:57.512 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:57 smithi060 ceph-mon[32745]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-04T12:20:57.513 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:57 smithi060 ceph-mon[32745]: osdmap e57: 9 total, 8 up, 9 in 2024-03-04T12:20:57.513 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:57 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:57.513 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:57 smithi060 ceph-mon[32745]: from='osd.8 [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:20:57.513 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:57 smithi060 ceph-mon[32745]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:20:57.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:57 smithi063 ceph-mon[36717]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:57.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:57 smithi063 ceph-mon[36717]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-04T12:20:57.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:57 smithi063 ceph-mon[36717]: osdmap e57: 9 total, 8 up, 9 in 2024-03-04T12:20:57.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:57 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:57.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:57 smithi063 ceph-mon[36717]: from='osd.8 [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:20:57.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:57 smithi063 ceph-mon[36717]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:20:57.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:57 smithi086 ceph-mon[36851]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:57.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:57 smithi086 ceph-mon[36851]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-04T12:20:57.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:57 smithi086 ceph-mon[36851]: osdmap e57: 9 total, 8 up, 9 in 2024-03-04T12:20:57.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:57 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:57.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:57 smithi086 ceph-mon[36851]: from='osd.8 [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:20:57.694 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:57 smithi086 ceph-mon[36851]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:20:58.003 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:20:57 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:20:57.774+0000 7fef6a4bc700 -1 osd.8 0 waiting for initial osdmap 2024-03-04T12:20:58.003 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:20:57 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:20:57.784+0000 7fef622a7700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T12:20:58.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:58 smithi060 ceph-mon[32745]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T12:20:58.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:58 smithi060 ceph-mon[32745]: osdmap e58: 9 total, 8 up, 9 in 2024-03-04T12:20:58.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:58 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:58.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:58 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:58.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:58 smithi060 ceph-mon[32745]: from='osd.8 ' entity='osd.8' 2024-03-04T12:20:58.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:58 smithi086 ceph-mon[36851]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T12:20:58.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:58 smithi086 ceph-mon[36851]: osdmap e58: 9 total, 8 up, 9 in 2024-03-04T12:20:58.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:58 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:58.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:58 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:58.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:58 smithi086 ceph-mon[36851]: from='osd.8 ' entity='osd.8' 2024-03-04T12:20:58.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:58 smithi063 ceph-mon[36717]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T12:20:58.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:58 smithi063 ceph-mon[36717]: osdmap e58: 9 total, 8 up, 9 in 2024-03-04T12:20:58.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:58 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:58.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:58 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:58.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:58 smithi063 ceph-mon[36717]: from='osd.8 ' entity='osd.8' 2024-03-04T12:20:59.190 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-04T12:20:59.190 INFO:teuthology.orchestra.run.smithi060.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-04T12:20:59.190 INFO:teuthology.orchestra.run.smithi060.stderr: stderr: 10+0 records in 2024-03-04T12:20:59.190 INFO:teuthology.orchestra.run.smithi060.stderr:10+0 records out 2024-03-04T12:20:59.190 INFO:teuthology.orchestra.run.smithi060.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00838891 s, 1.2 GB/s 2024-03-04T12:20:59.190 INFO:teuthology.orchestra.run.smithi060.stderr:--> Zapping successful for: 2024-03-04T12:20:59.409 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:59 smithi086 ceph-mon[36851]: purged_snaps scrub starts 2024-03-04T12:20:59.410 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:59 smithi086 ceph-mon[36851]: purged_snaps scrub ok 2024-03-04T12:20:59.410 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:59 smithi086 ceph-mon[36851]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:59.410 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:59 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:59.410 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:59 smithi086 ceph-mon[36851]: osd.8 [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944] boot 2024-03-04T12:20:59.410 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:59 smithi086 ceph-mon[36851]: osdmap e59: 9 total, 9 up, 9 in 2024-03-04T12:20:59.410 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:20:59 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:59.567 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:59 smithi060 ceph-mon[32745]: purged_snaps scrub starts 2024-03-04T12:20:59.567 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:59 smithi060 ceph-mon[32745]: purged_snaps scrub ok 2024-03-04T12:20:59.567 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:59 smithi060 ceph-mon[32745]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:59.567 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:59 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:59.567 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:59 smithi060 ceph-mon[32745]: osd.8 [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944] boot 2024-03-04T12:20:59.567 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:59 smithi060 ceph-mon[32745]: osdmap e59: 9 total, 9 up, 9 in 2024-03-04T12:20:59.567 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:20:59 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:59.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:59 smithi063 ceph-mon[36717]: purged_snaps scrub starts 2024-03-04T12:20:59.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:59 smithi063 ceph-mon[36717]: purged_snaps scrub ok 2024-03-04T12:20:59.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:59 smithi063 ceph-mon[36717]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 240 MiB used, 715 GiB / 715 GiB avail 2024-03-04T12:20:59.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:59 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:59.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:59 smithi063 ceph-mon[36717]: osd.8 [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944] boot 2024-03-04T12:20:59.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:59 smithi063 ceph-mon[36717]: osdmap e59: 9 total, 9 up, 9 in 2024-03-04T12:20:59.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:20:59 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:20:59.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi060:vg_nvme/lv_1 2024-03-04T12:21:00.232 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:21:00.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:00 smithi060 ceph-mon[32745]: osdmap e60: 9 total, 9 up, 9 in 2024-03-04T12:21:00.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:00 smithi060 ceph-mon[32745]: Detected new or changed devices on smithi086 2024-03-04T12:21:00.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:00.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:00.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:00.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:00.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:00.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:00 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi086 to 5693M 2024-03-04T12:21:00.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:00.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:00.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:00.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:00 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:00.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:00 smithi086 ceph-mon[36851]: osdmap e60: 9 total, 9 up, 9 in 2024-03-04T12:21:00.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:00 smithi086 ceph-mon[36851]: Detected new or changed devices on smithi086 2024-03-04T12:21:00.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:00.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:00.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:00.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:00.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:00.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:00 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi086 to 5693M 2024-03-04T12:21:00.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:00.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:00.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:00.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:00 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:00.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:00 smithi063 ceph-mon[36717]: osdmap e60: 9 total, 9 up, 9 in 2024-03-04T12:21:00.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:00 smithi063 ceph-mon[36717]: Detected new or changed devices on smithi086 2024-03-04T12:21:00.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:00.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:00.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:00.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:00.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:00.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:00 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi086 to 5693M 2024-03-04T12:21:00.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:00.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:00.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:00.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:00 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:01.637 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:01 smithi060 ceph-mon[32745]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:01.637 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:01 smithi060 ceph-mon[32745]: osdmap e61: 9 total, 9 up, 9 in 2024-03-04T12:21:01.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:01 smithi086 ceph-mon[36851]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:01.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:01 smithi086 ceph-mon[36851]: osdmap e61: 9 total, 9 up, 9 in 2024-03-04T12:21:01.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:01 smithi063 ceph-mon[36717]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:01.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:01 smithi063 ceph-mon[36717]: osdmap e61: 9 total, 9 up, 9 in 2024-03-04T12:21:01.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.895+0000 7f1633fb7700 1 -- 172.21.15.60:0/3678426957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f162c1028f0 msgr2=0x7f162c1091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:01.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.895+0000 7f1633fb7700 1 --2- 172.21.15.60:0/3678426957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f162c1028f0 0x7f162c1091a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f162400b0d0 tx=0x7f162400b3e0 comp rx=0 tx=0).stop 2024-03-04T12:21:01.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 -- 172.21.15.60:0/3678426957 shutdown_connections 2024-03-04T12:21:01.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 --2- 172.21.15.60:0/3678426957 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f162c1028f0 0x7f162c1091a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:01.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 --2- 172.21.15.60:0/3678426957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f162c100d50 0x7f162c101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:01.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 --2- 172.21.15.60:0/3678426957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f162c101f50 0x7f162c1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:01.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 -- 172.21.15.60:0/3678426957 >> 172.21.15.60:0/3678426957 conn(0x7f162c0fc2f0 msgr2=0x7f162c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:21:01.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 -- 172.21.15.60:0/3678426957 shutdown_connections 2024-03-04T12:21:01.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 -- 172.21.15.60:0/3678426957 wait complete. 2024-03-04T12:21:01.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 Processor -- start 2024-03-04T12:21:01.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 -- start start 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f162c100d50 0x7f162c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f162c101f50 0x7f162c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f162c1028f0 0x7f162c19f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162c10c990 con 0x7f162c100d50 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162c10c810 con 0x7f162c1028f0 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.896+0000 7f1633fb7700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162c10cb10 con 0x7f162c101f50 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1631d53700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f162c100d50 0x7f162c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1631d53700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f162c100d50 0x7f162c1980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:41892/0 (socket says 172.21.15.60:41892) 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1631d53700 1 -- 172.21.15.60:0/373600998 learned_addr learned my addr 172.21.15.60:0/373600998 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1631552700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f162c101f50 0x7f162c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1632554700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f162c1028f0 0x7f162c19f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1631d53700 1 -- 172.21.15.60:0/373600998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f162c101f50 msgr2=0x7f162c198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1631d53700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f162c101f50 0x7f162c198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:01.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1631d53700 1 -- 172.21.15.60:0/373600998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f162c1028f0 msgr2=0x7f162c19f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:01.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1631d53700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f162c1028f0 0x7f162c19f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:01.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1631d53700 1 -- 172.21.15.60:0/373600998 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c19fce0 con 0x7f162c100d50 2024-03-04T12:21:01.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1632554700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f162c1028f0 0x7f162c19f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:21:01.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1631552700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f162c101f50 0x7f162c198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:21:01.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1631d53700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f162c100d50 0x7f162c1980d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f162800c9a0 tx=0x7f162800cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:21:01.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1622ffd700 1 -- 172.21.15.60:0/373600998 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628004220 con 0x7f162c100d50 2024-03-04T12:21:01.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1622ffd700 1 -- 172.21.15.60:0/373600998 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1628007d40 con 0x7f162c100d50 2024-03-04T12:21:01.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1633fb7700 1 -- 172.21.15.60:0/373600998 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162c19ffd0 con 0x7f162c100d50 2024-03-04T12:21:01.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1622ffd700 1 -- 172.21.15.60:0/373600998 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162800f450 con 0x7f162c100d50 2024-03-04T12:21:01.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.897+0000 7f1633fb7700 1 -- 172.21.15.60:0/373600998 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f162c1a04b0 con 0x7f162c100d50 2024-03-04T12:21:01.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.898+0000 7f1633fb7700 1 -- 172.21.15.60:0/373600998 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f162c068b10 con 0x7f162c100d50 2024-03-04T12:21:01.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.899+0000 7f1622ffd700 1 -- 172.21.15.60:0/373600998 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f1628007860 con 0x7f162c100d50 2024-03-04T12:21:01.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.900+0000 7f1622ffd700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f16180a1e90 0x7f16180a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:21:01.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.900+0000 7f1631552700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f16180a1e90 0x7f16180a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:21:01.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.900+0000 7f1622ffd700 1 -- 172.21.15.60:0/373600998 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5829+0+0 (secure 0 0 0) 0x7f16280cd2b0 con 0x7f162c100d50 2024-03-04T12:21:01.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.900+0000 7f1631552700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f16180a1e90 0x7f16180a4350 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f161c0099d0 tx=0x7f161c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:21:01.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:01.905+0000 7f1622ffd700 1 -- 172.21.15.60:0/373600998 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f162c068b10 con 0x7f162c100d50 2024-03-04T12:21:02.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:02.118+0000 7f1633fb7700 1 -- 172.21.15.60:0/373600998 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f162c0fe470 con 0x7f16180a1e90 2024-03-04T12:21:02.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:02 smithi060 ceph-mon[32745]: osdmap e62: 9 total, 9 up, 9 in 2024-03-04T12:21:02.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:02 smithi060 ceph-mon[32745]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:21:02.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:02 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:21:02.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:02 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:21:02.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:02 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:02.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:02 smithi086 ceph-mon[36851]: osdmap e62: 9 total, 9 up, 9 in 2024-03-04T12:21:02.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:02 smithi086 ceph-mon[36851]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:21:02.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:02 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:21:02.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:02 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:21:02.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:02 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:02.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:02 smithi063 ceph-mon[36717]: osdmap e62: 9 total, 9 up, 9 in 2024-03-04T12:21:02.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:02 smithi063 ceph-mon[36717]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi060:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:21:02.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:02 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:21:02.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:02 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:21:02.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:02 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:03.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:03 smithi060 ceph-mon[32745]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:03.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:03 smithi060 ceph-mon[32745]: osdmap e63: 9 total, 9 up, 9 in 2024-03-04T12:21:03.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:03 smithi086 ceph-mon[36851]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:03.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:03 smithi086 ceph-mon[36851]: osdmap e63: 9 total, 9 up, 9 in 2024-03-04T12:21:03.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:03 smithi063 ceph-mon[36717]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:03.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:03 smithi063 ceph-mon[36717]: osdmap e63: 9 total, 9 up, 9 in 2024-03-04T12:21:05.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:05 smithi060 ceph-mon[32745]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-04T12:21:05.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:05 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1206285383' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92ab3932-3338-42a8-ae91-6d348af0c507"}]: dispatch 2024-03-04T12:21:05.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:05 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1206285383' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92ab3932-3338-42a8-ae91-6d348af0c507"}]': finished 2024-03-04T12:21:05.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:05 smithi060 ceph-mon[32745]: osdmap e64: 10 total, 9 up, 10 in 2024-03-04T12:21:05.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:05.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:05 smithi086 ceph-mon[36851]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-04T12:21:05.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:05 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1206285383' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92ab3932-3338-42a8-ae91-6d348af0c507"}]: dispatch 2024-03-04T12:21:05.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:05 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1206285383' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92ab3932-3338-42a8-ae91-6d348af0c507"}]': finished 2024-03-04T12:21:05.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:05 smithi086 ceph-mon[36851]: osdmap e64: 10 total, 9 up, 10 in 2024-03-04T12:21:05.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:05.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:05 smithi063 ceph-mon[36717]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-04T12:21:05.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:05 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1206285383' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92ab3932-3338-42a8-ae91-6d348af0c507"}]: dispatch 2024-03-04T12:21:05.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:05 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1206285383' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92ab3932-3338-42a8-ae91-6d348af0c507"}]': finished 2024-03-04T12:21:05.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:05 smithi063 ceph-mon[36717]: osdmap e64: 10 total, 9 up, 10 in 2024-03-04T12:21:05.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:06.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:06 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1027510531' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:21:06.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:06 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1027510531' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:21:06.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:06 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1027510531' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:21:07.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:07 smithi060 ceph-mon[32745]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-04T12:21:07.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:07 smithi086 ceph-mon[36851]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-04T12:21:07.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:07 smithi063 ceph-mon[36717]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-04T12:21:09.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:09 smithi060 ceph-mon[32745]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-04T12:21:09.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:09 smithi086 ceph-mon[36851]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-04T12:21:09.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:09 smithi063 ceph-mon[36717]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-04T12:21:11.629 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:11 smithi060 ceph-mon[32745]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-04T12:21:11.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:11 smithi086 ceph-mon[36851]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-04T12:21:11.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:11 smithi063 ceph-mon[36717]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-04T12:21:13.572 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:13 smithi060 ceph-mon[32745]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-04T12:21:13.572 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:13 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-04T12:21:13.572 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:13 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:13.573 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:13 smithi060 ceph-mon[32745]: Deploying daemon osd.9 on smithi060 2024-03-04T12:21:13.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:13 smithi086 ceph-mon[36851]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-04T12:21:13.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:13 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-04T12:21:13.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:13 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:13.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:13 smithi086 ceph-mon[36851]: Deploying daemon osd.9 on smithi060 2024-03-04T12:21:13.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:13 smithi063 ceph-mon[36717]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-04T12:21:13.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:13 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-04T12:21:13.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:13 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:13.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:13 smithi063 ceph-mon[36717]: Deploying daemon osd.9 on smithi060 2024-03-04T12:21:15.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:15 smithi060 ceph-mon[32745]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:15.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:15 smithi086 ceph-mon[36851]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:15.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:15 smithi063 ceph-mon[36717]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:17.548 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:17 smithi060 ceph-mon[32745]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:17.833 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:17 smithi086 ceph-mon[36851]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:17.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:17 smithi063 ceph-mon[36717]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:19.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:19 smithi060 ceph-mon[32745]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:19.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:19.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:19.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:21:20.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:19 smithi086 ceph-mon[36851]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:20.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:20.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:20.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:21:20.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:19 smithi063 ceph-mon[36717]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:20.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:20.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:20.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:21:21.904 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:21 smithi060 ceph-mon[32745]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:22.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:21 smithi086 ceph-mon[36851]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:22.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:21 smithi063 ceph-mon[36717]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:22.646 INFO:teuthology.orchestra.run.smithi060.stdout:Created osd(s) 9 on host 'smithi060' 2024-03-04T12:21:22.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.640+0000 7f1622ffd700 1 -- 172.21.15.60:0/373600998 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f162c0fe470 con 0x7f16180a1e90 2024-03-04T12:21:22.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.643+0000 7f1633fb7700 1 -- 172.21.15.60:0/373600998 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f16180a1e90 msgr2=0x7f16180a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:22.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.643+0000 7f1633fb7700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f16180a1e90 0x7f16180a4350 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f161c0099d0 tx=0x7f161c008040 comp rx=0 tx=0).stop 2024-03-04T12:21:22.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.643+0000 7f1633fb7700 1 -- 172.21.15.60:0/373600998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f162c100d50 msgr2=0x7f162c1980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:22.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.643+0000 7f1633fb7700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f162c100d50 0x7f162c1980d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f162800c9a0 tx=0x7f162800cd60 comp rx=0 tx=0).stop 2024-03-04T12:21:22.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.643+0000 7f1633fb7700 1 -- 172.21.15.60:0/373600998 shutdown_connections 2024-03-04T12:21:22.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.643+0000 7f1633fb7700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f162c1028f0 0x7f162c19f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:22.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.643+0000 7f1633fb7700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f16180a1e90 0x7f16180a4350 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:22.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.643+0000 7f1633fb7700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f162c100d50 0x7f162c1980d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:22.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.643+0000 7f1633fb7700 1 --2- 172.21.15.60:0/373600998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f162c101f50 0x7f162c198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:22.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.643+0000 7f1633fb7700 1 -- 172.21.15.60:0/373600998 >> 172.21.15.60:0/373600998 conn(0x7f162c0fc2f0 msgr2=0x7f162c0fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:21:22.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.643+0000 7f1633fb7700 1 -- 172.21.15.60:0/373600998 shutdown_connections 2024-03-04T12:21:22.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:21:22.643+0000 7f1633fb7700 1 -- 172.21.15.60:0/373600998 wait complete. 2024-03-04T12:21:22.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:22 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:22.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:22 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:22.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:22 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:22.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:22 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:22.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:22 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:22.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:22 smithi060 ceph-mon[32745]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-04T12:21:22.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:22 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:22.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:22 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:22.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:22 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:21:23.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:22 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:23.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:22 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:23.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:22 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:23.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:22 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:23.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:22 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:23.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:22 smithi086 ceph-mon[36851]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-04T12:21:23.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:22 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:23.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:22 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:23.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:22 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:21:23.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:22 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:23.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:22 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:23.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:22 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:23.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:22 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:23.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:22 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:23.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:22 smithi063 ceph-mon[36717]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-04T12:21:23.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:22 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:23.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:22 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:23.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:22 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:21:23.302 DEBUG:teuthology.orchestra.run.smithi060:osd.9> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.9.service 2024-03-04T12:21:23.304 INFO:tasks.cephadm:Deploying osd.10 on smithi063 with /dev/vg_nvme/lv_1... 2024-03-04T12:21:23.304 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-04T12:21:23.565 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T12:21:23.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:23 smithi060 ceph-mon[32745]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:23.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:23 smithi060 ceph-mon[32745]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-04T12:21:23.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:23 smithi060 ceph-mon[32745]: osdmap e65: 10 total, 9 up, 10 in 2024-03-04T12:21:23.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:23 smithi060 ceph-mon[32745]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-03-04T12:21:23.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:23 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:24.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:23 smithi063 ceph-mon[36717]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:24.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:23 smithi063 ceph-mon[36717]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-04T12:21:24.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:23 smithi063 ceph-mon[36717]: osdmap e65: 10 total, 9 up, 10 in 2024-03-04T12:21:24.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:23 smithi063 ceph-mon[36717]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-03-04T12:21:24.055 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:23 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:24.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:23 smithi086 ceph-mon[36851]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:24.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:23 smithi086 ceph-mon[36851]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-04T12:21:24.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:23 smithi086 ceph-mon[36851]: osdmap e65: 10 total, 9 up, 10 in 2024-03-04T12:21:24.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:23 smithi086 ceph-mon[36851]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]: dispatch 2024-03-04T12:21:24.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:23 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:24.508 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:21:24 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:21:24.338+0000 7fce4bed2700 -1 osd.9 0 waiting for initial osdmap 2024-03-04T12:21:24.508 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:21:24 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:21:24.349+0000 7fce43cbd700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T12:21:25.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:24 smithi086 ceph-mon[36851]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-03-04T12:21:25.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:24 smithi086 ceph-mon[36851]: osdmap e66: 10 total, 9 up, 10 in 2024-03-04T12:21:25.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:24 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:25.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:24 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:25.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:24 smithi086 ceph-mon[36851]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' 2024-03-04T12:21:25.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:24 smithi063 ceph-mon[36717]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-03-04T12:21:25.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:24 smithi063 ceph-mon[36717]: osdmap e66: 10 total, 9 up, 10 in 2024-03-04T12:21:25.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:24 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:25.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:24 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:25.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:24 smithi063 ceph-mon[36717]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' 2024-03-04T12:21:25.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:24 smithi060 ceph-mon[32745]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi060", "root=default"]}]': finished 2024-03-04T12:21:25.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:24 smithi060 ceph-mon[32745]: osdmap e66: 10 total, 9 up, 10 in 2024-03-04T12:21:25.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:24 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:25.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:24 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:25.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:24 smithi060 ceph-mon[32745]: from='osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384]' entity='osd.9' 2024-03-04T12:21:25.876 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-04T12:21:25.876 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-04T12:21:25.876 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-03-04T12:21:25.876 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-03-04T12:21:25.876 INFO:teuthology.orchestra.run.smithi063.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115654 s, 907 MB/s 2024-03-04T12:21:25.876 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-03-04T12:21:25.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:25 smithi060 ceph-mon[32745]: purged_snaps scrub starts 2024-03-04T12:21:25.927 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:25 smithi060 ceph-mon[32745]: purged_snaps scrub ok 2024-03-04T12:21:25.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:25 smithi060 ceph-mon[32745]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:25.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:25 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:25.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:25 smithi060 ceph-mon[32745]: osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384] boot 2024-03-04T12:21:25.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:25 smithi060 ceph-mon[32745]: osdmap e67: 10 total, 10 up, 10 in 2024-03-04T12:21:25.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:25 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:26.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:25 smithi086 ceph-mon[36851]: purged_snaps scrub starts 2024-03-04T12:21:26.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:25 smithi086 ceph-mon[36851]: purged_snaps scrub ok 2024-03-04T12:21:26.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:25 smithi086 ceph-mon[36851]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:26.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:25 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:26.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:25 smithi086 ceph-mon[36851]: osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384] boot 2024-03-04T12:21:26.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:25 smithi086 ceph-mon[36851]: osdmap e67: 10 total, 10 up, 10 in 2024-03-04T12:21:26.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:25 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:26.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:25 smithi063 ceph-mon[36717]: purged_snaps scrub starts 2024-03-04T12:21:26.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:25 smithi063 ceph-mon[36717]: purged_snaps scrub ok 2024-03-04T12:21:26.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:25 smithi063 ceph-mon[36717]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-04T12:21:26.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:25 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:26.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:25 smithi063 ceph-mon[36717]: osd.9 [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384] boot 2024-03-04T12:21:26.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:25 smithi063 ceph-mon[36717]: osdmap e67: 10 total, 10 up, 10 in 2024-03-04T12:21:26.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:25 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:21:26.720 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi063:vg_nvme/lv_1 2024-03-04T12:21:26.979 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T12:21:27.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:27.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: osdmap e68: 10 total, 10 up, 10 in 2024-03-04T12:21:27.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: Detected new or changed devices on smithi060 2024-03-04T12:21:27.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:27.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:27.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:27.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:27.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:27.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:27.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi060 to 4273M 2024-03-04T12:21:27.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:27.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:27.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:27.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:27 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:27.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:27.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: osdmap e68: 10 total, 10 up, 10 in 2024-03-04T12:21:27.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: Detected new or changed devices on smithi060 2024-03-04T12:21:27.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:27.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:27.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:27.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:27.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:27.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:27.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi060 to 4273M 2024-03-04T12:21:27.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:27.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:27.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:27.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:27 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: osdmap e68: 10 total, 10 up, 10 in 2024-03-04T12:21:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: Detected new or changed devices on smithi060 2024-03-04T12:21:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi060 to 4273M 2024-03-04T12:21:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:27.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:27.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:27 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:28.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.764+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2821985187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8b01028e0 msgr2=0x7fd8b0104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.764+0000 7fd8b69c1700 1 --2- 172.21.15.63:0/2821985187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8b01028e0 0x7fd8b0104cd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009990 tx=0x7fd8a4009ca0 comp rx=0 tx=0).stop 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.764+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2821985187 shutdown_connections 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.764+0000 7fd8b69c1700 1 --2- 172.21.15.63:0/2821985187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0107b40 0x7fd8b0109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.764+0000 7fd8b69c1700 1 --2- 172.21.15.63:0/2821985187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8b01028e0 0x7fd8b0104cd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.764+0000 7fd8b69c1700 1 --2- 172.21.15.63:0/2821985187 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0105210 0x7fd8b0107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.764+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2821985187 >> 172.21.15.63:0/2821985187 conn(0x7fd8b00fc2f0 msgr2=0x7fd8b00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.764+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2821985187 shutdown_connections 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.764+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2821985187 wait complete. 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.765+0000 7fd8b69c1700 1 Processor -- start 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.765+0000 7fd8b69c1700 1 -- start start 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.765+0000 7fd8b69c1700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8b01028e0 0x7fd8b01980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.765+0000 7fd8b69c1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0105210 0x7fd8b0198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:21:28.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.765+0000 7fd8b69c1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0107b40 0x7fd8b019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.765+0000 7fd8b69c1700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b010daa0 con 0x7fd8b0105210 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.765+0000 7fd8b69c1700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b010d920 con 0x7fd8b01028e0 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.765+0000 7fd8b69c1700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b010dc20 con 0x7fd8b0107b40 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.765+0000 7fd8affff700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8b01028e0 0x7fd8b01980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.765+0000 7fd8affff700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8b01028e0 0x7fd8b01980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57070/0 (socket says 172.21.15.63:57070) 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.765+0000 7fd8affff700 1 -- 172.21.15.63:0/2546765554 learned_addr learned my addr 172.21.15.63:0/2546765554 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.765+0000 7fd8af7fe700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0105210 0x7fd8b0198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8b4f5e700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0107b40 0x7fd8b019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8affff700 1 -- 172.21.15.63:0/2546765554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0107b40 msgr2=0x7fd8b019f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8affff700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0107b40 0x7fd8b019f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8affff700 1 -- 172.21.15.63:0/2546765554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0105210 msgr2=0x7fd8b0198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8affff700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0105210 0x7fd8b0198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:28.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8affff700 1 -- 172.21.15.63:0/2546765554 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b019fcb0 con 0x7fd8b01028e0 2024-03-04T12:21:28.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8b4f5e700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0107b40 0x7fd8b019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:21:28.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8affff700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8b01028e0 0x7fd8b01980c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400b5c0 tx=0x7fd8a4009f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:21:28.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8ad7fa700 1 -- 172.21.15.63:0/2546765554 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4026070 con 0x7fd8b01028e0 2024-03-04T12:21:28.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2546765554 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b019ff40 con 0x7fd8b01028e0 2024-03-04T12:21:28.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8ad7fa700 1 -- 172.21.15.63:0/2546765554 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8a4004150 con 0x7fd8b01028e0 2024-03-04T12:21:28.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8ad7fa700 1 -- 172.21.15.63:0/2546765554 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a4004bd0 con 0x7fd8b01028e0 2024-03-04T12:21:28.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.766+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2546765554 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b01a0360 con 0x7fd8b01028e0 2024-03-04T12:21:28.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.767+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2546765554 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b0068ae0 con 0x7fd8b01028e0 2024-03-04T12:21:28.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.768+0000 7fd8ad7fa700 1 -- 172.21.15.63:0/2546765554 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fd8a4029070 con 0x7fd8b01028e0 2024-03-04T12:21:28.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.769+0000 7fd8ad7fa700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8980a1e10 0x7fd8980a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:21:28.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.769+0000 7fd8ad7fa700 1 -- 172.21.15.63:0/2546765554 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fd8a40ce0c0 con 0x7fd8b01028e0 2024-03-04T12:21:28.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.769+0000 7fd8af7fe700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8980a1e10 0x7fd8980a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:21:28.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.770+0000 7fd8af7fe700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8980a1e10 0x7fd8980a42d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000ac50 tx=0x7fd8a000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:21:28.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.774+0000 7fd8ad7fa700 1 -- 172.21.15.63:0/2546765554 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd8a405cd20 con 0x7fd8b01028e0 2024-03-04T12:21:28.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:28.990+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2546765554 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8b0062260 con 0x7fd8980a1e10 2024-03-04T12:21:29.624 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:29 smithi063 ceph-mon[36717]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:29.624 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:29 smithi063 ceph-mon[36717]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:21:29.624 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:29 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:21:29.625 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:29 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:21:29.625 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:29 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:29.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:29 smithi060 ceph-mon[32745]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:29.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:29 smithi060 ceph-mon[32745]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:21:29.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:29 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:21:29.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:29 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:21:29.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:29 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:29.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:29 smithi086 ceph-mon[36851]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:29.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:29 smithi086 ceph-mon[36851]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:21:29.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:29 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:21:29.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:29 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:21:29.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:29 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:31.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:31 smithi063 ceph-mon[36717]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:31.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:31 smithi060 ceph-mon[32745]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:31.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:31 smithi086 ceph-mon[36851]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:32.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:32 smithi063 ceph-mon[36717]: from='client.? 172.21.15.63:0/3414744496' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9fa85ae1-3566-4486-8586-05edcecc7848"}]: dispatch 2024-03-04T12:21:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:32 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9fa85ae1-3566-4486-8586-05edcecc7848"}]: dispatch 2024-03-04T12:21:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:32 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9fa85ae1-3566-4486-8586-05edcecc7848"}]': finished 2024-03-04T12:21:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:32 smithi063 ceph-mon[36717]: osdmap e69: 11 total, 10 up, 11 in 2024-03-04T12:21:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:32.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:32 smithi060 ceph-mon[32745]: from='client.? 172.21.15.63:0/3414744496' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9fa85ae1-3566-4486-8586-05edcecc7848"}]: dispatch 2024-03-04T12:21:32.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:32 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9fa85ae1-3566-4486-8586-05edcecc7848"}]: dispatch 2024-03-04T12:21:32.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:32 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9fa85ae1-3566-4486-8586-05edcecc7848"}]': finished 2024-03-04T12:21:32.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:32 smithi060 ceph-mon[32745]: osdmap e69: 11 total, 10 up, 11 in 2024-03-04T12:21:32.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:32 smithi086 ceph-mon[36851]: from='client.? 172.21.15.63:0/3414744496' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9fa85ae1-3566-4486-8586-05edcecc7848"}]: dispatch 2024-03-04T12:21:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:32 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9fa85ae1-3566-4486-8586-05edcecc7848"}]: dispatch 2024-03-04T12:21:32.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:32 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9fa85ae1-3566-4486-8586-05edcecc7848"}]': finished 2024-03-04T12:21:32.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:32 smithi086 ceph-mon[36851]: osdmap e69: 11 total, 10 up, 11 in 2024-03-04T12:21:32.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:33.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:33 smithi060 ceph-mon[32745]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:33.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:33 smithi060 ceph-mon[32745]: from='client.? 172.21.15.63:0/2266548282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:21:33.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:33 smithi086 ceph-mon[36851]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:33.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:33 smithi086 ceph-mon[36851]: from='client.? 172.21.15.63:0/2266548282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:21:33.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:33 smithi063 ceph-mon[36717]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:33.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:33 smithi063 ceph-mon[36717]: from='client.? 172.21.15.63:0/2266548282' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:21:35.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:35 smithi060 ceph-mon[32745]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:35.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:35 smithi086 ceph-mon[36851]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:35.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:35 smithi063 ceph-mon[36717]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:37.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:37 smithi063 ceph-mon[36717]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:37.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:37 smithi060 ceph-mon[32745]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:37.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:37 smithi086 ceph-mon[36851]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:39.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:39 smithi060 ceph-mon[32745]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:39.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:39 smithi086 ceph-mon[36851]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:39.850 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:39 smithi063 ceph-mon[36717]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:40.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:40 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-04T12:21:40.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:40 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:40.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:40 smithi063 ceph-mon[36717]: Deploying daemon osd.10 on smithi063 2024-03-04T12:21:40.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:40 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-04T12:21:40.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:40 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:40.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:40 smithi060 ceph-mon[32745]: Deploying daemon osd.10 on smithi063 2024-03-04T12:21:40.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:40 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-04T12:21:40.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:40 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:40.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:40 smithi086 ceph-mon[36851]: Deploying daemon osd.10 on smithi063 2024-03-04T12:21:41.674 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:41 smithi063 ceph-mon[36717]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:41.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:41 smithi060 ceph-mon[32745]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:41.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:41 smithi086 ceph-mon[36851]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:43.542 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:43 smithi063 ceph-mon[36717]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:43.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:43 smithi060 ceph-mon[32745]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:43.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:43 smithi086 ceph-mon[36851]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:45.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:45 smithi063 ceph-mon[36717]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:45.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:45 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:45.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:45 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:45.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:45 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:21:45.651 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:45 smithi086 ceph-mon[36851]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:45.652 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:45 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:45.652 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:45 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:45.652 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:45 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:21:45.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:45 smithi060 ceph-mon[32745]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:45.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:45 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:45.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:45 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:45.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:45 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:21:47.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:47 smithi063 ceph-mon[36717]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:47.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:47 smithi060 ceph-mon[32745]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:47.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:47 smithi086 ceph-mon[36851]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:48.864 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 10 on host 'smithi063' 2024-03-04T12:21:48.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.858+0000 7fd8ad7fa700 1 -- 172.21.15.63:0/2546765554 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fd8b0062260 con 0x7fd8980a1e10 2024-03-04T12:21:48.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.861+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2546765554 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8980a1e10 msgr2=0x7fd8980a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:48.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.861+0000 7fd8b69c1700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8980a1e10 0x7fd8980a42d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000ac50 tx=0x7fd8a000a380 comp rx=0 tx=0).stop 2024-03-04T12:21:48.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.861+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2546765554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8b01028e0 msgr2=0x7fd8b01980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:48.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.861+0000 7fd8b69c1700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8b01028e0 0x7fd8b01980c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400b5c0 tx=0x7fd8a4009f50 comp rx=0 tx=0).stop 2024-03-04T12:21:48.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.862+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2546765554 shutdown_connections 2024-03-04T12:21:48.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.862+0000 7fd8b69c1700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8b0107b40 0x7fd8b019f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:48.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.862+0000 7fd8b69c1700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8b01028e0 0x7fd8b01980c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:48.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.862+0000 7fd8b69c1700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8980a1e10 0x7fd8980a42d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:48.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.862+0000 7fd8b69c1700 1 --2- 172.21.15.63:0/2546765554 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd8b0105210 0x7fd8b0198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:48.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.862+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2546765554 >> 172.21.15.63:0/2546765554 conn(0x7fd8b00fc2f0 msgr2=0x7fd8b0103280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:21:48.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.862+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2546765554 shutdown_connections 2024-03-04T12:21:48.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-03-04T12:21:48.862+0000 7fd8b69c1700 1 -- 172.21.15.63:0/2546765554 wait complete. 2024-03-04T12:21:49.524 DEBUG:teuthology.orchestra.run.smithi063:osd.10> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.10.service 2024-03-04T12:21:49.527 INFO:tasks.cephadm:Deploying osd.11 on smithi086 with /dev/vg_nvme/lv_1... 2024-03-04T12:21:49.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-04T12:21:49.774 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:21:50.113 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:49 smithi086 ceph-mon[36851]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:50.113 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:50.113 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:50.113 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:49 smithi086 ceph-mon[36851]: from='osd.10 [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-04T12:21:50.113 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:49 smithi086 ceph-mon[36851]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-04T12:21:50.113 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:50.113 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:50.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:49 smithi060 ceph-mon[32745]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:50.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:50.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:50.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:49 smithi060 ceph-mon[32745]: from='osd.10 [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-04T12:21:50.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:49 smithi060 ceph-mon[32745]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-04T12:21:50.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:50.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:50.326 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:49 smithi063 ceph-mon[36717]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:50.326 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:50.326 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:50.327 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:49 smithi063 ceph-mon[36717]: from='osd.10 [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-04T12:21:50.327 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:49 smithi063 ceph-mon[36717]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-04T12:21:50.327 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:50.327 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:51.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:50 smithi063 ceph-mon[36717]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-04T12:21:51.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:50 smithi063 ceph-mon[36717]: osdmap e70: 11 total, 10 up, 11 in 2024-03-04T12:21:51.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:50 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:51.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:50 smithi063 ceph-mon[36717]: from='osd.10 [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:21:51.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:50 smithi063 ceph-mon[36717]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:21:51.101 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:21:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:21:50.865+0000 7f7296555700 -1 osd.10 0 waiting for initial osdmap 2024-03-04T12:21:51.101 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:21:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:21:50.879+0000 7f7292348700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T12:21:51.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:50 smithi060 ceph-mon[32745]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-04T12:21:51.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:50 smithi060 ceph-mon[32745]: osdmap e70: 11 total, 10 up, 11 in 2024-03-04T12:21:51.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:50 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:51.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:50 smithi060 ceph-mon[32745]: from='osd.10 [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:21:51.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:50 smithi060 ceph-mon[32745]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:21:51.217 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:50 smithi086 ceph-mon[36851]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-04T12:21:51.217 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:50 smithi086 ceph-mon[36851]: osdmap e70: 11 total, 10 up, 11 in 2024-03-04T12:21:51.217 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:50 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:51.217 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:50 smithi086 ceph-mon[36851]: from='osd.10 [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:21:51.218 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:50 smithi086 ceph-mon[36851]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-03-04T12:21:51.960 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-04T12:21:51.960 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-04T12:21:51.960 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-03-04T12:21:51.960 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-03-04T12:21:51.960 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127957 s, 819 MB/s 2024-03-04T12:21:51.960 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-03-04T12:21:52.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:51 smithi060 ceph-mon[32745]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:52.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:51 smithi060 ceph-mon[32745]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-04T12:21:52.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:51 smithi060 ceph-mon[32745]: osdmap e71: 11 total, 10 up, 11 in 2024-03-04T12:21:52.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:51 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:52.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:51 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:52.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:51 smithi086 ceph-mon[36851]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:52.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:51 smithi086 ceph-mon[36851]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-04T12:21:52.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:51 smithi086 ceph-mon[36851]: osdmap e71: 11 total, 10 up, 11 in 2024-03-04T12:21:52.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:51 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:52.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:51 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:52.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:51 smithi063 ceph-mon[36717]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:52.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:51 smithi063 ceph-mon[36717]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-03-04T12:21:52.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:51 smithi063 ceph-mon[36717]: osdmap e71: 11 total, 10 up, 11 in 2024-03-04T12:21:52.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:51 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:52.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:51 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:52.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_1 2024-03-04T12:21:52.978 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:21:53.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:52 smithi086 ceph-mon[36851]: purged_snaps scrub starts 2024-03-04T12:21:53.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:52 smithi086 ceph-mon[36851]: purged_snaps scrub ok 2024-03-04T12:21:53.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:53.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:52 smithi086 ceph-mon[36851]: osd.10 [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227] boot 2024-03-04T12:21:53.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:52 smithi086 ceph-mon[36851]: osdmap e72: 11 total, 11 up, 11 in 2024-03-04T12:21:53.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:52 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:53.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:52 smithi063 ceph-mon[36717]: purged_snaps scrub starts 2024-03-04T12:21:53.100 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:52 smithi063 ceph-mon[36717]: purged_snaps scrub ok 2024-03-04T12:21:53.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:53.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:52 smithi063 ceph-mon[36717]: osd.10 [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227] boot 2024-03-04T12:21:53.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:52 smithi063 ceph-mon[36717]: osdmap e72: 11 total, 11 up, 11 in 2024-03-04T12:21:53.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:52 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:53.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:52 smithi060 ceph-mon[32745]: purged_snaps scrub starts 2024-03-04T12:21:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:52 smithi060 ceph-mon[32745]: purged_snaps scrub ok 2024-03-04T12:21:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:52 smithi060 ceph-mon[32745]: osd.10 [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227] boot 2024-03-04T12:21:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:52 smithi060 ceph-mon[32745]: osdmap e72: 11 total, 11 up, 11 in 2024-03-04T12:21:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:52 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:21:54.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:54.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: osdmap e73: 11 total, 11 up, 11 in 2024-03-04T12:21:54.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:54.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:54.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:54.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:54.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:54.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:54.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:54.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:54.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:54.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:54.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:21:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: osdmap e73: 11 total, 11 up, 11 in 2024-03-04T12:21:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:54.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:54.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:21:54.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 307 MiB used, 894 GiB / 894 GiB avail 2024-03-04T12:21:54.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: osdmap e73: 11 total, 11 up, 11 in 2024-03-04T12:21:54.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:54.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:54.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:54.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:54.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:54.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:21:54.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:54.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:54.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:54.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:54.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:21:54.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.640+0000 7f15f6f04700 1 -- 172.21.15.86:0/1289883660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15f0105210 msgr2=0x7f15f0107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:54.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.640+0000 7f15f6f04700 1 --2- 172.21.15.86:0/1289883660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15f0105210 0x7f15f0107600 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009990 tx=0x7f15dc009ca0 comp rx=0 tx=0).stop 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 -- 172.21.15.86:0/1289883660 shutdown_connections 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 --2- 172.21.15.86:0/1289883660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15f0107b40 0x7f15f0109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 --2- 172.21.15.86:0/1289883660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f01028e0 0x7f15f0104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 --2- 172.21.15.86:0/1289883660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15f0105210 0x7f15f0107600 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 -- 172.21.15.86:0/1289883660 >> 172.21.15.86:0/1289883660 conn(0x7f15f00fc2f0 msgr2=0x7f15f00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 -- 172.21.15.86:0/1289883660 shutdown_connections 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 -- 172.21.15.86:0/1289883660 wait complete. 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 Processor -- start 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 -- start start 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15f01028e0 0x7f15f0102240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f0105210 0x7f15f0100890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15f0107b40 0x7f15f0100dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:21:54.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f010da30 con 0x7f15f0105210 2024-03-04T12:21:54.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f010d8b0 con 0x7f15f01028e0 2024-03-04T12:21:54.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.641+0000 7f15f6f04700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f010dbb0 con 0x7f15f0107b40 2024-03-04T12:21:54.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15f54a1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15f0107b40 0x7f15f0100dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:21:54.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15f54a1700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15f0107b40 0x7f15f0100dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56444/0 (socket says 172.21.15.86:56444) 2024-03-04T12:21:54.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15f54a1700 1 -- 172.21.15.86:0/4192651458 learned_addr learned my addr 172.21.15.86:0/4192651458 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-03-04T12:21:54.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15effff700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f0105210 0x7f15f0100890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:21:54.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15f4ca0700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15f01028e0 0x7f15f0102240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:21:54.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15f54a1700 1 -- 172.21.15.86:0/4192651458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15f01028e0 msgr2=0x7f15f0102240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:54.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15f54a1700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15f01028e0 0x7f15f0102240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:54.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15f54a1700 1 -- 172.21.15.86:0/4192651458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f0105210 msgr2=0x7f15f0100890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15f54a1700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f0105210 0x7f15f0100890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15f54a1700 1 -- 172.21.15.86:0/4192651458 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f0101680 con 0x7f15f0107b40 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15f54a1700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15f0107b40 0x7f15f0100dd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f15e8007d40 tx=0x7f15e800a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15edffb700 1 -- 172.21.15.86:0/4192651458 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e8016020 con 0x7f15f0107b40 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15edffb700 1 -- 172.21.15.86:0/4192651458 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15e800ad40 con 0x7f15f0107b40 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15f6f04700 1 -- 172.21.15.86:0/4192651458 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f01a2170 con 0x7f15f0107b40 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15edffb700 1 -- 172.21.15.86:0/4192651458 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e80115e0 con 0x7f15f0107b40 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.642+0000 7f15f6f04700 1 -- 172.21.15.86:0/4192651458 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f01a2630 con 0x7f15f0107b40 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.643+0000 7f15f6f04700 1 -- 172.21.15.86:0/4192651458 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f0068ae0 con 0x7f15f0107b40 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.644+0000 7f15edffb700 1 -- 172.21.15.86:0/4192651458 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f15e8019020 con 0x7f15f0107b40 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.645+0000 7f15edffb700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f15e00a1e10 0x7f15e00a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.645+0000 7f15edffb700 1 -- 172.21.15.86:0/4192651458 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f15e80cdae0 con 0x7f15f0107b40 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.645+0000 7f15f4ca0700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f15e00a1e10 0x7f15e00a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:21:54.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.646+0000 7f15f4ca0700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f15e00a1e10 0x7f15e00a42d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f15e400ba60 tx=0x7f15e4005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:21:54.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.649+0000 7f15edffb700 1 -- 172.21.15.86:0/4192651458 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f15e8020020 con 0x7f15f0107b40 2024-03-04T12:21:54.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:21:54.849+0000 7f15f6f04700 1 -- 172.21.15.86:0/4192651458 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f15f0062260 con 0x7f15e00a1e10 2024-03-04T12:21:55.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:54 smithi060 ceph-mon[32745]: Detected new or changed devices on smithi063 2024-03-04T12:21:55.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:54 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi063 to 4270M 2024-03-04T12:21:55.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:54 smithi060 ceph-mon[32745]: osdmap e74: 11 total, 11 up, 11 in 2024-03-04T12:21:55.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:54 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:55.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:54 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:55.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:54 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:55.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:54 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:21:55.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:54 smithi060 ceph-mon[32745]: osdmap e75: 11 total, 11 up, 11 in 2024-03-04T12:21:55.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:54 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:21:55.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:54 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:55.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:54 smithi086 ceph-mon[36851]: Detected new or changed devices on smithi063 2024-03-04T12:21:55.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:54 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi063 to 4270M 2024-03-04T12:21:55.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:54 smithi086 ceph-mon[36851]: osdmap e74: 11 total, 11 up, 11 in 2024-03-04T12:21:55.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:54 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:55.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:54 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:55.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:54 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:55.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:54 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:21:55.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:54 smithi086 ceph-mon[36851]: osdmap e75: 11 total, 11 up, 11 in 2024-03-04T12:21:55.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:54 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:21:55.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:54 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:55.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:54 smithi063 ceph-mon[36717]: Detected new or changed devices on smithi063 2024-03-04T12:21:55.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:54 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi063 to 4270M 2024-03-04T12:21:55.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:54 smithi063 ceph-mon[36717]: osdmap e74: 11 total, 11 up, 11 in 2024-03-04T12:21:55.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:54 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:55.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:54 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:21:55.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:54 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:21:55.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:54 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T12:21:55.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:54 smithi063 ceph-mon[36717]: osdmap e75: 11 total, 11 up, 11 in 2024-03-04T12:21:55.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:54 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T12:21:55.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:54 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:21:56.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:56 smithi086 ceph-mon[36851]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:21:56.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:56 smithi086 ceph-mon[36851]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:21:56.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:56 smithi086 ceph-mon[36851]: osdmap e76: 11 total, 11 up, 11 in 2024-03-04T12:21:56.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:56 smithi063 ceph-mon[36717]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:21:56.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:56 smithi063 ceph-mon[36717]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:21:56.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:56 smithi063 ceph-mon[36717]: osdmap e76: 11 total, 11 up, 11 in 2024-03-04T12:21:56.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:56 smithi060 ceph-mon[32745]: pgmap v239: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:21:56.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:56 smithi060 ceph-mon[32745]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:21:56.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:56 smithi060 ceph-mon[32745]: osdmap e76: 11 total, 11 up, 11 in 2024-03-04T12:21:58.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:58 smithi086 ceph-mon[36851]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:21:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:58 smithi086 ceph-mon[36851]: from='client.? 172.21.15.86:0/3641117188' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "344b7daf-897d-4217-8cba-192c507c09e1"}]: dispatch 2024-03-04T12:21:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:58 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "344b7daf-897d-4217-8cba-192c507c09e1"}]: dispatch 2024-03-04T12:21:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:58 smithi086 ceph-mon[36851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "344b7daf-897d-4217-8cba-192c507c09e1"}]': finished 2024-03-04T12:21:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:58 smithi086 ceph-mon[36851]: osdmap e77: 12 total, 11 up, 12 in 2024-03-04T12:21:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:58 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:21:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:58 smithi063 ceph-mon[36717]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:21:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:58 smithi063 ceph-mon[36717]: from='client.? 172.21.15.86:0/3641117188' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "344b7daf-897d-4217-8cba-192c507c09e1"}]: dispatch 2024-03-04T12:21:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:58 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "344b7daf-897d-4217-8cba-192c507c09e1"}]: dispatch 2024-03-04T12:21:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:58 smithi063 ceph-mon[36717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "344b7daf-897d-4217-8cba-192c507c09e1"}]': finished 2024-03-04T12:21:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:58 smithi063 ceph-mon[36717]: osdmap e77: 12 total, 11 up, 12 in 2024-03-04T12:21:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:58 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:21:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:58 smithi060 ceph-mon[32745]: pgmap v242: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:21:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:58 smithi060 ceph-mon[32745]: from='client.? 172.21.15.86:0/3641117188' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "344b7daf-897d-4217-8cba-192c507c09e1"}]: dispatch 2024-03-04T12:21:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:58 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "344b7daf-897d-4217-8cba-192c507c09e1"}]: dispatch 2024-03-04T12:21:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:58 smithi060 ceph-mon[32745]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "344b7daf-897d-4217-8cba-192c507c09e1"}]': finished 2024-03-04T12:21:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:58 smithi060 ceph-mon[32745]: osdmap e77: 12 total, 11 up, 12 in 2024-03-04T12:21:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:58 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:21:59.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:21:59 smithi086 ceph-mon[36851]: from='client.? 172.21.15.86:0/1535875764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:21:59.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:21:59 smithi063 ceph-mon[36717]: from='client.? 172.21.15.86:0/1535875764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:21:59.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:21:59 smithi060 ceph-mon[32745]: from='client.? 172.21.15.86:0/1535875764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T12:22:00.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:00 smithi086 ceph-mon[36851]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:00.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:00 smithi063 ceph-mon[36717]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:00.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:00 smithi060 ceph-mon[32745]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:02.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:02 smithi086 ceph-mon[36851]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T12:22:02.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:02 smithi063 ceph-mon[36717]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T12:22:02.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:02 smithi060 ceph-mon[32745]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T12:22:04.094 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:04 smithi086 ceph-mon[36851]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-04T12:22:04.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:04 smithi063 ceph-mon[36717]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-04T12:22:04.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:04 smithi060 ceph-mon[32745]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-04T12:22:06.290 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:06 smithi086 ceph-mon[36851]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-04T12:22:06.290 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:06 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-04T12:22:06.290 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:06 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:06 smithi063 ceph-mon[36717]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-04T12:22:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:06 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-04T12:22:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:06 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:06.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:06 smithi060 ceph-mon[32745]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-04T12:22:06.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:06 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-04T12:22:06.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:06 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:07.322 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:07 smithi086 ceph-mon[36851]: Deploying daemon osd.11 on smithi086 2024-03-04T12:22:07.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:07 smithi063 ceph-mon[36717]: Deploying daemon osd.11 on smithi086 2024-03-04T12:22:07.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:07 smithi060 ceph-mon[32745]: Deploying daemon osd.11 on smithi086 2024-03-04T12:22:08.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:08 smithi086 ceph-mon[36851]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T12:22:08.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:08 smithi063 ceph-mon[36717]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T12:22:08.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:08 smithi060 ceph-mon[32745]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T12:22:10.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:10 smithi086 ceph-mon[36851]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-04T12:22:10.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:10 smithi063 ceph-mon[36717]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-04T12:22:10.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:10 smithi060 ceph-mon[32745]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-03-04T12:22:12.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:12 smithi086 ceph-mon[36851]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-04T12:22:12.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:12 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:12.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:12 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:12.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:12 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:22:12.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:12 smithi063 ceph-mon[36717]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-04T12:22:12.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:12 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:12.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:12 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:12.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:12 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:22:12.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:12 smithi060 ceph-mon[32745]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-04T12:22:12.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:12 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:12.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:12 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:12.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:12 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:22:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:14 smithi086 ceph-mon[36851]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:14.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:14 smithi063 ceph-mon[36717]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:14.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:14 smithi060 ceph-mon[32745]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:15.375 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 11 on host 'smithi086' 2024-03-04T12:22:15.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.369+0000 7f15edffb700 1 -- 172.21.15.86:0/4192651458 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f15f0062260 con 0x7f15e00a1e10 2024-03-04T12:22:15.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.371+0000 7f15f6f04700 1 -- 172.21.15.86:0/4192651458 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f15e00a1e10 msgr2=0x7f15e00a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:15.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.371+0000 7f15f6f04700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f15e00a1e10 0x7f15e00a42d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f15e400ba60 tx=0x7f15e4005d50 comp rx=0 tx=0).stop 2024-03-04T12:22:15.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.371+0000 7f15f6f04700 1 -- 172.21.15.86:0/4192651458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15f0107b40 msgr2=0x7f15f0100dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:15.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.371+0000 7f15f6f04700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15f0107b40 0x7f15f0100dd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f15e8007d40 tx=0x7f15e800a340 comp rx=0 tx=0).stop 2024-03-04T12:22:15.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.371+0000 7f15f6f04700 1 -- 172.21.15.86:0/4192651458 shutdown_connections 2024-03-04T12:22:15.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.371+0000 7f15f6f04700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15f0107b40 0x7f15f0100dd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:15.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.371+0000 7f15f6f04700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15f01028e0 0x7f15f0102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:15.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.371+0000 7f15f6f04700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f15e00a1e10 0x7f15e00a42d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:15.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.371+0000 7f15f6f04700 1 --2- 172.21.15.86:0/4192651458 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f15f0105210 0x7f15f0100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:15.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.371+0000 7f15f6f04700 1 -- 172.21.15.86:0/4192651458 >> 172.21.15.86:0/4192651458 conn(0x7f15f00fc2f0 msgr2=0x7f15f0103220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:15.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.371+0000 7f15f6f04700 1 -- 172.21.15.86:0/4192651458 shutdown_connections 2024-03-04T12:22:15.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:15.371+0000 7f15f6f04700 1 -- 172.21.15.86:0/4192651458 wait complete. 2024-03-04T12:22:15.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:15 smithi086 ceph-mon[36851]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:15.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:15.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:22:15.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:15 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:22:15.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:15 smithi086 ceph-mon[36851]: from='osd.11 [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-04T12:22:15.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:15 smithi086 ceph-mon[36851]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-04T12:22:15.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:15 smithi060 ceph-mon[32745]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:15.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:15.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:22:15.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:15 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:22:15.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:15 smithi060 ceph-mon[32745]: from='osd.11 [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-04T12:22:15.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:15 smithi060 ceph-mon[32745]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-04T12:22:15.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:15 smithi063 ceph-mon[36717]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:15.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:15.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:22:15.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:15.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:15 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:22:15.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:15 smithi063 ceph-mon[36717]: from='osd.11 [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-04T12:22:15.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:15 smithi063 ceph-mon[36717]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-04T12:22:16.012 DEBUG:teuthology.orchestra.run.smithi086:osd.11> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.11.service 2024-03-04T12:22:16.014 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-04T12:22:16.014 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd stat -f json 2024-03-04T12:22:16.267 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:22:17.553 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:17 smithi086 ceph-mon[36851]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:17.553 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:17 smithi086 ceph-mon[36851]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-04T12:22:17.553 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:17 smithi086 ceph-mon[36851]: osdmap e78: 12 total, 11 up, 12 in 2024-03-04T12:22:17.553 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:17 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:17.553 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:17 smithi086 ceph-mon[36851]: from='osd.11 [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:22:17.553 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:17 smithi086 ceph-mon[36851]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:22:17.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:17 smithi063 ceph-mon[36717]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:17.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:17 smithi063 ceph-mon[36717]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-04T12:22:17.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:17 smithi063 ceph-mon[36717]: osdmap e78: 12 total, 11 up, 12 in 2024-03-04T12:22:17.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:17 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:17.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:17 smithi063 ceph-mon[36717]: from='osd.11 [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:22:17.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:17 smithi063 ceph-mon[36717]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:22:17.645 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:17 smithi060 ceph-mon[32745]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:17.645 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:17 smithi060 ceph-mon[32745]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-04T12:22:17.645 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:17 smithi060 ceph-mon[32745]: osdmap e78: 12 total, 11 up, 12 in 2024-03-04T12:22:17.645 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:17 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:17.645 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:17 smithi060 ceph-mon[32745]: from='osd.11 [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:22:17.645 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:17 smithi060 ceph-mon[32745]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-04T12:22:17.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:22:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:22:17.766+0000 7fd37dad0700 -1 osd.11 0 waiting for initial osdmap 2024-03-04T12:22:17.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:22:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:22:17.779+0000 7fd3758bb700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T12:22:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.164+0000 7f9691675700 1 -- 172.21.15.60:0/610441922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f968c107ba0 msgr2=0x7f968c109fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.164+0000 7f9691675700 1 --2- 172.21.15.60:0/610441922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f968c107ba0 0x7f968c109fb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f967c00b0a0 tx=0x7f967c00b3b0 comp rx=0 tx=0).stop 2024-03-04T12:22:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.164+0000 7f9691675700 1 -- 172.21.15.60:0/610441922 shutdown_connections 2024-03-04T12:22:18.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.164+0000 7f9691675700 1 --2- 172.21.15.60:0/610441922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f968c107ba0 0x7f968c109fb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.164+0000 7f9691675700 1 --2- 172.21.15.60:0/610441922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f968c102940 0x7f968c104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.164+0000 7f9691675700 1 --2- 172.21.15.60:0/610441922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f968c105270 0x7f968c107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.164+0000 7f9691675700 1 -- 172.21.15.60:0/610441922 >> 172.21.15.60:0/610441922 conn(0x7f968c0fc2f0 msgr2=0x7f968c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.164+0000 7f9691675700 1 -- 172.21.15.60:0/610441922 shutdown_connections 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.164+0000 7f9691675700 1 -- 172.21.15.60:0/610441922 wait complete. 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.164+0000 7f9691675700 1 Processor -- start 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f9691675700 1 -- start start 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f9691675700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f968c102940 0x7f968c1022a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f9691675700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f968c105270 0x7f968c1008f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f9691675700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f968c107ba0 0x7f968c100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f9691675700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c10da20 con 0x7f968c105270 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f9691675700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c10d8a0 con 0x7f968c102940 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f9691675700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c10dba0 con 0x7f968c107ba0 2024-03-04T12:22:18.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968a7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f968c105270 0x7f968c1008f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968affd700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f968c102940 0x7f968c1022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968affd700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f968c102940 0x7f968c1022a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:36842/0 (socket says 172.21.15.60:36842) 2024-03-04T12:22:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968a7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f968c105270 0x7f968c1008f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34622/0 (socket says 172.21.15.60:34622) 2024-03-04T12:22:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968affd700 1 -- 172.21.15.60:0/1688379946 learned_addr learned my addr 172.21.15.60:0/1688379946 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:18.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968b7fe700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f968c107ba0 0x7f968c100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968affd700 1 -- 172.21.15.60:0/1688379946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f968c107ba0 msgr2=0x7f968c100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968affd700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f968c107ba0 0x7f968c100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:18.174 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968affd700 1 -- 172.21.15.60:0/1688379946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f968c105270 msgr2=0x7f968c1008f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968affd700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f968c105270 0x7f968c1008f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968affd700 1 -- 172.21.15.60:0/1688379946 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c1016e0 con 0x7f968c102940 2024-03-04T12:22:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968b7fe700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f968c107ba0 0x7f968c100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:22:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968a7fc700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f968c105270 0x7f968c1008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:22:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.165+0000 7f968affd700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f968c102940 0x7f968c1022a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f968400d980 tx=0x7f968400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.166+0000 7f9683fff700 1 -- 172.21.15.60:0/1688379946 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9684017070 con 0x7f968c102940 2024-03-04T12:22:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.166+0000 7f9691675700 1 -- 172.21.15.60:0/1688379946 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968c1a21a0 con 0x7f968c102940 2024-03-04T12:22:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.166+0000 7f9683fff700 1 -- 172.21.15.60:0/1688379946 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9684005020 con 0x7f968c102940 2024-03-04T12:22:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.166+0000 7f9683fff700 1 -- 172.21.15.60:0/1688379946 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968400fbf0 con 0x7f968c102940 2024-03-04T12:22:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.166+0000 7f9691675700 1 -- 172.21.15.60:0/1688379946 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968c1a26b0 con 0x7f968c102940 2024-03-04T12:22:18.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.167+0000 7f9691675700 1 -- 172.21.15.60:0/1688379946 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f968c068ae0 con 0x7f968c102940 2024-03-04T12:22:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.168+0000 7f9683fff700 1 -- 172.21.15.60:0/1688379946 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f96840051d0 con 0x7f968c102940 2024-03-04T12:22:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.168+0000 7f9683fff700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f96780a1e10 0x7f96780a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.168+0000 7f968a7fc700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f96780a1e10 0x7f96780a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:18.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.169+0000 7f9683fff700 1 -- 172.21.15.60:0/1688379946 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f96840cd850 con 0x7f968c102940 2024-03-04T12:22:18.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.169+0000 7f968a7fc700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f96780a1e10 0x7f96780a42d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9674007fd0 tx=0x7f96740073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:18.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.172+0000 7f9683fff700 1 -- 172.21.15.60:0/1688379946 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9684020020 con 0x7f968c102940 2024-03-04T12:22:18.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.388+0000 7f9691675700 1 -- 172.21.15.60:0/1688379946 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f968c02dcb0 con 0x7f968c102940 2024-03-04T12:22:18.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.389+0000 7f9683fff700 1 -- 172.21.15.60:0/1688379946 <== mon.2 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7f968c02dcb0 con 0x7f968c102940 2024-03-04T12:22:18.395 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.391+0000 7f9691675700 1 -- 172.21.15.60:0/1688379946 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f96780a1e10 msgr2=0x7f96780a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.391+0000 7f9691675700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f96780a1e10 0x7f96780a42d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9674007fd0 tx=0x7f96740073d0 comp rx=0 tx=0).stop 2024-03-04T12:22:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.391+0000 7f9691675700 1 -- 172.21.15.60:0/1688379946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f968c102940 msgr2=0x7f968c1022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.391+0000 7f9691675700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f968c102940 0x7f968c1022a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f968400d980 tx=0x7f968400dc90 comp rx=0 tx=0).stop 2024-03-04T12:22:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.391+0000 7f9691675700 1 -- 172.21.15.60:0/1688379946 shutdown_connections 2024-03-04T12:22:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.391+0000 7f9691675700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f968c107ba0 0x7f968c100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.391+0000 7f9691675700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f968c102940 0x7f968c1022a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.391+0000 7f9691675700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f96780a1e10 0x7f96780a42d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.391+0000 7f9691675700 1 --2- 172.21.15.60:0/1688379946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f968c105270 0x7f968c1008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.391+0000 7f9691675700 1 -- 172.21.15.60:0/1688379946 >> 172.21.15.60:0/1688379946 conn(0x7f968c0fc2f0 msgr2=0x7f968c103280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.391+0000 7f9691675700 1 -- 172.21.15.60:0/1688379946 shutdown_connections 2024-03-04T12:22:18.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:18.391+0000 7f9691675700 1 -- 172.21.15.60:0/1688379946 wait complete. 2024-03-04T12:22:18.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:18 smithi086 ceph-mon[36851]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T12:22:18.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:18 smithi086 ceph-mon[36851]: osdmap e79: 12 total, 11 up, 12 in 2024-03-04T12:22:18.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:18 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:18.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:18 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:18.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:18 smithi086 ceph-mon[36851]: from='osd.11 ' entity='osd.11' 2024-03-04T12:22:18.600 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:18 smithi063 ceph-mon[36717]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T12:22:18.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:18 smithi063 ceph-mon[36717]: osdmap e79: 12 total, 11 up, 12 in 2024-03-04T12:22:18.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:18 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:18.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:18 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:18.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:18 smithi063 ceph-mon[36717]: from='osd.11 ' entity='osd.11' 2024-03-04T12:22:18.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:18 smithi060 ceph-mon[32745]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-04T12:22:18.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:18 smithi060 ceph-mon[32745]: osdmap e79: 12 total, 11 up, 12 in 2024-03-04T12:22:18.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:18.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:18 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:18.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:18 smithi060 ceph-mon[32745]: from='osd.11 ' entity='osd.11' 2024-03-04T12:22:19.209 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1709554911,"num_in_osds":12,"osd_in_since":1709554917,"num_remapped_pgs":0} 2024-03-04T12:22:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: purged_snaps scrub starts 2024-03-04T12:22:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: purged_snaps scrub ok 2024-03-04T12:22:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1688379946' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T12:22:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: osd.11 [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201] boot 2024-03-04T12:22:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: osdmap e80: 12 total, 12 up, 12 in 2024-03-04T12:22:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:19.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:19.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:19.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:22:19.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:22:19.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:22:19.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:22:19.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:19 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:19.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: purged_snaps scrub starts 2024-03-04T12:22:19.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: purged_snaps scrub ok 2024-03-04T12:22:19.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:19.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:19.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1688379946' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T12:22:19.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: osd.11 [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201] boot 2024-03-04T12:22:19.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: osdmap e80: 12 total, 12 up, 12 in 2024-03-04T12:22:19.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:19.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:19.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:19.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:22:19.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:22:19.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:22:19.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:22:19.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:19 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:19.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: purged_snaps scrub starts 2024-03-04T12:22:19.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: purged_snaps scrub ok 2024-03-04T12:22:19.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-04T12:22:19.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:19.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1688379946' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T12:22:19.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: osd.11 [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201] boot 2024-03-04T12:22:19.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: osdmap e80: 12 total, 12 up, 12 in 2024-03-04T12:22:19.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:22:19.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:19.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:19.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:22:19.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:22:19.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:22:19.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:22:19.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:19 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:20.210 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd stat -f json 2024-03-04T12:22:20.469 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:22:20.590 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:20 smithi060 ceph-mon[32745]: Detected new or changed devices on smithi086 2024-03-04T12:22:20.591 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:20 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi086 to 4270M 2024-03-04T12:22:20.591 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:20.591 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:22:20.591 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:20 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:20.591 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:20 smithi060 ceph-mon[32745]: osdmap e81: 12 total, 12 up, 12 in 2024-03-04T12:22:20.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:20 smithi086 ceph-mon[36851]: Detected new or changed devices on smithi086 2024-03-04T12:22:20.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:20 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi086 to 4270M 2024-03-04T12:22:20.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:20 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:20.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:20 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:22:20.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:20 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:20.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:20 smithi086 ceph-mon[36851]: osdmap e81: 12 total, 12 up, 12 in 2024-03-04T12:22:20.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:20 smithi063 ceph-mon[36717]: Detected new or changed devices on smithi086 2024-03-04T12:22:20.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:20 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi086 to 4270M 2024-03-04T12:22:20.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:20 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:20.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:20 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:22:20.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:20 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:20.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:20 smithi063 ceph-mon[36717]: osdmap e81: 12 total, 12 up, 12 in 2024-03-04T12:22:21.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:21 smithi063 ceph-mon[36717]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:21.607 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:21 smithi060 ceph-mon[32745]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:21.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:21 smithi086 ceph-mon[36851]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:22.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.178+0000 7fd8511e2700 1 -- 172.21.15.60:0/1709841520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd84c074880 msgr2=0x7fd84c074d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:22.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.178+0000 7fd8511e2700 1 --2- 172.21.15.60:0/1709841520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd84c074880 0x7fd84c074d00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd840009990 tx=0x7fd840009ca0 comp rx=0 tx=0).stop 2024-03-04T12:22:22.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.178+0000 7fd8511e2700 1 -- 172.21.15.60:0/1709841520 shutdown_connections 2024-03-04T12:22:22.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.178+0000 7fd8511e2700 1 --2- 172.21.15.60:0/1709841520 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd84c10e420 0x7fd84c110810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:22.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.178+0000 7fd8511e2700 1 --2- 172.21.15.60:0/1709841520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd84c075e50 0x7fd84c0742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:22.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.178+0000 7fd8511e2700 1 --2- 172.21.15.60:0/1709841520 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd84c074880 0x7fd84c074d00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:22.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.178+0000 7fd8511e2700 1 -- 172.21.15.60:0/1709841520 >> 172.21.15.60:0/1709841520 conn(0x7fd84c0fd400 msgr2=0x7fd84c0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:22.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.178+0000 7fd8511e2700 1 -- 172.21.15.60:0/1709841520 shutdown_connections 2024-03-04T12:22:22.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.178+0000 7fd8511e2700 1 -- 172.21.15.60:0/1709841520 wait complete. 2024-03-04T12:22:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.178+0000 7fd8511e2700 1 Processor -- start 2024-03-04T12:22:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd8511e2700 1 -- start start 2024-03-04T12:22:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd8511e2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd84c074880 0x7fd84c19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd8511e2700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd84c075e50 0x7fd84c19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd8511e2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd84c10e420 0x7fd84c1a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd8511e2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c1140a0 con 0x7fd84c074880 2024-03-04T12:22:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd8511e2700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c113f20 con 0x7fd84c075e50 2024-03-04T12:22:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd8511e2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c114220 con 0x7fd84c10e420 2024-03-04T12:22:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd84ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd84c074880 0x7fd84c19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd84ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd84c074880 0x7fd84c19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34642/0 (socket says 172.21.15.60:34642) 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd84ad9d700 1 -- 172.21.15.60:0/1040270280 learned_addr learned my addr 172.21.15.60:0/1040270280 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd84b59e700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd84c10e420 0x7fd84c1a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd84a59c700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd84c075e50 0x7fd84c19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd84ad9d700 1 -- 172.21.15.60:0/1040270280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd84c10e420 msgr2=0x7fd84c1a5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd84ad9d700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd84c10e420 0x7fd84c1a5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd84ad9d700 1 -- 172.21.15.60:0/1040270280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd84c075e50 msgr2=0x7fd84c19ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd84ad9d700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd84c075e50 0x7fd84c19ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd84ad9d700 1 -- 172.21.15.60:0/1040270280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c1a6340 con 0x7fd84c074880 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd84a59c700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd84c075e50 0x7fd84c19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.179+0000 7fd84b59e700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd84c10e420 0x7fd84c1a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.180+0000 7fd84ad9d700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd84c074880 0x7fd84c19e6e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00d9b0 tx=0x7fd83c00dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.180+0000 7fd833fff700 1 -- 172.21.15.60:0/1040270280 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c017070 con 0x7fd84c074880 2024-03-04T12:22:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.180+0000 7fd8511e2700 1 -- 172.21.15.60:0/1040270280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84c1a6630 con 0x7fd84c074880 2024-03-04T12:22:22.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.180+0000 7fd833fff700 1 -- 172.21.15.60:0/1040270280 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd83c009b40 con 0x7fd84c074880 2024-03-04T12:22:22.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.180+0000 7fd833fff700 1 -- 172.21.15.60:0/1040270280 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c00f620 con 0x7fd84c074880 2024-03-04T12:22:22.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.180+0000 7fd8511e2700 1 -- 172.21.15.60:0/1040270280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84c1a6bf0 con 0x7fd84c074880 2024-03-04T12:22:22.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.181+0000 7fd8511e2700 1 -- 172.21.15.60:0/1040270280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84c068ae0 con 0x7fd84c074880 2024-03-04T12:22:22.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.182+0000 7fd833fff700 1 -- 172.21.15.60:0/1040270280 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fd83c009cf0 con 0x7fd84c074880 2024-03-04T12:22:22.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.183+0000 7fd833fff700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8340a1e10 0x7fd8340a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:22.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.183+0000 7fd84a59c700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8340a1e10 0x7fd8340a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:22.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.183+0000 7fd833fff700 1 -- 172.21.15.60:0/1040270280 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd83c0cd570 con 0x7fd84c074880 2024-03-04T12:22:22.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.183+0000 7fd84a59c700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8340a1e10 0x7fd8340a42d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8400039c0 tx=0x7fd840005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:22.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.187+0000 7fd833fff700 1 -- 172.21.15.60:0/1040270280 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd83c09ada0 con 0x7fd84c074880 2024-03-04T12:22:22.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.392+0000 7fd8511e2700 1 -- 172.21.15.60:0/1040270280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fd84c062a30 con 0x7fd84c074880 2024-03-04T12:22:22.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.393+0000 7fd833fff700 1 -- 172.21.15.60:0/1040270280 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7fd83c09a6a0 con 0x7fd84c074880 2024-03-04T12:22:22.399 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:22.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.395+0000 7fd8511e2700 1 -- 172.21.15.60:0/1040270280 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8340a1e10 msgr2=0x7fd8340a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:22.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.395+0000 7fd8511e2700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8340a1e10 0x7fd8340a42d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8400039c0 tx=0x7fd840005fb0 comp rx=0 tx=0).stop 2024-03-04T12:22:22.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.395+0000 7fd8511e2700 1 -- 172.21.15.60:0/1040270280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd84c074880 msgr2=0x7fd84c19e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:22.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.395+0000 7fd8511e2700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd84c074880 0x7fd84c19e6e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00d9b0 tx=0x7fd83c00dd70 comp rx=0 tx=0).stop 2024-03-04T12:22:22.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.395+0000 7fd8511e2700 1 -- 172.21.15.60:0/1040270280 shutdown_connections 2024-03-04T12:22:22.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.395+0000 7fd8511e2700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd84c10e420 0x7fd84c1a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:22.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.395+0000 7fd8511e2700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8340a1e10 0x7fd8340a42d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:22.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.395+0000 7fd8511e2700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd84c074880 0x7fd84c19e6e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:22.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.395+0000 7fd8511e2700 1 --2- 172.21.15.60:0/1040270280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd84c075e50 0x7fd84c19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:22.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.395+0000 7fd8511e2700 1 -- 172.21.15.60:0/1040270280 >> 172.21.15.60:0/1040270280 conn(0x7fd84c0fd400 msgr2=0x7fd84c1017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:22.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.395+0000 7fd8511e2700 1 -- 172.21.15.60:0/1040270280 shutdown_connections 2024-03-04T12:22:22.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:22.395+0000 7fd8511e2700 1 -- 172.21.15.60:0/1040270280 wait complete. 2024-03-04T12:22:23.153 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1709554938,"num_in_osds":12,"osd_in_since":1709554917,"num_remapped_pgs":0} 2024-03-04T12:22:23.154 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T12:22:23.399 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:22:23.427 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:23 smithi060 ceph-mon[32745]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:23.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:23 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1040270280' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T12:22:23.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:23 smithi063 ceph-mon[36717]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:23.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:23 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1040270280' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T12:22:23.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:23 smithi086 ceph-mon[36851]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:23.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:23 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1040270280' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T12:22:25.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.226+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2667677492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b0107e50 msgr2=0x7f02b0108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:25.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.226+0000 7f02b7fb2700 1 --2- 172.21.15.60:0/2667677492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b0107e50 0x7f02b0108230 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0099c0 tx=0x7f02ac009cd0 comp rx=0 tx=0).stop 2024-03-04T12:22:25.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.226+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2667677492 shutdown_connections 2024-03-04T12:22:25.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.226+0000 7f02b7fb2700 1 --2- 172.21.15.60:0/2667677492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b0102830 0x7f02b010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.226+0000 7f02b7fb2700 1 --2- 172.21.15.60:0/2667677492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b0107e50 0x7f02b0108230 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.226+0000 7f02b7fb2700 1 --2- 172.21.15.60:0/2667677492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02b0101e70 0x7f02b01022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.226+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2667677492 >> 172.21.15.60:0/2667677492 conn(0x7f02b0078f50 msgr2=0x7f02b0079360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.227+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2667677492 shutdown_connections 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.227+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2667677492 wait complete. 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.227+0000 7f02b7fb2700 1 Processor -- start 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.227+0000 7f02b7fb2700 1 -- start start 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.227+0000 7f02b7fb2700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b0101e70 0x7f02b0198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.227+0000 7f02b7fb2700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02b0102830 0x7f02b0198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.227+0000 7f02b7fb2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b0107e50 0x7f02b019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.227+0000 7f02b7fb2700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0112150 con 0x7f02b0107e50 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.227+0000 7f02b7fb2700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0111fd0 con 0x7f02b0102830 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.227+0000 7f02b7fb2700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b01122d0 con 0x7f02b0101e70 2024-03-04T12:22:25.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b654f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b0107e50 0x7f02b019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b654f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b0107e50 0x7f02b019ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34658/0 (socket says 172.21.15.60:34658) 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b654f700 1 -- 172.21.15.60:0/2837827145 learned_addr learned my addr 172.21.15.60:0/2837827145 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b5d4e700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b0101e70 0x7f02b0198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b554d700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02b0102830 0x7f02b0198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b654f700 1 -- 172.21.15.60:0/2837827145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b0101e70 msgr2=0x7f02b0198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b654f700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b0101e70 0x7f02b0198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b654f700 1 -- 172.21.15.60:0/2837827145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02b0102830 msgr2=0x7f02b0198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b654f700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02b0102830 0x7f02b0198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b654f700 1 -- 172.21.15.60:0/2837827145 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b019d570 con 0x7f02b0107e50 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b554d700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02b0102830 0x7f02b0198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b654f700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b0107e50 0x7f02b019ce30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f02a800ada0 tx=0x7f02a800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02a6ffd700 1 -- 172.21.15.60:0/2837827145 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8017020 con 0x7f02b0107e50 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02a6ffd700 1 -- 172.21.15.60:0/2837827145 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f02a800a0d0 con 0x7f02b0107e50 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2837827145 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b019d860 con 0x7f02b0107e50 2024-03-04T12:22:25.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.228+0000 7f02a6ffd700 1 -- 172.21.15.60:0/2837827145 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8005480 con 0x7f02b0107e50 2024-03-04T12:22:25.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.229+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2837827145 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b01a50e0 con 0x7f02b0107e50 2024-03-04T12:22:25.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.230+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2837827145 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b0068ae0 con 0x7f02b0107e50 2024-03-04T12:22:25.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.231+0000 7f02a6ffd700 1 -- 172.21.15.60:0/2837827145 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f02a8005620 con 0x7f02b0107e50 2024-03-04T12:22:25.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.231+0000 7f02a6ffd700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f029c0a1e90 0x7f029c0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:25.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.231+0000 7f02b5d4e700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f029c0a1e90 0x7f029c0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:25.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.231+0000 7f02a6ffd700 1 -- 172.21.15.60:0/2837827145 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f02a80ccfb0 con 0x7f02b0107e50 2024-03-04T12:22:25.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.232+0000 7f02b5d4e700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f029c0a1e90 0x7f029c0a4350 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f02ac000c00 tx=0x7f02ac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:25.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.236+0000 7f02a6ffd700 1 -- 172.21.15.60:0/2837827145 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f02a8021020 con 0x7f02b0107e50 2024-03-04T12:22:25.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.439+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2837827145 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02b01998c0 con 0x7f02b0107e50 2024-03-04T12:22:25.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.440+0000 7f02a6ffd700 1 -- 172.21.15.60:0/2837827145 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18325 (secure 0 0 0) 0x7f02a809a910 con 0x7f02b0107e50 2024-03-04T12:22:25.446 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:25.446 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":81,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:22:20.316992+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:22:25.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.442+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2837827145 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f029c0a1e90 msgr2=0x7f029c0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:25.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.442+0000 7f02b7fb2700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f029c0a1e90 0x7f029c0a4350 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f02ac000c00 tx=0x7f02ac005fb0 comp rx=0 tx=0).stop 2024-03-04T12:22:25.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.442+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2837827145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b0107e50 msgr2=0x7f02b019ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:25.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.442+0000 7f02b7fb2700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b0107e50 0x7f02b019ce30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f02a800ada0 tx=0x7f02a800c330 comp rx=0 tx=0).stop 2024-03-04T12:22:25.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.442+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2837827145 shutdown_connections 2024-03-04T12:22:25.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.442+0000 7f02b7fb2700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f029c0a1e90 0x7f029c0a4350 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:25.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.442+0000 7f02b7fb2700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f02b0107e50 0x7f02b019ce30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:25.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.442+0000 7f02b7fb2700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02b0101e70 0x7f02b0198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:25.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.442+0000 7f02b7fb2700 1 --2- 172.21.15.60:0/2837827145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f02b0102830 0x7f02b0198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:25.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.442+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2837827145 >> 172.21.15.60:0/2837827145 conn(0x7f02b0078f50 msgr2=0x7f02b0100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:25.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.442+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2837827145 shutdown_connections 2024-03-04T12:22:25.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:25.442+0000 7f02b7fb2700 1 -- 172.21.15.60:0/2837827145 wait complete. 2024-03-04T12:22:25.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:25 smithi060 ceph-mon[32745]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:25.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:25 smithi086 ceph-mon[36851]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:25.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:25 smithi063 ceph-mon[36717]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:26.130 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T12:22:26.130 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-04T12:22:26.376 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:22:26.611 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:26 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2837827145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:22:26.707 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:26 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2837827145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:22:26.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:26 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2837827145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:22:27.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:27 smithi086 ceph-mon[36851]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:27.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:27 smithi060 ceph-mon[32745]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:27.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:27 smithi063 ceph-mon[36717]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.206+0000 7f9d40c60700 1 -- 172.21.15.60:0/989737875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d3c101e70 msgr2=0x7f9d3c1022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.206+0000 7f9d40c60700 1 --2- 172.21.15.60:0/989737875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d3c101e70 0x7f9d3c1022f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9d300099c0 tx=0x7f9d30009cd0 comp rx=0 tx=0).stop 2024-03-04T12:22:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.206+0000 7f9d40c60700 1 -- 172.21.15.60:0/989737875 shutdown_connections 2024-03-04T12:22:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.206+0000 7f9d40c60700 1 --2- 172.21.15.60:0/989737875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d3c102830 0x7f9d3c10ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.206+0000 7f9d40c60700 1 --2- 172.21.15.60:0/989737875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d3c107e50 0x7f9d3c108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:28.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.206+0000 7f9d40c60700 1 --2- 172.21.15.60:0/989737875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d3c101e70 0x7f9d3c1022f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:28.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.206+0000 7f9d40c60700 1 -- 172.21.15.60:0/989737875 >> 172.21.15.60:0/989737875 conn(0x7f9d3c078f50 msgr2=0x7f9d3c079360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:28.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.206+0000 7f9d40c60700 1 -- 172.21.15.60:0/989737875 shutdown_connections 2024-03-04T12:22:28.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.206+0000 7f9d40c60700 1 -- 172.21.15.60:0/989737875 wait complete. 2024-03-04T12:22:28.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.207+0000 7f9d40c60700 1 Processor -- start 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.207+0000 7f9d40c60700 1 -- start start 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.207+0000 7f9d40c60700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d3c101e70 0x7f9d3c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d40c60700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d3c102830 0x7f9d3c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d40c60700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d3c107e50 0x7f9d3c19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d40c60700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c1121d0 con 0x7f9d3c107e50 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d40c60700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c112050 con 0x7f9d3c101e70 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d40c60700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3c112350 con 0x7f9d3c102830 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d3ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d3c107e50 0x7f9d3c19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d3ad9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d3c107e50 0x7f9d3c19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59212/0 (socket says 172.21.15.60:59212) 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d3ad9d700 1 -- 172.21.15.60:0/846904391 learned_addr learned my addr 172.21.15.60:0/846904391 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d39d9b700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d3c102830 0x7f9d3c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d3a59c700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d3c101e70 0x7f9d3c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d3ad9d700 1 -- 172.21.15.60:0/846904391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d3c102830 msgr2=0x7f9d3c198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:28.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d3ad9d700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d3c102830 0x7f9d3c198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:28.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d3ad9d700 1 -- 172.21.15.60:0/846904391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d3c101e70 msgr2=0x7f9d3c198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:28.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d3ad9d700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d3c101e70 0x7f9d3c198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:28.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d3ad9d700 1 -- 172.21.15.60:0/846904391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c19d690 con 0x7f9d3c107e50 2024-03-04T12:22:28.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d3ad9d700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d3c107e50 0x7f9d3c19cf50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400b970 tx=0x7f9d3400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:28.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d2b7fe700 1 -- 172.21.15.60:0/846904391 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3400d860 con 0x7f9d3c107e50 2024-03-04T12:22:28.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d40c60700 1 -- 172.21.15.60:0/846904391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3c19d980 con 0x7f9d3c107e50 2024-03-04T12:22:28.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d2b7fe700 1 -- 172.21.15.60:0/846904391 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d3400e430 con 0x7f9d3c107e50 2024-03-04T12:22:28.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d2b7fe700 1 -- 172.21.15.60:0/846904391 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d34017660 con 0x7f9d3c107e50 2024-03-04T12:22:28.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.208+0000 7f9d40c60700 1 -- 172.21.15.60:0/846904391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3c1a5200 con 0x7f9d3c107e50 2024-03-04T12:22:28.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.209+0000 7f9d40c60700 1 -- 172.21.15.60:0/846904391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3c068ae0 con 0x7f9d3c107e50 2024-03-04T12:22:28.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.210+0000 7f9d2b7fe700 1 -- 172.21.15.60:0/846904391 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f9d3400e5e0 con 0x7f9d3c107e50 2024-03-04T12:22:28.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.211+0000 7f9d2b7fe700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9d240a1e10 0x7f9d240a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:28.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.211+0000 7f9d3a59c700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9d240a1e10 0x7f9d240a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:28.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.211+0000 7f9d2b7fe700 1 -- 172.21.15.60:0/846904391 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9d340cd4a0 con 0x7f9d3c107e50 2024-03-04T12:22:28.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.211+0000 7f9d3a59c700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9d240a1e10 0x7f9d240a42d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009a10 tx=0x7f9d2c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:28.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.215+0000 7f9d2b7fe700 1 -- 172.21.15.60:0/846904391 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9d34013030 con 0x7f9d3c107e50 2024-03-04T12:22:28.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.417+0000 7f9d40c60700 1 -- 172.21.15.60:0/846904391 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9d3c1999e0 con 0x7f9d3c107e50 2024-03-04T12:22:28.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.417+0000 7f9d2b7fe700 1 -- 172.21.15.60:0/846904391 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9d3409ad70 con 0x7f9d3c107e50 2024-03-04T12:22:28.425 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:22:28.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.420+0000 7f9d40c60700 1 -- 172.21.15.60:0/846904391 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9d240a1e10 msgr2=0x7f9d240a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.420+0000 7f9d40c60700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9d240a1e10 0x7f9d240a42d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c009a10 tx=0x7f9d2c009190 comp rx=0 tx=0).stop 2024-03-04T12:22:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.420+0000 7f9d40c60700 1 -- 172.21.15.60:0/846904391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d3c107e50 msgr2=0x7f9d3c19cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.420+0000 7f9d40c60700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d3c107e50 0x7f9d3c19cf50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9d3400b970 tx=0x7f9d3400bc80 comp rx=0 tx=0).stop 2024-03-04T12:22:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.420+0000 7f9d40c60700 1 -- 172.21.15.60:0/846904391 shutdown_connections 2024-03-04T12:22:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.420+0000 7f9d40c60700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9d240a1e10 0x7f9d240a42d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.420+0000 7f9d40c60700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9d3c107e50 0x7f9d3c19cf50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.420+0000 7f9d40c60700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d3c101e70 0x7f9d3c198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.420+0000 7f9d40c60700 1 --2- 172.21.15.60:0/846904391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d3c102830 0x7f9d3c198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.420+0000 7f9d40c60700 1 -- 172.21.15.60:0/846904391 >> 172.21.15.60:0/846904391 conn(0x7f9d3c078f50 msgr2=0x7f9d3c100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.420+0000 7f9d40c60700 1 -- 172.21.15.60:0/846904391 shutdown_connections 2024-03-04T12:22:28.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:28.420+0000 7f9d40c60700 1 -- 172.21.15.60:0/846904391 wait complete. 2024-03-04T12:22:29.222 INFO:tasks.cephadm:Adding mds.a on smithi060 2024-03-04T12:22:29.223 INFO:tasks.cephadm:Adding mds.d on smithi060 2024-03-04T12:22:29.223 INFO:tasks.cephadm:Adding mds.g on smithi060 2024-03-04T12:22:29.223 INFO:tasks.cephadm:Adding mds.j on smithi060 2024-03-04T12:22:29.223 INFO:tasks.cephadm:Adding mds.b on smithi063 2024-03-04T12:22:29.223 INFO:tasks.cephadm:Adding mds.e on smithi063 2024-03-04T12:22:29.223 INFO:tasks.cephadm:Adding mds.h on smithi063 2024-03-04T12:22:29.223 INFO:tasks.cephadm:Adding mds.k on smithi063 2024-03-04T12:22:29.223 INFO:tasks.cephadm:Adding mds.c on smithi086 2024-03-04T12:22:29.223 INFO:tasks.cephadm:Adding mds.f on smithi086 2024-03-04T12:22:29.223 INFO:tasks.cephadm:Adding mds.i on smithi086 2024-03-04T12:22:29.223 INFO:tasks.cephadm:Adding mds.l on smithi086 2024-03-04T12:22:29.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph orch apply mds all '12;smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l' 2024-03-04T12:22:29.476 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:22:29.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:29 smithi086 ceph-mon[36851]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:29.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:29 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/846904391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:29 smithi060 ceph-mon[32745]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:29.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:29 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/846904391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:22:29.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:29 smithi063 ceph-mon[36717]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:29.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:29 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/846904391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:22:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.107+0000 7fb95c77a700 1 -- 172.21.15.86:0/1262190856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb954101e50 msgr2=0x7fb954102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.107+0000 7fb95c77a700 1 --2- 172.21.15.86:0/1262190856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb954101e50 0x7fb954102270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb950009990 tx=0x7fb950009ca0 comp rx=0 tx=0).stop 2024-03-04T12:22:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.108+0000 7fb95c77a700 1 -- 172.21.15.86:0/1262190856 shutdown_connections 2024-03-04T12:22:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.108+0000 7fb95c77a700 1 --2- 172.21.15.86:0/1262190856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb954103970 0x7fb95410a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.108+0000 7fb95c77a700 1 --2- 172.21.15.86:0/1262190856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb954101e50 0x7fb954102270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.108+0000 7fb95c77a700 1 --2- 172.21.15.86:0/1262190856 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb954102fb0 0x7fb954103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.108+0000 7fb95c77a700 1 -- 172.21.15.86:0/1262190856 >> 172.21.15.86:0/1262190856 conn(0x7fb9540fd430 msgr2=0x7fb9540ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.108+0000 7fb95c77a700 1 -- 172.21.15.86:0/1262190856 shutdown_connections 2024-03-04T12:22:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.108+0000 7fb95c77a700 1 -- 172.21.15.86:0/1262190856 wait complete. 2024-03-04T12:22:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.108+0000 7fb95c77a700 1 Processor -- start 2024-03-04T12:22:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.108+0000 7fb95c77a700 1 -- start start 2024-03-04T12:22:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb95c77a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb954101e50 0x7fb954198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb95c77a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb954102fb0 0x7fb954198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb95c77a700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb954103970 0x7fb95419f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb95c77a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95410d9c0 con 0x7fb954101e50 2024-03-04T12:22:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb95c77a700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95410d840 con 0x7fb954103970 2024-03-04T12:22:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb95c77a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95410db40 con 0x7fb954102fb0 2024-03-04T12:22:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb959d15700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb954102fb0 0x7fb954198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb959d15700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb954102fb0 0x7fb954198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35016/0 (socket says 172.21.15.86:35016) 2024-03-04T12:22:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb959d15700 1 -- 172.21.15.86:0/1854248127 learned_addr learned my addr 172.21.15.86:0/1854248127 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb959d15700 1 -- 172.21.15.86:0/1854248127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb954103970 msgr2=0x7fb95419f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb959d15700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb954103970 0x7fb95419f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb959d15700 1 -- 172.21.15.86:0/1854248127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb954101e50 msgr2=0x7fb954198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb95a516700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb954101e50 0x7fb954198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb959d15700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb954101e50 0x7fb954198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb959d15700 1 -- 172.21.15.86:0/1854248127 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95419fc60 con 0x7fb954102fb0 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb959d15700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb954102fb0 0x7fb954198540 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb94400d980 tx=0x7fb94400dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb94b7fe700 1 -- 172.21.15.86:0/1854248127 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb944016070 con 0x7fb954102fb0 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb94b7fe700 1 -- 172.21.15.86:0/1854248127 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb944009b40 con 0x7fb954102fb0 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb95c77a700 1 -- 172.21.15.86:0/1854248127 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95419ff50 con 0x7fb954102fb0 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb94b7fe700 1 -- 172.21.15.86:0/1854248127 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94400f620 con 0x7fb954102fb0 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.109+0000 7fb95c77a700 1 -- 172.21.15.86:0/1854248127 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9541a0400 con 0x7fb954102fb0 2024-03-04T12:22:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.110+0000 7fb95c77a700 1 -- 172.21.15.86:0/1854248127 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb954068ae0 con 0x7fb954102fb0 2024-03-04T12:22:31.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.112+0000 7fb94b7fe700 1 -- 172.21.15.86:0/1854248127 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fb944019070 con 0x7fb954102fb0 2024-03-04T12:22:31.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.112+0000 7fb94b7fe700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb9400a1e10 0x7fb9400a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:31.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.113+0000 7fb94b7fe700 1 -- 172.21.15.86:0/1854248127 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb9440cd5c0 con 0x7fb954102fb0 2024-03-04T12:22:31.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.113+0000 7fb95a516700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb9400a1e10 0x7fb9400a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:31.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.113+0000 7fb95a516700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb9400a1e10 0x7fb9400a42d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb9500039c0 tx=0x7fb950005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:31.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.116+0000 7fb94b7fe700 1 -- 172.21.15.86:0/1854248127 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb94409adf0 con 0x7fb954102fb0 2024-03-04T12:22:31.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.317+0000 7fb95c77a700 1 -- 172.21.15.86:0/1854248127 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fb954062520 con 0x7fb9400a1e10 2024-03-04T12:22:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.325+0000 7fb94b7fe700 1 -- 172.21.15.86:0/1854248127 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fb954062520 con 0x7fb9400a1e10 2024-03-04T12:22:31.331 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mds.all update... 2024-03-04T12:22:31.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.327+0000 7fb95c77a700 1 -- 172.21.15.86:0/1854248127 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb9400a1e10 msgr2=0x7fb9400a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:31.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.327+0000 7fb95c77a700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb9400a1e10 0x7fb9400a42d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb9500039c0 tx=0x7fb950005fb0 comp rx=0 tx=0).stop 2024-03-04T12:22:31.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.327+0000 7fb95c77a700 1 -- 172.21.15.86:0/1854248127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb954102fb0 msgr2=0x7fb954198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:31.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.328+0000 7fb95c77a700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb954102fb0 0x7fb954198540 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb94400d980 tx=0x7fb94400dd40 comp rx=0 tx=0).stop 2024-03-04T12:22:31.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.328+0000 7fb95c77a700 1 -- 172.21.15.86:0/1854248127 shutdown_connections 2024-03-04T12:22:31.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.328+0000 7fb95c77a700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb954103970 0x7fb95419f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:31.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.328+0000 7fb95c77a700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb9400a1e10 0x7fb9400a42d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:31.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.328+0000 7fb95c77a700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb954101e50 0x7fb954198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:31.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.328+0000 7fb95c77a700 1 --2- 172.21.15.86:0/1854248127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb954102fb0 0x7fb954198540 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:31.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.328+0000 7fb95c77a700 1 -- 172.21.15.86:0/1854248127 >> 172.21.15.86:0/1854248127 conn(0x7fb9540fd430 msgr2=0x7fb954106340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:31.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.328+0000 7fb95c77a700 1 -- 172.21.15.86:0/1854248127 shutdown_connections 2024-03-04T12:22:31.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-04T12:22:31.328+0000 7fb95c77a700 1 -- 172.21.15.86:0/1854248127 wait complete. 2024-03-04T12:22:31.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:31 smithi086 ceph-mon[36851]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:31.597 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:31 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:31.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:31 smithi060 ceph-mon[32745]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:31.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:31 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:31.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:31 smithi063 ceph-mon[36717]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:31.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:31 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:31.960 DEBUG:teuthology.orchestra.run.smithi060:mds.a> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.a.service 2024-03-04T12:22:31.963 DEBUG:teuthology.orchestra.run.smithi060:mds.d> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.d.service 2024-03-04T12:22:32.006 DEBUG:teuthology.orchestra.run.smithi060:mds.g> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.g.service 2024-03-04T12:22:32.049 DEBUG:teuthology.orchestra.run.smithi060:mds.j> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.j.service 2024-03-04T12:22:32.093 DEBUG:teuthology.orchestra.run.smithi063:mds.b> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.b.service 2024-03-04T12:22:32.096 DEBUG:teuthology.orchestra.run.smithi063:mds.e> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.e.service 2024-03-04T12:22:32.139 DEBUG:teuthology.orchestra.run.smithi063:mds.h> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.h.service 2024-03-04T12:22:32.181 DEBUG:teuthology.orchestra.run.smithi063:mds.k> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.k.service 2024-03-04T12:22:32.224 DEBUG:teuthology.orchestra.run.smithi086:mds.c> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.c.service 2024-03-04T12:22:32.227 DEBUG:teuthology.orchestra.run.smithi086:mds.f> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.f.service 2024-03-04T12:22:32.270 DEBUG:teuthology.orchestra.run.smithi086:mds.i> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.i.service 2024-03-04T12:22:32.313 DEBUG:teuthology.orchestra.run.smithi086:mds.l> sudo journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.l.service 2024-03-04T12:22:32.357 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-04T12:22:32.358 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:22:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.466+0000 7fd165644640 1 -- 172.21.15.60:0/1134499941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd160101e40 msgr2=0x7fd160104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.466+0000 7fd165644640 1 --2- 172.21.15.60:0/1134499941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd160101e40 0x7fd160104230 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd158009900 tx=0x7fd15802bc30 comp rx=0 tx=0).stop 2024-03-04T12:22:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.466+0000 7fd165644640 1 -- 172.21.15.60:0/1134499941 shutdown_connections 2024-03-04T12:22:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.466+0000 7fd165644640 1 --2- 172.21.15.60:0/1134499941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1601070a0 0x7fd160109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.466+0000 7fd165644640 1 --2- 172.21.15.60:0/1134499941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd160104770 0x7fd160106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.466+0000 7fd165644640 1 --2- 172.21.15.60:0/1134499941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd160101e40 0x7fd160104230 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.466+0000 7fd165644640 1 -- 172.21.15.60:0/1134499941 >> 172.21.15.60:0/1134499941 conn(0x7fd1600fb7d0 msgr2=0x7fd1600fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.466+0000 7fd165644640 1 -- 172.21.15.60:0/1134499941 shutdown_connections 2024-03-04T12:22:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.466+0000 7fd165644640 1 -- 172.21.15.60:0/1134499941 wait complete. 2024-03-04T12:22:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.466+0000 7fd165644640 1 Processor -- start 2024-03-04T12:22:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.466+0000 7fd165644640 1 -- start start 2024-03-04T12:22:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd165644640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd160101e40 0x7fd160199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd165644640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd160104770 0x7fd160199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd165644640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1601070a0 0x7fd1601a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd165644640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601011a0 con 0x7fd160101e40 2024-03-04T12:22:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd165644640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd160101020 con 0x7fd1601070a0 2024-03-04T12:22:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd165644640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd160101320 con 0x7fd160104770 2024-03-04T12:22:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd15effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd160101e40 0x7fd160199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd15effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd160101e40 0x7fd160199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59232/0 (socket says 172.21.15.60:59232) 2024-03-04T12:22:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd15effd640 1 -- 172.21.15.60:0/4162554632 learned_addr learned my addr 172.21.15.60:0/4162554632 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd15f7fe640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1601070a0 0x7fd1601a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd15e7fc640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd160104770 0x7fd160199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd15effd640 1 -- 172.21.15.60:0/4162554632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd160104770 msgr2=0x7fd160199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd15effd640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd160104770 0x7fd160199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd15effd640 1 -- 172.21.15.60:0/4162554632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1601070a0 msgr2=0x7fd1601a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd15effd640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1601070a0 0x7fd1601a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd15effd640 1 -- 172.21.15.60:0/4162554632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1601a14a0 con 0x7fd160101e40 2024-03-04T12:22:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd15e7fc640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd160104770 0x7fd160199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:22:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd15effd640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd160101e40 0x7fd160199880 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd158000c00 tx=0x7fd158005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd157fff640 1 -- 172.21.15.60:0/4162554632 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158043020 con 0x7fd160101e40 2024-03-04T12:22:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd165644640 1 -- 172.21.15.60:0/4162554632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1601a1730 con 0x7fd160101e40 2024-03-04T12:22:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd157fff640 1 -- 172.21.15.60:0/4162554632 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1580054e0 con 0x7fd160101e40 2024-03-04T12:22:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd157fff640 1 -- 172.21.15.60:0/4162554632 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158003e60 con 0x7fd160101e40 2024-03-04T12:22:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.467+0000 7fd165644640 1 -- 172.21.15.60:0/4162554632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1601a1c80 con 0x7fd160101e40 2024-03-04T12:22:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.468+0000 7fd165644640 1 -- 172.21.15.60:0/4162554632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1600fd2c0 con 0x7fd160101e40 2024-03-04T12:22:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.469+0000 7fd157fff640 1 -- 172.21.15.60:0/4162554632 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fd158046020 con 0x7fd160101e40 2024-03-04T12:22:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.470+0000 7fd157fff640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd14c0a1d80 0x7fd14c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.470+0000 7fd15e7fc640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd14c0a1d80 0x7fd14c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.470+0000 7fd157fff640 1 -- 172.21.15.60:0/4162554632 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd1580efb70 con 0x7fd160101e40 2024-03-04T12:22:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.471+0000 7fd15e7fc640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd14c0a1d80 0x7fd14c0a4240 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd1480058c0 tx=0x7fd148005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.475+0000 7fd157fff640 1 -- 172.21.15.60:0/4162554632 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd15804d020 con 0x7fd160101e40 2024-03-04T12:22:32.594 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:32 smithi086 ceph-mon[36851]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:22:32.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:32 smithi086 ceph-mon[36851]: Saving service mds.all spec with placement smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l;count:12 2024-03-04T12:22:32.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:22:32.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:32.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:22:32.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:32.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:32.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:32.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:32 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:32.595 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:32 smithi086 ceph-mon[36851]: Deploying daemon mds.l on smithi086 2024-03-04T12:22:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:32 smithi063 ceph-mon[36717]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:22:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:32 smithi063 ceph-mon[36717]: Saving service mds.all spec with placement smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l;count:12 2024-03-04T12:22:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:22:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:22:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:32 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:32.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:32 smithi063 ceph-mon[36717]: Deploying daemon mds.l on smithi086 2024-03-04T12:22:32.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.656+0000 7fd165644640 1 -- 172.21.15.60:0/4162554632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd160061f30 con 0x7fd160101e40 2024-03-04T12:22:32.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.657+0000 7fd157fff640 1 -- 172.21.15.60:0/4162554632 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18325 (secure 0 0 0) 0x7fd1580bd440 con 0x7fd160101e40 2024-03-04T12:22:32.663 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:32.663 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":81,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:22:20.316992+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:22:32.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.659+0000 7fd165644640 1 -- 172.21.15.60:0/4162554632 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd14c0a1d80 msgr2=0x7fd14c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:32.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.659+0000 7fd165644640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd14c0a1d80 0x7fd14c0a4240 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd1480058c0 tx=0x7fd148005850 comp rx=0 tx=0).stop 2024-03-04T12:22:32.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.659+0000 7fd165644640 1 -- 172.21.15.60:0/4162554632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd160101e40 msgr2=0x7fd160199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:32.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.659+0000 7fd165644640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd160101e40 0x7fd160199880 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd158000c00 tx=0x7fd158005dd0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.659+0000 7fd165644640 1 -- 172.21.15.60:0/4162554632 shutdown_connections 2024-03-04T12:22:32.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.659+0000 7fd165644640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd14c0a1d80 0x7fd14c0a4240 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.659+0000 7fd165644640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1601070a0 0x7fd1601a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.659+0000 7fd165644640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd160104770 0x7fd160199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.659+0000 7fd165644640 1 --2- 172.21.15.60:0/4162554632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd160101e40 0x7fd160199880 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.659+0000 7fd165644640 1 -- 172.21.15.60:0/4162554632 >> 172.21.15.60:0/4162554632 conn(0x7fd1600fb7d0 msgr2=0x7fd160102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:32.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.660+0000 7fd165644640 1 -- 172.21.15.60:0/4162554632 shutdown_connections 2024-03-04T12:22:32.666 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.660+0000 7fd165644640 1 -- 172.21.15.60:0/4162554632 wait complete. 2024-03-04T12:22:32.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:32 smithi060 ceph-mon[32745]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:22:32.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:32 smithi060 ceph-mon[32745]: Saving service mds.all spec with placement smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l;count:12 2024-03-04T12:22:32.675 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:22:32.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:32.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:22:32.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:32.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:32.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:32.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:32 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:32.676 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:32 smithi060 ceph-mon[32745]: Deploying daemon mds.l on smithi086 2024-03-04T12:22:32.677 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T12:22:32.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:22:32.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.777+0000 7f2f07d59640 1 -- 172.21.15.60:0/3450704504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f00101440 msgr2=0x7f2f001018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:32.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.777+0000 7f2f07d59640 1 --2- 172.21.15.60:0/3450704504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f00101440 0x7f2f001018c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009900 tx=0x7f2efc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.777+0000 7f2f07d59640 1 -- 172.21.15.60:0/3450704504 shutdown_connections 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.777+0000 7f2f07d59640 1 --2- 172.21.15.60:0/3450704504 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f00101e00 0x7f2f00108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.777+0000 7f2f07d59640 1 --2- 172.21.15.60:0/3450704504 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f00101440 0x7f2f001018c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.777+0000 7f2f07d59640 1 --2- 172.21.15.60:0/3450704504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f00100250 0x7f2f00100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.777+0000 7f2f07d59640 1 -- 172.21.15.60:0/3450704504 >> 172.21.15.60:0/3450704504 conn(0x7f2f000fb7e0 msgr2=0x7f2f000fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.777+0000 7f2f07d59640 1 -- 172.21.15.60:0/3450704504 shutdown_connections 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.777+0000 7f2f07d59640 1 -- 172.21.15.60:0/3450704504 wait complete. 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.777+0000 7f2f07d59640 1 Processor -- start 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.777+0000 7f2f07d59640 1 -- start start 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f07d59640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f00100250 0x7f2f001976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f07d59640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f00101440 0x7f2f00197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f07d59640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f00101e00 0x7f2f0019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:32.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f07d59640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0010bee0 con 0x7f2f00101440 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f07d59640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0010bd60 con 0x7f2f00101e00 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f07d59640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0010c060 con 0x7f2f00100250 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f052f4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f00101440 0x7f2f00197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f052f4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f00101440 0x7f2f00197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59264/0 (socket says 172.21.15.60:59264) 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f052f4640 1 -- 172.21.15.60:0/2696241029 learned_addr learned my addr 172.21.15.60:0/2696241029 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f062f6640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f00101e00 0x7f2f0019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f05af5640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f00100250 0x7f2f001976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f052f4640 1 -- 172.21.15.60:0/2696241029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f00100250 msgr2=0x7f2f001976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f052f4640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f00100250 0x7f2f001976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f052f4640 1 -- 172.21.15.60:0/2696241029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f00101e00 msgr2=0x7f2f0019ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f052f4640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f00101e00 0x7f2f0019ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f052f4640 1 -- 172.21.15.60:0/2696241029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0019f2a0 con 0x7f2f00101440 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f062f6640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f00101e00 0x7f2f0019ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f052f4640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f00101440 0x7f2f00197c30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009230 tx=0x7f2efc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:32.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2ef2ffd640 1 -- 172.21.15.60:0/2696241029 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc041070 con 0x7f2f00101440 2024-03-04T12:22:32.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f07d59640 1 -- 172.21.15.60:0/2696241029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f0019f530 con 0x7f2f00101440 2024-03-04T12:22:32.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2ef2ffd640 1 -- 172.21.15.60:0/2696241029 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2efc0044d0 con 0x7f2f00101440 2024-03-04T12:22:32.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2ef2ffd640 1 -- 172.21.15.60:0/2696241029 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc034440 con 0x7f2f00101440 2024-03-04T12:22:32.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.778+0000 7f2f07d59640 1 -- 172.21.15.60:0/2696241029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f0019fa20 con 0x7f2f00101440 2024-03-04T12:22:32.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.779+0000 7f2f07d59640 1 -- 172.21.15.60:0/2696241029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0019fd40 con 0x7f2f00101440 2024-03-04T12:22:32.787 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.780+0000 7f2ef2ffd640 1 -- 172.21.15.60:0/2696241029 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f2efc044070 con 0x7f2f00101440 2024-03-04T12:22:32.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.781+0000 7f2ef2ffd640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2eec0a1d80 0x7f2eec0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:32.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.781+0000 7f2f05af5640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2eec0a1d80 0x7f2eec0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:32.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.781+0000 7f2ef2ffd640 1 -- 172.21.15.60:0/2696241029 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2efc0ef710 con 0x7f2f00101440 2024-03-04T12:22:32.788 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.781+0000 7f2f05af5640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2eec0a1d80 0x7f2eec0a4240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2f001012a0 tx=0x7f2ef4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:32.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.785+0000 7f2ef2ffd640 1 -- 172.21.15.60:0/2696241029 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2efc0bcf40 con 0x7f2f00101440 2024-03-04T12:22:32.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.969+0000 7f2f07d59640 1 -- 172.21.15.60:0/2696241029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2f001a0350 con 0x7f2f00101440 2024-03-04T12:22:32.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.969+0000 7f2ef2ffd640 1 -- 172.21.15.60:0/2696241029 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2efc0bc840 con 0x7f2f00101440 2024-03-04T12:22:32.975 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:22:32.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.972+0000 7f2f07d59640 1 -- 172.21.15.60:0/2696241029 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2eec0a1d80 msgr2=0x7f2eec0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:32.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.972+0000 7f2f07d59640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2eec0a1d80 0x7f2eec0a4240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2f001012a0 tx=0x7f2ef4005f90 comp rx=0 tx=0).stop 2024-03-04T12:22:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.972+0000 7f2f07d59640 1 -- 172.21.15.60:0/2696241029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f00101440 msgr2=0x7f2f00197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.972+0000 7f2f07d59640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f00101440 0x7f2f00197c30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2efc009230 tx=0x7f2efc005d50 comp rx=0 tx=0).stop 2024-03-04T12:22:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.972+0000 7f2f07d59640 1 -- 172.21.15.60:0/2696241029 shutdown_connections 2024-03-04T12:22:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.972+0000 7f2f07d59640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2eec0a1d80 0x7f2eec0a4240 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.972+0000 7f2f07d59640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f00101e00 0x7f2f0019ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.972+0000 7f2f07d59640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f00101440 0x7f2f00197c30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.972+0000 7f2f07d59640 1 --2- 172.21.15.60:0/2696241029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f00100250 0x7f2f001976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.972+0000 7f2f07d59640 1 -- 172.21.15.60:0/2696241029 >> 172.21.15.60:0/2696241029 conn(0x7f2f000fb7e0 msgr2=0x7f2f000fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.972+0000 7f2f07d59640 1 -- 172.21.15.60:0/2696241029 shutdown_connections 2024-03-04T12:22:32.978 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:32.972+0000 7f2f07d59640 1 -- 172.21.15.60:0/2696241029 wait complete. 2024-03-04T12:22:32.988 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:22:33.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.085+0000 7f27b88dc640 1 -- 172.21.15.60:0/52871411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b4100270 msgr2=0x7f27b4100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:33.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 --2- 172.21.15.60:0/52871411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b4100270 0x7f27b4100690 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f27a4009900 tx=0x7f27a402bc30 comp rx=0 tx=0).stop 2024-03-04T12:22:33.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 -- 172.21.15.60:0/52871411 shutdown_connections 2024-03-04T12:22:33.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 --2- 172.21.15.60:0/52871411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b4101e20 0x7f27b41086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:33.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 --2- 172.21.15.60:0/52871411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b4101460 0x7f27b41018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:33.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 --2- 172.21.15.60:0/52871411 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b4100270 0x7f27b4100690 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:33.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 -- 172.21.15.60:0/52871411 >> 172.21.15.60:0/52871411 conn(0x7f27b40fb7c0 msgr2=0x7f27b40fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:33.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 -- 172.21.15.60:0/52871411 shutdown_connections 2024-03-04T12:22:33.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 -- 172.21.15.60:0/52871411 wait complete. 2024-03-04T12:22:33.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 Processor -- start 2024-03-04T12:22:33.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 -- start start 2024-03-04T12:22:33.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b4100270 0x7f27b41975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:33.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b4101460 0x7f27b4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:33.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b4101e20 0x7f27b419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.086+0000 7f27b88dc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b410bf00 con 0x7f27b4101460 2024-03-04T12:22:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27b88dc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b410bd80 con 0x7f27b4100270 2024-03-04T12:22:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27b88dc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27b410c080 con 0x7f27b4101e20 2024-03-04T12:22:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27abfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b4101460 0x7f27b4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27abfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b4101460 0x7f27b4197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59284/0 (socket says 172.21.15.60:59284) 2024-03-04T12:22:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27abfff640 1 -- 172.21.15.60:0/1595823285 learned_addr learned my addr 172.21.15.60:0/1595823285 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27b2d9d640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b4101e20 0x7f27b419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27b259c640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b4100270 0x7f27b41975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:33.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27abfff640 1 -- 172.21.15.60:0/1595823285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b4101e20 msgr2=0x7f27b419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27abfff640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b4101e20 0x7f27b419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27abfff640 1 -- 172.21.15.60:0/1595823285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b4100270 msgr2=0x7f27b41975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27abfff640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b4100270 0x7f27b41975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27abfff640 1 -- 172.21.15.60:0/1595823285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b419f1d0 con 0x7f27b4101460 2024-03-04T12:22:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27abfff640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b4101460 0x7f27b4197af0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f279c00c920 tx=0x7f279c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27ab7fe640 1 -- 172.21.15.60:0/1595823285 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c007c70 con 0x7f27b4101460 2024-03-04T12:22:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27ab7fe640 1 -- 172.21.15.60:0/1595823285 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f279c007e10 con 0x7f27b4101460 2024-03-04T12:22:33.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27b88dc640 1 -- 172.21.15.60:0/1595823285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b419f4c0 con 0x7f27b4101460 2024-03-04T12:22:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27ab7fe640 1 -- 172.21.15.60:0/1595823285 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c018890 con 0x7f27b4101460 2024-03-04T12:22:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.087+0000 7f27b88dc640 1 -- 172.21.15.60:0/1595823285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b419fa10 con 0x7f27b4101460 2024-03-04T12:22:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.088+0000 7f27b88dc640 1 -- 172.21.15.60:0/1595823285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b419fcd0 con 0x7f27b4101460 2024-03-04T12:22:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.089+0000 7f27ab7fe640 1 -- 172.21.15.60:0/1595823285 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f279c018a70 con 0x7f27b4101460 2024-03-04T12:22:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.090+0000 7f27ab7fe640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f27a00a1d80 0x7f27a00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.090+0000 7f27b259c640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f27a00a1d80 0x7f27a00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.090+0000 7f27ab7fe640 1 -- 172.21.15.60:0/1595823285 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f279c0cdab0 con 0x7f27b4101460 2024-03-04T12:22:33.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.090+0000 7f27b259c640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f27a00a1d80 0x7f27a00a4240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f27a4036040 tx=0x7f27a4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:33.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.094+0000 7f27ab7fe640 1 -- 172.21.15.60:0/1595823285 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f279c014030 con 0x7f27b4101460 2024-03-04T12:22:33.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.318+0000 7f27b88dc640 1 -- 172.21.15.60:0/1595823285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27b419fea0 con 0x7f27b4101460 2024-03-04T12:22:33.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.318+0000 7f27ab7fe640 1 -- 172.21.15.60:0/1595823285 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f279c00de20 con 0x7f27b4101460 2024-03-04T12:22:33.324 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:33.324 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-04T12:22:33.324 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 1 2024-03-04T12:22:33.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.320+0000 7f27b88dc640 1 -- 172.21.15.60:0/1595823285 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f27a00a1d80 msgr2=0x7f27a00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:33.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.320+0000 7f27b88dc640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f27a00a1d80 0x7f27a00a4240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f27a4036040 tx=0x7f27a4005fd0 comp rx=0 tx=0).stop 2024-03-04T12:22:33.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.320+0000 7f27b88dc640 1 -- 172.21.15.60:0/1595823285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b4101460 msgr2=0x7f27b4197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:33.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.320+0000 7f27b88dc640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b4101460 0x7f27b4197af0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f279c00c920 tx=0x7f279c00cde0 comp rx=0 tx=0).stop 2024-03-04T12:22:33.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.320+0000 7f27b88dc640 1 -- 172.21.15.60:0/1595823285 shutdown_connections 2024-03-04T12:22:33.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.320+0000 7f27b88dc640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f27a00a1d80 0x7f27a00a4240 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:33.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.320+0000 7f27b88dc640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27b4101e20 0x7f27b419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:33.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.320+0000 7f27b88dc640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27b4101460 0x7f27b4197af0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:33.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.320+0000 7f27b88dc640 1 --2- 172.21.15.60:0/1595823285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27b4100270 0x7f27b41975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:33.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.320+0000 7f27b88dc640 1 -- 172.21.15.60:0/1595823285 >> 172.21.15.60:0/1595823285 conn(0x7f27b40fb7c0 msgr2=0x7f27b40694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:33.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.320+0000 7f27b88dc640 1 -- 172.21.15.60:0/1595823285 shutdown_connections 2024-03-04T12:22:33.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:33.320+0000 7f27b88dc640 1 -- 172.21.15.60:0/1595823285 wait complete. 2024-03-04T12:22:33.666 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:33 smithi086 ceph-mon[36851]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:33.666 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:33 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/4162554632' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:22:33.666 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:33 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2696241029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:22:33.666 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:33 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1595823285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:33.677 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:33 smithi060 ceph-mon[32745]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:33.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:33 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/4162554632' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:22:33.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:33 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2696241029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:22:33.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:33 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1595823285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:33.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:33 smithi063 ceph-mon[36717]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:33.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:33 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/4162554632' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:22:33.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:33 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2696241029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:22:33.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:33 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1595823285' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:34.340 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:22:34 smithi086 systemd[1]: Starting Ceph mds.l for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:22:34.598 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:22:34 smithi086 podman[67200]: 2024-03-04 12:22:34.23752441 +0000 UTC m=+0.017443503 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:22:34.598 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:22:34 smithi086 podman[67200]: 2024-03-04 12:22:34.338899201 +0000 UTC m=+0.118818328 container create 20368d8c0c29ca80f03bf1065d2fbb7864a9b627b3c0c2ddc807b3c42eadd0cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-l, ceph=True, org.label-schema.build-date=20240222, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:22:34.598 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:22:34 smithi086 podman[67200]: 2024-03-04 12:22:34.575439364 +0000 UTC m=+0.355358465 container init 20368d8c0c29ca80f03bf1065d2fbb7864a9b627b3c0c2ddc807b3c42eadd0cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-l, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-04T12:22:34.598 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:22:34 smithi086 podman[67200]: 2024-03-04 12:22:34.578098467 +0000 UTC m=+0.358017551 container start 20368d8c0c29ca80f03bf1065d2fbb7864a9b627b3c0c2ddc807b3c42eadd0cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-l, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-04T12:22:35.097 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:22:34 smithi086 ceph-mds[67220]: -- 172.21.15.86:0/3559613611 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5620101fba40 con 0x5620101ce800 2024-03-04T12:22:35.098 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:22:34 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-l[67216]: starting mds.l at 2024-03-04T12:22:35.098 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:22:34 smithi086 bash[67200]: 20368d8c0c29ca80f03bf1065d2fbb7864a9b627b3c0c2ddc807b3c42eadd0cb 2024-03-04T12:22:35.098 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:22:34 smithi086 systemd[1]: Started Ceph mds.l for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:22:35.338 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:22:35.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.448+0000 7fb67df89640 1 -- 172.21.15.60:0/2980469969 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb678100260 msgr2=0x7fb678100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:35.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.448+0000 7fb67df89640 1 --2- 172.21.15.60:0/2980469969 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb678100260 0x7fb678100680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb660009900 tx=0x7fb66002bc30 comp rx=0 tx=0).stop 2024-03-04T12:22:35.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.448+0000 7fb67df89640 1 -- 172.21.15.60:0/2980469969 shutdown_connections 2024-03-04T12:22:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.448+0000 7fb67df89640 1 --2- 172.21.15.60:0/2980469969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb678101e10 0x7fb6781086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.448+0000 7fb67df89640 1 --2- 172.21.15.60:0/2980469969 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb678101450 0x7fb6781018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.448+0000 7fb67df89640 1 --2- 172.21.15.60:0/2980469969 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb678100260 0x7fb678100680 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.448+0000 7fb67df89640 1 -- 172.21.15.60:0/2980469969 >> 172.21.15.60:0/2980469969 conn(0x7fb6780fb7d0 msgr2=0x7fb6780fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.448+0000 7fb67df89640 1 -- 172.21.15.60:0/2980469969 shutdown_connections 2024-03-04T12:22:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.448+0000 7fb67df89640 1 -- 172.21.15.60:0/2980469969 wait complete. 2024-03-04T12:22:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.448+0000 7fb67df89640 1 Processor -- start 2024-03-04T12:22:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.448+0000 7fb67df89640 1 -- start start 2024-03-04T12:22:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb67df89640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb678100260 0x7fb6781976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb67df89640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb678101450 0x7fb678197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb67df89640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb678101e10 0x7fb67819ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:35.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb67df89640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67810bf10 con 0x7fb678101e10 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb67df89640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67810bd90 con 0x7fb678100260 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb67df89640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67810c090 con 0x7fb678101450 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb677fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb678101e10 0x7fb67819ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb677fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb678101e10 0x7fb67819ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34718/0 (socket says 172.21.15.60:34718) 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb677fff640 1 -- 172.21.15.60:0/3522713246 learned_addr learned my addr 172.21.15.60:0/3522713246 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb676ffd640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb678101450 0x7fb678197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb6777fe640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb678100260 0x7fb6781976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb677fff640 1 -- 172.21.15.60:0/3522713246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb678101450 msgr2=0x7fb678197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb677fff640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb678101450 0x7fb678197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb677fff640 1 -- 172.21.15.60:0/3522713246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb678100260 msgr2=0x7fb6781976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb677fff640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb678100260 0x7fb6781976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb677fff640 1 -- 172.21.15.60:0/3522713246 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67819f2f0 con 0x7fb678101e10 2024-03-04T12:22:35.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb676ffd640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb678101450 0x7fb678197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:22:35.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb677fff640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb678101e10 0x7fb67819ec80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb670009fd0 tx=0x7fb67000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:35.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.449+0000 7fb674ff9640 1 -- 172.21.15.60:0/3522713246 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb670017070 con 0x7fb678101e10 2024-03-04T12:22:35.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.450+0000 7fb674ff9640 1 -- 172.21.15.60:0/3522713246 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb670003e50 con 0x7fb678101e10 2024-03-04T12:22:35.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.450+0000 7fb67df89640 1 -- 172.21.15.60:0/3522713246 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67819f5e0 con 0x7fb678101e10 2024-03-04T12:22:35.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.450+0000 7fb674ff9640 1 -- 172.21.15.60:0/3522713246 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6700106c0 con 0x7fb678101e10 2024-03-04T12:22:35.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.450+0000 7fb67df89640 1 -- 172.21.15.60:0/3522713246 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67819fae0 con 0x7fb678101e10 2024-03-04T12:22:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.451+0000 7fb67df89640 1 -- 172.21.15.60:0/3522713246 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67819fd40 con 0x7fb678101e10 2024-03-04T12:22:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.452+0000 7fb674ff9640 1 -- 172.21.15.60:0/3522713246 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fb670003970 con 0x7fb678101e10 2024-03-04T12:22:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.452+0000 7fb674ff9640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb6640a1d80 0x7fb6640a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.452+0000 7fb6777fe640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb6640a1d80 0x7fb6640a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.453+0000 7fb674ff9640 1 -- 172.21.15.60:0/3522713246 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb6700cd6d0 con 0x7fb678101e10 2024-03-04T12:22:35.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.453+0000 7fb6777fe640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb6640a1d80 0x7fb6640a4240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb66002c170 tx=0x7fb66002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:35.462 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.457+0000 7fb674ff9640 1 -- 172.21.15.60:0/3522713246 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb67009a8b0 con 0x7fb678101e10 2024-03-04T12:22:35.668 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:35 smithi063 ceph-mon[36717]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:35.668 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:35 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] up:boot 2024-03-04T12:22:35.668 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:35 smithi063 ceph-mon[36717]: fsmap 1 up:standby 2024-03-04T12:22:35.668 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-04T12:22:35.669 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:35.669 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:35.669 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:35.669 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:35.669 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:35.669 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:35 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:35.669 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:35 smithi063 ceph-mon[36717]: Deploying daemon mds.k on smithi063 2024-03-04T12:22:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.679+0000 7fb67df89640 1 -- 172.21.15.60:0/3522713246 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6781a00d0 con 0x7fb678101e10 2024-03-04T12:22:35.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.680+0000 7fb674ff9640 1 -- 172.21.15.60:0/3522713246 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1322 (secure 0 0 0) 0x7fb67009e950 con 0x7fb678101e10 2024-03-04T12:22:35.685 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:35.685 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-04T12:22:35.686 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 2 2024-03-04T12:22:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.682+0000 7fb67df89640 1 -- 172.21.15.60:0/3522713246 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb6640a1d80 msgr2=0x7fb6640a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.682+0000 7fb67df89640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb6640a1d80 0x7fb6640a4240 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb66002c170 tx=0x7fb66002df90 comp rx=0 tx=0).stop 2024-03-04T12:22:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.682+0000 7fb67df89640 1 -- 172.21.15.60:0/3522713246 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb678101e10 msgr2=0x7fb67819ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.682+0000 7fb67df89640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb678101e10 0x7fb67819ec80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb670009fd0 tx=0x7fb67000b260 comp rx=0 tx=0).stop 2024-03-04T12:22:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.682+0000 7fb67df89640 1 -- 172.21.15.60:0/3522713246 shutdown_connections 2024-03-04T12:22:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.682+0000 7fb67df89640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb6640a1d80 0x7fb6640a4240 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.682+0000 7fb67df89640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb678101e10 0x7fb67819ec80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.682+0000 7fb67df89640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb678101450 0x7fb678197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.682+0000 7fb67df89640 1 --2- 172.21.15.60:0/3522713246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb678100260 0x7fb6781976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.682+0000 7fb67df89640 1 -- 172.21.15.60:0/3522713246 >> 172.21.15.60:0/3522713246 conn(0x7fb6780fb7d0 msgr2=0x7fb6780fd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:35.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.682+0000 7fb67df89640 1 -- 172.21.15.60:0/3522713246 shutdown_connections 2024-03-04T12:22:35.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:35.682+0000 7fb67df89640 1 -- 172.21.15.60:0/3522713246 wait complete. 2024-03-04T12:22:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:35 smithi060 ceph-mon[32745]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:35 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] up:boot 2024-03-04T12:22:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:35 smithi060 ceph-mon[32745]: fsmap 1 up:standby 2024-03-04T12:22:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-04T12:22:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:35.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:35.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:35.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:35 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:35.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:35 smithi060 ceph-mon[32745]: Deploying daemon mds.k on smithi063 2024-03-04T12:22:36.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:35 smithi086 ceph-mon[36851]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:36.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:35 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] up:boot 2024-03-04T12:22:36.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:35 smithi086 ceph-mon[36851]: fsmap 1 up:standby 2024-03-04T12:22:36.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-04T12:22:36.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:36.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:36.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:36.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:36.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:36.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:35 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:36.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:35 smithi086 ceph-mon[36851]: Deploying daemon mds.k on smithi063 2024-03-04T12:22:36.724 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:36 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3522713246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:36.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:36 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3522713246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:37.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:36 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3522713246' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:37.700 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:22:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.815+0000 7f722224a640 1 -- 172.21.15.60:0/790740923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721c102560 msgr2=0x7f721c1029e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.815+0000 7f722224a640 1 --2- 172.21.15.60:0/790740923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721c102560 0x7f721c1029e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7204009900 tx=0x7f720402bc30 comp rx=0 tx=0).stop 2024-03-04T12:22:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.815+0000 7f722224a640 1 -- 172.21.15.60:0/790740923 shutdown_connections 2024-03-04T12:22:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.815+0000 7f722224a640 1 --2- 172.21.15.60:0/790740923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f721c102f20 0x7f721c1097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.815+0000 7f722224a640 1 --2- 172.21.15.60:0/790740923 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721c102560 0x7f721c1029e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.815+0000 7f722224a640 1 --2- 172.21.15.60:0/790740923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f721c101370 0x7f721c101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.815+0000 7f722224a640 1 -- 172.21.15.60:0/790740923 >> 172.21.15.60:0/790740923 conn(0x7f721c0fc8c0 msgr2=0x7f721c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.815+0000 7f722224a640 1 -- 172.21.15.60:0/790740923 shutdown_connections 2024-03-04T12:22:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.815+0000 7f722224a640 1 -- 172.21.15.60:0/790740923 wait complete. 2024-03-04T12:22:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.815+0000 7f722224a640 1 Processor -- start 2024-03-04T12:22:37.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f722224a640 1 -- start start 2024-03-04T12:22:37.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f722224a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721c101370 0x7f721c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:37.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f722224a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f721c102560 0x7f721c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:37.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f722224a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f721c102f20 0x7f721c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f722224a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c10d000 con 0x7f721c101370 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f722224a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c10ce80 con 0x7f721c102560 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f722224a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721c10d180 con 0x7f721c102f20 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f721b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721c101370 0x7f721c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f721b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721c101370 0x7f721c1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34732/0 (socket says 172.21.15.60:34732) 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f721b7fe640 1 -- 172.21.15.60:0/730487488 learned_addr learned my addr 172.21.15.60:0/730487488 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f721bfff640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f721c102f20 0x7f721c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f721affd640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f721c102560 0x7f721c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f721b7fe640 1 -- 172.21.15.60:0/730487488 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f721c102f20 msgr2=0x7f721c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f721b7fe640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f721c102f20 0x7f721c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f721b7fe640 1 -- 172.21.15.60:0/730487488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f721c102560 msgr2=0x7f721c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f721b7fe640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f721c102560 0x7f721c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:37.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f721b7fe640 1 -- 172.21.15.60:0/730487488 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c19f200 con 0x7f721c101370 2024-03-04T12:22:37.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.816+0000 7f721affd640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f721c102560 0x7f721c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:22:37.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.817+0000 7f721b7fe640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721c101370 0x7f721c1975e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f720c00ccb0 tx=0x7f720c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:37.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.817+0000 7f7218ff9640 1 -- 172.21.15.60:0/730487488 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c003990 con 0x7f721c101370 2024-03-04T12:22:37.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.817+0000 7f722224a640 1 -- 172.21.15.60:0/730487488 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721c19f4f0 con 0x7f721c101370 2024-03-04T12:22:37.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.817+0000 7f7218ff9640 1 -- 172.21.15.60:0/730487488 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f720c003b30 con 0x7f721c101370 2024-03-04T12:22:37.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.817+0000 7f7218ff9640 1 -- 172.21.15.60:0/730487488 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c00f7b0 con 0x7f721c101370 2024-03-04T12:22:37.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.817+0000 7f722224a640 1 -- 172.21.15.60:0/730487488 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721c19fa60 con 0x7f721c101370 2024-03-04T12:22:37.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.818+0000 7f722224a640 1 -- 172.21.15.60:0/730487488 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721c108690 con 0x7f721c101370 2024-03-04T12:22:37.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.819+0000 7f7218ff9640 1 -- 172.21.15.60:0/730487488 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f720c00f9b0 con 0x7f721c101370 2024-03-04T12:22:37.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.820+0000 7f7218ff9640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72080a1d80 0x7f72080a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:37.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.820+0000 7f7218ff9640 1 -- 172.21.15.60:0/730487488 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f720c0cd9e0 con 0x7f721c101370 2024-03-04T12:22:37.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.820+0000 7f721affd640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72080a1d80 0x7f72080a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:37.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.820+0000 7f721affd640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72080a1d80 0x7f72080a4240 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7204006010 tx=0x7f720402c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:37.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:37.824+0000 7f7218ff9640 1 -- 172.21.15.60:0/730487488 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f720c014030 con 0x7f721c101370 2024-03-04T12:22:37.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:37 smithi063 ceph-mon[36717]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:37.851 INFO:journalctl@ceph.mds.k.smithi063.stdout:Mar 04 12:22:37 smithi063 systemd[1]: Started Ceph mds.k for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:22:37.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:37 smithi060 ceph-mon[32745]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:38.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.046+0000 7f722224a640 1 -- 172.21.15.60:0/730487488 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f721c02ceb0 con 0x7f721c101370 2024-03-04T12:22:38.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.046+0000 7f7218ff9640 1 -- 172.21.15.60:0/730487488 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1322 (secure 0 0 0) 0x7f720c09b2b0 con 0x7f721c101370 2024-03-04T12:22:38.052 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:38.052 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-04T12:22:38.053 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 2 2024-03-04T12:22:38.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.048+0000 7f722224a640 1 -- 172.21.15.60:0/730487488 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72080a1d80 msgr2=0x7f72080a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:38.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.048+0000 7f722224a640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72080a1d80 0x7f72080a4240 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7204006010 tx=0x7f720402c8a0 comp rx=0 tx=0).stop 2024-03-04T12:22:38.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.048+0000 7f722224a640 1 -- 172.21.15.60:0/730487488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721c101370 msgr2=0x7f721c1975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:38.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.048+0000 7f722224a640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721c101370 0x7f721c1975e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f720c00ccb0 tx=0x7f720c00a620 comp rx=0 tx=0).stop 2024-03-04T12:22:38.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.049+0000 7f722224a640 1 -- 172.21.15.60:0/730487488 shutdown_connections 2024-03-04T12:22:38.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.049+0000 7f722224a640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72080a1d80 0x7f72080a4240 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:38.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.049+0000 7f722224a640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f721c102f20 0x7f721c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:38.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.049+0000 7f722224a640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f721c102560 0x7f721c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:38.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.049+0000 7f722224a640 1 --2- 172.21.15.60:0/730487488 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f721c101370 0x7f721c1975e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:38.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.049+0000 7f722224a640 1 -- 172.21.15.60:0/730487488 >> 172.21.15.60:0/730487488 conn(0x7f721c0fc8c0 msgr2=0x7f721c105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:38.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.049+0000 7f722224a640 1 -- 172.21.15.60:0/730487488 shutdown_connections 2024-03-04T12:22:38.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:38.049+0000 7f722224a640 1 -- 172.21.15.60:0/730487488 wait complete. 2024-03-04T12:22:38.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:37 smithi086 ceph-mon[36851]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:39.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:38 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:39.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:38 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:39.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:38 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:39.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:38 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:39.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:38 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:39.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:38 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:39.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:38 smithi086 ceph-mon[36851]: Deploying daemon mds.j on smithi060 2024-03-04T12:22:39.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:38 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/730487488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:39.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:38 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] up:boot 2024-03-04T12:22:39.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:38 smithi086 ceph-mon[36851]: fsmap 2 up:standby 2024-03-04T12:22:39.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:38 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-04T12:22:39.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:38 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:39.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:38 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:39.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:38 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:39.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:38 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:39.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:38 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:39.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:38 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:39.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:38 smithi063 ceph-mon[36717]: Deploying daemon mds.j on smithi060 2024-03-04T12:22:39.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:38 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/730487488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:39.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:38 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] up:boot 2024-03-04T12:22:39.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:38 smithi063 ceph-mon[36717]: fsmap 2 up:standby 2024-03-04T12:22:39.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:38 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-04T12:22:39.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:38 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:39.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:38 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:39.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:38 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:39.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:38 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:39.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:38 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:39.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:38 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:39.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:38 smithi060 ceph-mon[32745]: Deploying daemon mds.j on smithi060 2024-03-04T12:22:39.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:38 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/730487488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:39.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:38 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] up:boot 2024-03-04T12:22:39.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:38 smithi060 ceph-mon[32745]: fsmap 2 up:standby 2024-03-04T12:22:39.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:38 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-04T12:22:40.030 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:39 smithi063 ceph-mon[36717]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:40.066 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:22:40.094 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:39 smithi060 ceph-mon[32745]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:40.095 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:22:40 smithi060 systemd[1]: Starting Ceph mds.j for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:22:40.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:39 smithi086 ceph-mon[36851]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:40.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.183+0000 7f83fc0d1640 1 -- 172.21.15.60:0/2557597390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83f4101e50 msgr2=0x7f83f4104240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.183+0000 7f83fc0d1640 1 --2- 172.21.15.60:0/2557597390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83f4101e50 0x7f83f4104240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f83f0009900 tx=0x7f83f002bc30 comp rx=0 tx=0).stop 2024-03-04T12:22:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.183+0000 7f83fc0d1640 1 -- 172.21.15.60:0/2557597390 shutdown_connections 2024-03-04T12:22:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.183+0000 7f83fc0d1640 1 --2- 172.21.15.60:0/2557597390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83f41070b0 0x7f83f4109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.183+0000 7f83fc0d1640 1 --2- 172.21.15.60:0/2557597390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83f4104780 0x7f83f4106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.183+0000 7f83fc0d1640 1 --2- 172.21.15.60:0/2557597390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83f4101e50 0x7f83f4104240 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.183+0000 7f83fc0d1640 1 -- 172.21.15.60:0/2557597390 >> 172.21.15.60:0/2557597390 conn(0x7f83f40fb840 msgr2=0x7f83f40fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.183+0000 7f83fc0d1640 1 -- 172.21.15.60:0/2557597390 shutdown_connections 2024-03-04T12:22:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.183+0000 7f83fc0d1640 1 -- 172.21.15.60:0/2557597390 wait complete. 2024-03-04T12:22:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fc0d1640 1 Processor -- start 2024-03-04T12:22:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fc0d1640 1 -- start start 2024-03-04T12:22:40.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fc0d1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83f4101e50 0x7f83f41017c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fc0d1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83f4104780 0x7f83f40ffe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fc0d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83f41070b0 0x7f83f4100340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fc0d1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f410cff0 con 0x7f83f41070b0 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fc0d1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f410ce70 con 0x7f83f4104780 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fc0d1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f410d170 con 0x7f83f4101e50 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fa66e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83f41070b0 0x7f83f4100340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fa66e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83f41070b0 0x7f83f4100340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34754/0 (socket says 172.21.15.60:34754) 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83f966c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83f4104780 0x7f83f40ffe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fa66e640 1 -- 172.21.15.60:0/869549228 learned_addr learned my addr 172.21.15.60:0/869549228 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83f9e6d640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83f4101e50 0x7f83f41017c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fa66e640 1 -- 172.21.15.60:0/869549228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83f4101e50 msgr2=0x7f83f41017c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fa66e640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83f4101e50 0x7f83f41017c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fa66e640 1 -- 172.21.15.60:0/869549228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83f4104780 msgr2=0x7f83f40ffe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:40.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fa66e640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83f4104780 0x7f83f40ffe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:40.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.184+0000 7f83fa66e640 1 -- 172.21.15.60:0/869549228 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f4100bd0 con 0x7f83f41070b0 2024-03-04T12:22:40.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.185+0000 7f83fa66e640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83f41070b0 0x7f83f4100340 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f83ec009fd0 tx=0x7f83ec00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:40.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.185+0000 7f83eaffd640 1 -- 172.21.15.60:0/869549228 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ec017070 con 0x7f83f41070b0 2024-03-04T12:22:40.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.185+0000 7f83fc0d1640 1 -- 172.21.15.60:0/869549228 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f41a17c0 con 0x7f83f41070b0 2024-03-04T12:22:40.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.185+0000 7f83eaffd640 1 -- 172.21.15.60:0/869549228 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f83ec003e50 con 0x7f83f41070b0 2024-03-04T12:22:40.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.185+0000 7f83eaffd640 1 -- 172.21.15.60:0/869549228 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ec0106c0 con 0x7f83f41070b0 2024-03-04T12:22:40.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.185+0000 7f83fc0d1640 1 -- 172.21.15.60:0/869549228 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f41a1cc0 con 0x7f83f41070b0 2024-03-04T12:22:40.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.186+0000 7f83fc0d1640 1 -- 172.21.15.60:0/869549228 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f40fd330 con 0x7f83f41070b0 2024-03-04T12:22:40.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.187+0000 7f83eaffd640 1 -- 172.21.15.60:0/869549228 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f83ec003970 con 0x7f83f41070b0 2024-03-04T12:22:40.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.187+0000 7f83eaffd640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f83e00a1d80 0x7f83e00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:40.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.187+0000 7f83f9e6d640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f83e00a1d80 0x7f83e00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:40.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.188+0000 7f83eaffd640 1 -- 172.21.15.60:0/869549228 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f83ec0cd580 con 0x7f83f41070b0 2024-03-04T12:22:40.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.188+0000 7f83f9e6d640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f83e00a1d80 0x7f83e00a4240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f83f002c7b0 tx=0x7f83f0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:40.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.192+0000 7f83eaffd640 1 -- 172.21.15.60:0/869549228 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f83ec09a760 con 0x7f83f41070b0 2024-03-04T12:22:40.367 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:22:40 smithi060 podman[73887]: 2024-03-04 12:22:40.130065229 +0000 UTC m=+0.027539145 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:22:40.367 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:22:40 smithi060 podman[73887]: 2024-03-04 12:22:40.242967772 +0000 UTC m=+0.140441648 container create 1364bf2fcc86c1c266fd2cbedd871185afb7dd66e2d7fedef659b017029734af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-04T12:22:40.430 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.425+0000 7f83fc0d1640 1 -- 172.21.15.60:0/869549228 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83f4061f30 con 0x7f83f41070b0 2024-03-04T12:22:40.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.425+0000 7f83eaffd640 1 -- 172.21.15.60:0/869549228 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2113 (secure 0 0 0) 0x7f83ec0b8e20 con 0x7f83f41070b0 2024-03-04T12:22:40.431 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:40.431 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-04T12:22:40.431 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 3 2024-03-04T12:22:40.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.427+0000 7f83fc0d1640 1 -- 172.21.15.60:0/869549228 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f83e00a1d80 msgr2=0x7f83e00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:40.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.427+0000 7f83fc0d1640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f83e00a1d80 0x7f83e00a4240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f83f002c7b0 tx=0x7f83f0005fd0 comp rx=0 tx=0).stop 2024-03-04T12:22:40.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.427+0000 7f83fc0d1640 1 -- 172.21.15.60:0/869549228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83f41070b0 msgr2=0x7f83f4100340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:40.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.427+0000 7f83fc0d1640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83f41070b0 0x7f83f4100340 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f83ec009fd0 tx=0x7f83ec00b260 comp rx=0 tx=0).stop 2024-03-04T12:22:40.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.427+0000 7f83fc0d1640 1 -- 172.21.15.60:0/869549228 shutdown_connections 2024-03-04T12:22:40.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.427+0000 7f83fc0d1640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f83e00a1d80 0x7f83e00a4240 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:40.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.427+0000 7f83fc0d1640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f83f41070b0 0x7f83f4100340 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:40.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.427+0000 7f83fc0d1640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83f4104780 0x7f83f40ffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:40.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.427+0000 7f83fc0d1640 1 --2- 172.21.15.60:0/869549228 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83f4101e50 0x7f83f41017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:40.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.427+0000 7f83fc0d1640 1 -- 172.21.15.60:0/869549228 >> 172.21.15.60:0/869549228 conn(0x7f83f40fb840 msgr2=0x7f83f41026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:40.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.427+0000 7f83fc0d1640 1 -- 172.21.15.60:0/869549228 shutdown_connections 2024-03-04T12:22:40.433 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:40.427+0000 7f83fc0d1640 1 -- 172.21.15.60:0/869549228 wait complete. 2024-03-04T12:22:40.678 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:22:40 smithi060 podman[73887]: 2024-03-04 12:22:40.48780656 +0000 UTC m=+0.385280436 container init 1364bf2fcc86c1c266fd2cbedd871185afb7dd66e2d7fedef659b017029734af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-j, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-04T12:22:40.678 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:22:40 smithi060 podman[73887]: 2024-03-04 12:22:40.490834346 +0000 UTC m=+0.388308233 container start 1364bf2fcc86c1c266fd2cbedd871185afb7dd66e2d7fedef659b017029734af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-j, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:22:40.678 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:22:40 smithi060 ceph-mds[73927]: -- 172.21.15.60:0/939941033 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561a0e6d3a40 con 0x561a0e6a6c00 2024-03-04T12:22:40.678 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:22:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-j[73921]: starting mds.j at 2024-03-04T12:22:40.678 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:22:40 smithi060 bash[73887]: 1364bf2fcc86c1c266fd2cbedd871185afb7dd66e2d7fedef659b017029734af 2024-03-04T12:22:40.678 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:22:40 smithi060 systemd[1]: Started Ceph mds.j for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:22:41.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:40 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/869549228' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:41.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:40 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:41.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:40 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:41.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:40 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:41.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:40 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:41.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:40 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:41.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:40 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:41.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:40 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/869549228' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:41.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:40 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:41.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:40 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:41.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:40 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:41.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:40 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:41.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:40 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:41.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:40 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:41.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:40 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/869549228' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:41.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:40 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:41.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:40 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:41.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:40 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:41.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:40 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:41.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:40 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:41.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:40 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:41.912 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:41 smithi060 ceph-mon[32745]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:41.915 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:41 smithi060 ceph-mon[32745]: Deploying daemon mds.d on smithi060 2024-03-04T12:22:41.915 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:41 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.60:6834/49412012,v1:172.21.15.60:6835/49412012] up:boot 2024-03-04T12:22:41.915 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:41 smithi060 ceph-mon[32745]: fsmap 3 up:standby 2024-03-04T12:22:41.915 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:41 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-04T12:22:42.055 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:41 smithi086 ceph-mon[36851]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:42.056 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:41 smithi086 ceph-mon[36851]: Deploying daemon mds.d on smithi060 2024-03-04T12:22:42.056 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:41 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.60:6834/49412012,v1:172.21.15.60:6835/49412012] up:boot 2024-03-04T12:22:42.056 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:41 smithi086 ceph-mon[36851]: fsmap 3 up:standby 2024-03-04T12:22:42.056 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:41 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-04T12:22:42.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:41 smithi063 ceph-mon[36717]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:42.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:41 smithi063 ceph-mon[36717]: Deploying daemon mds.d on smithi060 2024-03-04T12:22:42.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:41 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.60:6834/49412012,v1:172.21.15.60:6835/49412012] up:boot 2024-03-04T12:22:42.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:41 smithi063 ceph-mon[36717]: fsmap 3 up:standby 2024-03-04T12:22:42.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:41 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-04T12:22:42.444 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:22:42.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.719+0000 7f86b4419640 1 -- 172.21.15.60:0/1972191485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac102f20 msgr2=0x7f86ac1097b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:42.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.719+0000 7f86b4419640 1 --2- 172.21.15.60:0/1972191485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac102f20 0x7f86ac1097b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f86a4009ca0 tx=0x7f86a402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.719+0000 7f86b4419640 1 -- 172.21.15.60:0/1972191485 shutdown_connections 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.719+0000 7f86b4419640 1 --2- 172.21.15.60:0/1972191485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac102f20 0x7f86ac1097b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.719+0000 7f86b4419640 1 --2- 172.21.15.60:0/1972191485 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ac102560 0x7f86ac1029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.719+0000 7f86b4419640 1 --2- 172.21.15.60:0/1972191485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86ac101370 0x7f86ac101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.719+0000 7f86b4419640 1 -- 172.21.15.60:0/1972191485 >> 172.21.15.60:0/1972191485 conn(0x7f86ac0fc8c0 msgr2=0x7f86ac0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.719+0000 7f86b4419640 1 -- 172.21.15.60:0/1972191485 shutdown_connections 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.719+0000 7f86b4419640 1 -- 172.21.15.60:0/1972191485 wait complete. 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.719+0000 7f86b4419640 1 Processor -- start 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b4419640 1 -- start start 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b4419640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ac101370 0x7f86ac197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b4419640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86ac102560 0x7f86ac197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b4419640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac102f20 0x7f86ac19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:42.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b4419640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac10d000 con 0x7f86ac102f20 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b4419640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac10ce80 con 0x7f86ac101370 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b4419640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac10d180 con 0x7f86ac102560 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b29b6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac102f20 0x7f86ac19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b29b6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac102f20 0x7f86ac19eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34776/0 (socket says 172.21.15.60:34776) 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b29b6640 1 -- 172.21.15.60:0/2500720636 learned_addr learned my addr 172.21.15.60:0/2500720636 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b19b4640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86ac102560 0x7f86ac197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b21b5640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ac101370 0x7f86ac197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b29b6640 1 -- 172.21.15.60:0/2500720636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86ac102560 msgr2=0x7f86ac197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b29b6640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86ac102560 0x7f86ac197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b29b6640 1 -- 172.21.15.60:0/2500720636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ac101370 msgr2=0x7f86ac197550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b29b6640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ac101370 0x7f86ac197550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b29b6640 1 -- 172.21.15.60:0/2500720636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac19f170 con 0x7f86ac102f20 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b21b5640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ac101370 0x7f86ac197550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:22:42.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.720+0000 7f86b29b6640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac102f20 0x7f86ac19eb00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f86a402c700 tx=0x7f86a4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:42.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.721+0000 7f86a37fe640 1 -- 172.21.15.60:0/2500720636 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a4039040 con 0x7f86ac102f20 2024-03-04T12:22:42.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.721+0000 7f86a37fe640 1 -- 172.21.15.60:0/2500720636 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86a4044070 con 0x7f86ac102f20 2024-03-04T12:22:42.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.721+0000 7f86b4419640 1 -- 172.21.15.60:0/2500720636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ac19f400 con 0x7f86ac102f20 2024-03-04T12:22:42.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.721+0000 7f86a37fe640 1 -- 172.21.15.60:0/2500720636 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a40042a0 con 0x7f86ac102f20 2024-03-04T12:22:42.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.721+0000 7f86b4419640 1 -- 172.21.15.60:0/2500720636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ac19f8f0 con 0x7f86ac102f20 2024-03-04T12:22:42.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.722+0000 7f86b4419640 1 -- 172.21.15.60:0/2500720636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ac1085c0 con 0x7f86ac102f20 2024-03-04T12:22:42.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.723+0000 7f86a37fe640 1 -- 172.21.15.60:0/2500720636 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f86a4032690 con 0x7f86ac102f20 2024-03-04T12:22:42.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.723+0000 7f86a37fe640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f86980a1d80 0x7f86980a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:42.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.724+0000 7f86a37fe640 1 -- 172.21.15.60:0/2500720636 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f86a40efe60 con 0x7f86ac102f20 2024-03-04T12:22:42.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.724+0000 7f86b21b5640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f86980a1d80 0x7f86980a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:42.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.724+0000 7f86b21b5640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f86980a1d80 0x7f86980a4240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f86a80058c0 tx=0x7f86a8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:42.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.732+0000 7f86a37fe640 1 -- 172.21.15.60:0/2500720636 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f86a40bd720 con 0x7f86ac102f20 2024-03-04T12:22:42.928 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:22:42 smithi060 systemd[1]: Starting Ceph mds.d for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:22:42.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.967+0000 7f86b4419640 1 -- 172.21.15.60:0/2500720636 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86ac061f30 con 0x7f86ac102f20 2024-03-04T12:22:42.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.967+0000 7f86a37fe640 1 -- 172.21.15.60:0/2500720636 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2898 (secure 0 0 0) 0x7f86a40bd020 con 0x7f86ac102f20 2024-03-04T12:22:42.973 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:42.973 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-04T12:22:42.973 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 4 2024-03-04T12:22:42.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.969+0000 7f86b4419640 1 -- 172.21.15.60:0/2500720636 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f86980a1d80 msgr2=0x7f86980a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:42.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.969+0000 7f86b4419640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f86980a1d80 0x7f86980a4240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f86a80058c0 tx=0x7f86a8005850 comp rx=0 tx=0).stop 2024-03-04T12:22:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.969+0000 7f86b4419640 1 -- 172.21.15.60:0/2500720636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac102f20 msgr2=0x7f86ac19eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.969+0000 7f86b4419640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac102f20 0x7f86ac19eb00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f86a402c700 tx=0x7f86a4032f20 comp rx=0 tx=0).stop 2024-03-04T12:22:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.969+0000 7f86b4419640 1 -- 172.21.15.60:0/2500720636 shutdown_connections 2024-03-04T12:22:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.969+0000 7f86b4419640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f86980a1d80 0x7f86980a4240 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.969+0000 7f86b4419640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86ac102f20 0x7f86ac19eb00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.969+0000 7f86b4419640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86ac102560 0x7f86ac197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.969+0000 7f86b4419640 1 --2- 172.21.15.60:0/2500720636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86ac101370 0x7f86ac197550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.969+0000 7f86b4419640 1 -- 172.21.15.60:0/2500720636 >> 172.21.15.60:0/2500720636 conn(0x7f86ac0fc8c0 msgr2=0x7f86ac105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.970+0000 7f86b4419640 1 -- 172.21.15.60:0/2500720636 shutdown_connections 2024-03-04T12:22:42.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:42.970+0000 7f86b4419640 1 -- 172.21.15.60:0/2500720636 wait complete. 2024-03-04T12:22:43.254 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:22:43 smithi060 podman[74265]: 2024-03-04 12:22:42.967099829 +0000 UTC m=+0.019011497 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:22:43.255 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:22:43 smithi060 podman[74265]: 2024-03-04 12:22:43.09141226 +0000 UTC m=+0.143323920 container create 06f28728ec761d925e616cd61bf4e7edb9fc60de13eaf6b753f33df1edc55575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-04T12:22:43.678 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:22:43 smithi060 podman[74265]: 2024-03-04 12:22:43.352449259 +0000 UTC m=+0.404360920 container init 06f28728ec761d925e616cd61bf4e7edb9fc60de13eaf6b753f33df1edc55575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-04T12:22:43.678 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:22:43 smithi060 podman[74265]: 2024-03-04 12:22:43.355052452 +0000 UTC m=+0.406964118 container start 06f28728ec761d925e616cd61bf4e7edb9fc60de13eaf6b753f33df1edc55575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-d, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-04T12:22:43.678 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:22:43 smithi060 ceph-mds[74285]: -- 172.21.15.60:0/963784815 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d4a52c5a40 con 0x55d4a5298800 2024-03-04T12:22:43.678 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:22:43 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-d[74281]: starting mds.d at 2024-03-04T12:22:43.678 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:22:43 smithi060 bash[74265]: 06f28728ec761d925e616cd61bf4e7edb9fc60de13eaf6b753f33df1edc55575 2024-03-04T12:22:43.678 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:22:43 smithi060 systemd[1]: Started Ceph mds.d for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:22:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:43 smithi086 ceph-mon[36851]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:43 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2500720636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:44.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:43 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:43 smithi063 ceph-mon[36717]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:43 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2500720636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:44.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:43 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:43 smithi060 ceph-mon[32745]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:43 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2500720636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:43 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:44.988 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:22:45.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:44 smithi086 ceph-mon[36851]: Deploying daemon mds.e on smithi063 2024-03-04T12:22:45.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:44 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.60:6836/3896783798,v1:172.21.15.60:6837/3896783798] up:boot 2024-03-04T12:22:45.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:44 smithi086 ceph-mon[36851]: fsmap 4 up:standby 2024-03-04T12:22:45.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:44 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-04T12:22:45.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:44 smithi063 ceph-mon[36717]: Deploying daemon mds.e on smithi063 2024-03-04T12:22:45.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:44 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.60:6836/3896783798,v1:172.21.15.60:6837/3896783798] up:boot 2024-03-04T12:22:45.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:44 smithi063 ceph-mon[36717]: fsmap 4 up:standby 2024-03-04T12:22:45.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:44 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-04T12:22:45.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.100+0000 7fda39acd640 1 -- 172.21.15.60:0/1062369763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda34102570 msgr2=0x7fda341029f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:45.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.100+0000 7fda39acd640 1 --2- 172.21.15.60:0/1062369763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda34102570 0x7fda341029f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fda28009900 tx=0x7fda2802bc30 comp rx=0 tx=0).stop 2024-03-04T12:22:45.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.100+0000 7fda39acd640 1 -- 172.21.15.60:0/1062369763 shutdown_connections 2024-03-04T12:22:45.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.100+0000 7fda39acd640 1 --2- 172.21.15.60:0/1062369763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda34102f30 0x7fda341097c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:45.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.100+0000 7fda39acd640 1 --2- 172.21.15.60:0/1062369763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda34102570 0x7fda341029f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:45.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.100+0000 7fda39acd640 1 --2- 172.21.15.60:0/1062369763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda34101380 0x7fda341017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:45.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.100+0000 7fda39acd640 1 -- 172.21.15.60:0/1062369763 >> 172.21.15.60:0/1062369763 conn(0x7fda340fc8f0 msgr2=0x7fda340fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:45.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.100+0000 7fda39acd640 1 -- 172.21.15.60:0/1062369763 shutdown_connections 2024-03-04T12:22:45.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.100+0000 7fda39acd640 1 -- 172.21.15.60:0/1062369763 wait complete. 2024-03-04T12:22:45.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.100+0000 7fda39acd640 1 Processor -- start 2024-03-04T12:22:45.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda39acd640 1 -- start start 2024-03-04T12:22:45.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda39acd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda34101380 0x7fda340750e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:45.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda39acd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda34102570 0x7fda34073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:45.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda39acd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda34102f30 0x7fda34073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:45.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda39acd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3410d060 con 0x7fda34102f30 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda39acd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3410cee0 con 0x7fda34101380 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda39acd640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3410d1e0 con 0x7fda34102570 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda33fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda34102f30 0x7fda34073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda33fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda34102f30 0x7fda34073c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34806/0 (socket says 172.21.15.60:34806) 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda33fff640 1 -- 172.21.15.60:0/612010561 learned_addr learned my addr 172.21.15.60:0/612010561 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda337fe640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda34101380 0x7fda340750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda32ffd640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda34102570 0x7fda34073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda33fff640 1 -- 172.21.15.60:0/612010561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda34102570 msgr2=0x7fda34073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda33fff640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda34102570 0x7fda34073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda33fff640 1 -- 172.21.15.60:0/612010561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda34101380 msgr2=0x7fda340750e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda33fff640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda34101380 0x7fda340750e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda33fff640 1 -- 172.21.15.60:0/612010561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda340744f0 con 0x7fda34102f30 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda337fe640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda34101380 0x7fda340750e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:22:45.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda33fff640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda34102f30 0x7fda34073c60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fda2c00eb10 tx=0x7fda2c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:45.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.101+0000 7fda30ff9640 1 -- 172.21.15.60:0/612010561 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c004b20 con 0x7fda34102f30 2024-03-04T12:22:45.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.102+0000 7fda30ff9640 1 -- 172.21.15.60:0/612010561 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda2c004cc0 con 0x7fda34102f30 2024-03-04T12:22:45.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.102+0000 7fda39acd640 1 -- 172.21.15.60:0/612010561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda341a1720 con 0x7fda34102f30 2024-03-04T12:22:45.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.102+0000 7fda30ff9640 1 -- 172.21.15.60:0/612010561 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c010760 con 0x7fda34102f30 2024-03-04T12:22:45.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.102+0000 7fda39acd640 1 -- 172.21.15.60:0/612010561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda341a1c40 con 0x7fda34102f30 2024-03-04T12:22:45.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.103+0000 7fda39acd640 1 -- 172.21.15.60:0/612010561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda340681b0 con 0x7fda34102f30 2024-03-04T12:22:45.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.104+0000 7fda30ff9640 1 -- 172.21.15.60:0/612010561 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fda2c005020 con 0x7fda34102f30 2024-03-04T12:22:45.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.104+0000 7fda30ff9640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fda1c0a1e00 0x7fda1c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:45.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.105+0000 7fda337fe640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fda1c0a1e00 0x7fda1c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:45.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.105+0000 7fda30ff9640 1 -- 172.21.15.60:0/612010561 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fda2c0cdf30 con 0x7fda34102f30 2024-03-04T12:22:45.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.105+0000 7fda337fe640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fda1c0a1e00 0x7fda1c0a42c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda24005d00 tx=0x7fda24005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:45.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.109+0000 7fda30ff9640 1 -- 172.21.15.60:0/612010561 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fda2c09b800 con 0x7fda34102f30 2024-03-04T12:22:45.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:44 smithi060 ceph-mon[32745]: Deploying daemon mds.e on smithi063 2024-03-04T12:22:45.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:44 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.60:6836/3896783798,v1:172.21.15.60:6837/3896783798] up:boot 2024-03-04T12:22:45.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:44 smithi060 ceph-mon[32745]: fsmap 4 up:standby 2024-03-04T12:22:45.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:44 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-04T12:22:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.328+0000 7fda39acd640 1 -- 172.21.15.60:0/612010561 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda3402ceb0 con 0x7fda34102f30 2024-03-04T12:22:45.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.329+0000 7fda30ff9640 1 -- 172.21.15.60:0/612010561 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3689 (secure 0 0 0) 0x7fda2c09b100 con 0x7fda34102f30 2024-03-04T12:22:45.335 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:45.335 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-04T12:22:45.335 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 5 2024-03-04T12:22:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.331+0000 7fda39acd640 1 -- 172.21.15.60:0/612010561 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fda1c0a1e00 msgr2=0x7fda1c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.331+0000 7fda39acd640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fda1c0a1e00 0x7fda1c0a42c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda24005d00 tx=0x7fda24005c90 comp rx=0 tx=0).stop 2024-03-04T12:22:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.331+0000 7fda39acd640 1 -- 172.21.15.60:0/612010561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda34102f30 msgr2=0x7fda34073c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.331+0000 7fda39acd640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda34102f30 0x7fda34073c60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fda2c00eb10 tx=0x7fda2c00c2d0 comp rx=0 tx=0).stop 2024-03-04T12:22:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.331+0000 7fda39acd640 1 -- 172.21.15.60:0/612010561 shutdown_connections 2024-03-04T12:22:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.331+0000 7fda39acd640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fda1c0a1e00 0x7fda1c0a42c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.331+0000 7fda39acd640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fda34102f30 0x7fda34073c60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.331+0000 7fda39acd640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda34102570 0x7fda34073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:45.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.331+0000 7fda39acd640 1 --2- 172.21.15.60:0/612010561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda34101380 0x7fda340750e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.331+0000 7fda39acd640 1 -- 172.21.15.60:0/612010561 >> 172.21.15.60:0/612010561 conn(0x7fda340fc8f0 msgr2=0x7fda34105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.331+0000 7fda39acd640 1 -- 172.21.15.60:0/612010561 shutdown_connections 2024-03-04T12:22:45.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:45.331+0000 7fda39acd640 1 -- 172.21.15.60:0/612010561 wait complete. 2024-03-04T12:22:46.066 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:45 smithi063 ceph-mon[36717]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:46.067 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:45 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/612010561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:46.067 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:22:45 smithi063 systemd[1]: Starting Ceph mds.e for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:22:46.097 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:45 smithi086 ceph-mon[36851]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:46.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:45 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/612010561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:46.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:45 smithi060 ceph-mon[32745]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:46.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:45 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/612010561' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:46.334 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:22:46 smithi063 podman[66777]: 2024-03-04 12:22:45.964243357 +0000 UTC m=+0.018507423 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:22:46.334 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:22:46 smithi063 podman[66777]: 2024-03-04 12:22:46.076267063 +0000 UTC m=+0.130531113 container create 9856d0e13ea092a944c00453def0834b7b998bd25d7afce79118de00be69163c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-04T12:22:46.334 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:22:46 smithi063 podman[66777]: 2024-03-04 12:22:46.328908285 +0000 UTC m=+0.383172336 container init 9856d0e13ea092a944c00453def0834b7b998bd25d7afce79118de00be69163c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:22:46.601 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:22:46 smithi063 podman[66777]: 2024-03-04 12:22:46.331801695 +0000 UTC m=+0.386065746 container start 9856d0e13ea092a944c00453def0834b7b998bd25d7afce79118de00be69163c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-04T12:22:46.601 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:22:46 smithi063 ceph-mds[66796]: -- 172.21.15.63:0/467490518 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558fe978ba40 con 0x558fe975ec00 2024-03-04T12:22:46.601 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:22:46 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e[66792]: starting mds.e at 2024-03-04T12:22:46.601 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:22:46 smithi063 bash[66777]: 9856d0e13ea092a944c00453def0834b7b998bd25d7afce79118de00be69163c 2024-03-04T12:22:46.601 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:22:46 smithi063 systemd[1]: Started Ceph mds.e for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:22:47.348 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:22:47.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.459+0000 7f2be2e44640 1 -- 172.21.15.60:0/1183284284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bdc101450 msgr2=0x7f2bdc1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.459+0000 7f2be2e44640 1 --2- 172.21.15.60:0/1183284284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bdc101450 0x7f2bdc1018d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009900 tx=0x7f2bd002bc30 comp rx=0 tx=0).stop 2024-03-04T12:22:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.459+0000 7f2be2e44640 1 -- 172.21.15.60:0/1183284284 shutdown_connections 2024-03-04T12:22:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.459+0000 7f2be2e44640 1 --2- 172.21.15.60:0/1183284284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bdc101e10 0x7f2bdc1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.459+0000 7f2be2e44640 1 --2- 172.21.15.60:0/1183284284 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bdc101450 0x7f2bdc1018d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.459+0000 7f2be2e44640 1 --2- 172.21.15.60:0/1183284284 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc100260 0x7f2bdc100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.459+0000 7f2be2e44640 1 -- 172.21.15.60:0/1183284284 >> 172.21.15.60:0/1183284284 conn(0x7f2bdc0fb7f0 msgr2=0x7f2bdc0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.459+0000 7f2be2e44640 1 -- 172.21.15.60:0/1183284284 shutdown_connections 2024-03-04T12:22:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.459+0000 7f2be2e44640 1 -- 172.21.15.60:0/1183284284 wait complete. 2024-03-04T12:22:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.459+0000 7f2be2e44640 1 Processor -- start 2024-03-04T12:22:47.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.459+0000 7f2be2e44640 1 -- start start 2024-03-04T12:22:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2be2e44640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bdc100260 0x7f2bdc197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2be2e44640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bdc101450 0x7f2bdc197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2be2e44640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc101e10 0x7f2bdc19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2be2e44640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc10bef0 con 0x7f2bdc101450 2024-03-04T12:22:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2be2e44640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc10bd70 con 0x7f2bdc101e10 2024-03-04T12:22:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2be2e44640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc10c070 con 0x7f2bdc100260 2024-03-04T12:22:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2bdbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bdc101450 0x7f2bdc197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2bdbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bdc101450 0x7f2bdc197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54550/0 (socket says 172.21.15.60:54550) 2024-03-04T12:22:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2bdbfff640 1 -- 172.21.15.60:0/2857325798 learned_addr learned my addr 172.21.15.60:0/2857325798 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2be0be0640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bdc100260 0x7f2bdc197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:47.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2be13e1640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc101e10 0x7f2bdc19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2bdbfff640 1 -- 172.21.15.60:0/2857325798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bdc100260 msgr2=0x7f2bdc197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2bdbfff640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bdc100260 0x7f2bdc197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2bdbfff640 1 -- 172.21.15.60:0/2857325798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc101e10 msgr2=0x7f2bdc19ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2bdbfff640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc101e10 0x7f2bdc19ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2bdbfff640 1 -- 172.21.15.60:0/2857325798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc19f260 con 0x7f2bdc101450 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2be0be0640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bdc100260 0x7f2bdc197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2be13e1640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc101e10 0x7f2bdc19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2bdbfff640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bdc101450 0x7f2bdc197b80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009230 tx=0x7f2bd0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2bd9ffb640 1 -- 172.21.15.60:0/2857325798 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0041070 con 0x7f2bdc101450 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2bd9ffb640 1 -- 172.21.15.60:0/2857325798 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2bd00044d0 con 0x7f2bdc101450 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2be2e44640 1 -- 172.21.15.60:0/2857325798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bdc19f4f0 con 0x7f2bdc101450 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.460+0000 7f2bd9ffb640 1 -- 172.21.15.60:0/2857325798 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0034440 con 0x7f2bdc101450 2024-03-04T12:22:47.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.461+0000 7f2be2e44640 1 -- 172.21.15.60:0/2857325798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bdc19fa30 con 0x7f2bdc101450 2024-03-04T12:22:47.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.462+0000 7f2be2e44640 1 -- 172.21.15.60:0/2857325798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bdc19fc90 con 0x7f2bdc101450 2024-03-04T12:22:47.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.463+0000 7f2bd9ffb640 1 -- 172.21.15.60:0/2857325798 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f2bd0044070 con 0x7f2bdc101450 2024-03-04T12:22:47.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.463+0000 7f2bd9ffb640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2bc40a1e00 0x7f2bc40a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:47.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.463+0000 7f2be0be0640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2bc40a1e00 0x7f2bc40a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:47.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.463+0000 7f2bd9ffb640 1 -- 172.21.15.60:0/2857325798 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2bd00ef9c0 con 0x7f2bdc101450 2024-03-04T12:22:47.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.464+0000 7f2be0be0640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2bc40a1e00 0x7f2bc40a42c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc1012b0 tx=0x7f2bd4009540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:47.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.467+0000 7f2bd9ffb640 1 -- 172.21.15.60:0/2857325798 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2bd00bd1f0 con 0x7f2bdc101450 2024-03-04T12:22:47.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:47 smithi060 ceph-mon[32745]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:47.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:47 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:47.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:47 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:47.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:47 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:47.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:47 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:47.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:47 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:47.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:47 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:47.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:47 smithi060 ceph-mon[32745]: Deploying daemon mds.f on smithi086 2024-03-04T12:22:47.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:47 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.63:6834/2453356505,v1:172.21.15.63:6835/2453356505] up:boot 2024-03-04T12:22:47.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:47 smithi060 ceph-mon[32745]: fsmap 5 up:standby 2024-03-04T12:22:47.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:47 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T12:22:47.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.688+0000 7f2be2e44640 1 -- 172.21.15.60:0/2857325798 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bdc19fe60 con 0x7f2bdc101450 2024-03-04T12:22:47.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.688+0000 7f2bd9ffb640 1 -- 172.21.15.60:0/2857325798 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4480 (secure 0 0 0) 0x7f2bd00bcaf0 con 0x7f2bdc101450 2024-03-04T12:22:47.694 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:47.694 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-04T12:22:47.695 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 6 2024-03-04T12:22:47.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.691+0000 7f2be2e44640 1 -- 172.21.15.60:0/2857325798 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2bc40a1e00 msgr2=0x7f2bc40a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.691+0000 7f2be2e44640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2bc40a1e00 0x7f2bc40a42c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc1012b0 tx=0x7f2bd4009540 comp rx=0 tx=0).stop 2024-03-04T12:22:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.691+0000 7f2be2e44640 1 -- 172.21.15.60:0/2857325798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bdc101450 msgr2=0x7f2bdc197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.691+0000 7f2be2e44640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bdc101450 0x7f2bdc197b80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009230 tx=0x7f2bd0005d50 comp rx=0 tx=0).stop 2024-03-04T12:22:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.691+0000 7f2be2e44640 1 -- 172.21.15.60:0/2857325798 shutdown_connections 2024-03-04T12:22:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.691+0000 7f2be2e44640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2bc40a1e00 0x7f2bc40a42c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.691+0000 7f2be2e44640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc101e10 0x7f2bdc19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.691+0000 7f2be2e44640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2bdc101450 0x7f2bdc197b80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.691+0000 7f2be2e44640 1 --2- 172.21.15.60:0/2857325798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bdc100260 0x7f2bdc197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.691+0000 7f2be2e44640 1 -- 172.21.15.60:0/2857325798 >> 172.21.15.60:0/2857325798 conn(0x7f2bdc0fb7f0 msgr2=0x7f2bdc069a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.691+0000 7f2be2e44640 1 -- 172.21.15.60:0/2857325798 shutdown_connections 2024-03-04T12:22:47.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:47.691+0000 7f2be2e44640 1 -- 172.21.15.60:0/2857325798 wait complete. 2024-03-04T12:22:47.837 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:47 smithi086 ceph-mon[36851]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:47.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:47 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:47.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:47 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:47.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:47 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:47.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:47 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:47.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:47 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:47.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:47 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:47.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:47 smithi086 ceph-mon[36851]: Deploying daemon mds.f on smithi086 2024-03-04T12:22:47.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:47 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.63:6834/2453356505,v1:172.21.15.63:6835/2453356505] up:boot 2024-03-04T12:22:47.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:47 smithi086 ceph-mon[36851]: fsmap 5 up:standby 2024-03-04T12:22:47.838 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:47 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T12:22:47.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:47 smithi063 ceph-mon[36717]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:47.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:47 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:47.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:47 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:47.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:47 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:47.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:47 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:47.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:47 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:47.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:47 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:47.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:47 smithi063 ceph-mon[36717]: Deploying daemon mds.f on smithi086 2024-03-04T12:22:47.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:47 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.63:6834/2453356505,v1:172.21.15.63:6835/2453356505] up:boot 2024-03-04T12:22:47.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:47 smithi063 ceph-mon[36717]: fsmap 5 up:standby 2024-03-04T12:22:47.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:47 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T12:22:48.809 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:48 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2857325798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:48.809 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:22:48 smithi086 systemd[1]: Starting Ceph mds.f for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:22:48.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:48 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2857325798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:48.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:48 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2857325798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:49.085 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:22:48 smithi086 podman[67524]: 2024-03-04 12:22:48.706273101 +0000 UTC m=+0.017849662 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:22:49.085 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:22:48 smithi086 podman[67524]: 2024-03-04 12:22:48.844114518 +0000 UTC m=+0.155691059 container create 089cf647a6e0676c0c25bc7ccc1f9c6ff059ab88c4ec204ea4829ef42ffca9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-f, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:22:49.348 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:22:49 smithi086 podman[67524]: 2024-03-04 12:22:49.080227034 +0000 UTC m=+0.391803606 container init 089cf647a6e0676c0c25bc7ccc1f9c6ff059ab88c4ec204ea4829ef42ffca9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-f, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-04T12:22:49.348 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:22:49 smithi086 podman[67524]: 2024-03-04 12:22:49.083295792 +0000 UTC m=+0.394872351 container start 089cf647a6e0676c0c25bc7ccc1f9c6ff059ab88c4ec204ea4829ef42ffca9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-f, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-04T12:22:49.348 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:22:49 smithi086 ceph-mds[67544]: -- 172.21.15.86:0/158682166 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5560233fda40 con 0x5560233d0800 2024-03-04T12:22:49.348 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:22:49 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-f[67540]: starting mds.f at 2024-03-04T12:22:49.348 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:22:49 smithi086 bash[67524]: 089cf647a6e0676c0c25bc7ccc1f9c6ff059ab88c4ec204ea4829ef42ffca9ef 2024-03-04T12:22:49.349 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:22:49 smithi086 systemd[1]: Started Ceph mds.f for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:22:49.708 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:22:49.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.818+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2679395716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cdc069630 msgr2=0x7f8cdc0fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.818+0000 7f8ce37e6640 1 --2- 172.21.15.60:0/2679395716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cdc069630 0x7f8cdc0fff50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009900 tx=0x7f8ccc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:22:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2679395716 shutdown_connections 2024-03-04T12:22:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 --2- 172.21.15.60:0/2679395716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cdc100be0 0x7f8cdc101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 --2- 172.21.15.60:0/2679395716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cdc069630 0x7f8cdc0fff50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 --2- 172.21.15.60:0/2679395716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cdc069180 0x7f8cdc0ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2679395716 >> 172.21.15.60:0/2679395716 conn(0x7f8cdc0faf60 msgr2=0x7f8cdc0fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2679395716 shutdown_connections 2024-03-04T12:22:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2679395716 wait complete. 2024-03-04T12:22:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 Processor -- start 2024-03-04T12:22:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 -- start start 2024-03-04T12:22:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cdc069180 0x7f8cdc102990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cdc069630 0x7f8cdc19c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cdc100be0 0x7f8cdc19e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc10be80 con 0x7f8cdc100be0 2024-03-04T12:22:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc10bd00 con 0x7f8cdc069630 2024-03-04T12:22:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.819+0000 7f8ce37e6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc10c000 con 0x7f8cdc069180 2024-03-04T12:22:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce1d83640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cdc100be0 0x7f8cdc19e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce1d83640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cdc100be0 0x7f8cdc19e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54570/0 (socket says 172.21.15.60:54570) 2024-03-04T12:22:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce1d83640 1 -- 172.21.15.60:0/2167575044 learned_addr learned my addr 172.21.15.60:0/2167575044 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce1582640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cdc069180 0x7f8cdc102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce0d81640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cdc069630 0x7f8cdc19c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce1d83640 1 -- 172.21.15.60:0/2167575044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cdc069180 msgr2=0x7f8cdc102990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce1d83640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cdc069180 0x7f8cdc102990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce1d83640 1 -- 172.21.15.60:0/2167575044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cdc069630 msgr2=0x7f8cdc19c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce1d83640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cdc069630 0x7f8cdc19c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce1d83640 1 -- 172.21.15.60:0/2167575044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cdc19f280 con 0x7f8cdc100be0 2024-03-04T12:22:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce1d83640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cdc100be0 0x7f8cdc19e9c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400eaa0 tx=0x7f8cd400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8cd27fc640 1 -- 172.21.15.60:0/2167575044 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd400cdb0 con 0x7f8cdc100be0 2024-03-04T12:22:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8cd27fc640 1 -- 172.21.15.60:0/2167575044 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8cd4005020 con 0x7f8cdc100be0 2024-03-04T12:22:49.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2167575044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cdc19f570 con 0x7f8cdc100be0 2024-03-04T12:22:49.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8cd27fc640 1 -- 172.21.15.60:0/2167575044 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd40116b0 con 0x7f8cdc100be0 2024-03-04T12:22:49.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.820+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2167575044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cdc19fa40 con 0x7f8cdc100be0 2024-03-04T12:22:49.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.821+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2167575044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cdc107830 con 0x7f8cdc100be0 2024-03-04T12:22:49.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.823+0000 7f8cd27fc640 1 -- 172.21.15.60:0/2167575044 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f8cd4005560 con 0x7f8cdc100be0 2024-03-04T12:22:49.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.823+0000 7f8cd27fc640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8cc80a1d80 0x7f8cc80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:49.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.823+0000 7f8ce1582640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8cc80a1d80 0x7f8cc80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:49.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.823+0000 7f8cd27fc640 1 -- 172.21.15.60:0/2167575044 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8cd40cdf20 con 0x7f8cdc100be0 2024-03-04T12:22:49.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.824+0000 7f8ce1582640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8cc80a1d80 0x7f8cc80a4240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8005fd0 tx=0x7f8cd800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:49.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:49.827+0000 7f8cd27fc640 1 -- 172.21.15.60:0/2167575044 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8cd4016030 con 0x7f8cdc100be0 2024-03-04T12:22:49.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:49 smithi086 ceph-mon[36851]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:49.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:49.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:49.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:49.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:49.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:49.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:49 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:49.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:49 smithi063 ceph-mon[36717]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:49.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:49.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:49.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:49.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:49.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:49.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:49 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:49.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:49 smithi060 ceph-mon[32745]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:49.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:49.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:49.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:49.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:49.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:49.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:49 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:50.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.046+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2167575044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cdc02ceb0 con 0x7f8cdc100be0 2024-03-04T12:22:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.047+0000 7f8cd27fc640 1 -- 172.21.15.60:0/2167575044 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5271 (secure 0 0 0) 0x7f8cd409b7f0 con 0x7f8cdc100be0 2024-03-04T12:22:50.052 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:50.052 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-04T12:22:50.052 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 7 2024-03-04T12:22:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.049+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2167575044 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8cc80a1d80 msgr2=0x7f8cc80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:50.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.049+0000 7f8ce37e6640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8cc80a1d80 0x7f8cc80a4240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8005fd0 tx=0x7f8cd800b410 comp rx=0 tx=0).stop 2024-03-04T12:22:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.049+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2167575044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cdc100be0 msgr2=0x7f8cdc19e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.049+0000 7f8ce37e6640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cdc100be0 0x7f8cdc19e9c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8cd400eaa0 tx=0x7f8cd400ef60 comp rx=0 tx=0).stop 2024-03-04T12:22:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.049+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2167575044 shutdown_connections 2024-03-04T12:22:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.049+0000 7f8ce37e6640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8cc80a1d80 0x7f8cc80a4240 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.049+0000 7f8ce37e6640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8cdc100be0 0x7f8cdc19e9c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.049+0000 7f8ce37e6640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8cdc069630 0x7f8cdc19c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.049+0000 7f8ce37e6640 1 --2- 172.21.15.60:0/2167575044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cdc069180 0x7f8cdc102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.049+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2167575044 >> 172.21.15.60:0/2167575044 conn(0x7f8cdc0faf60 msgr2=0x7f8cdc0fc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.049+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2167575044 shutdown_connections 2024-03-04T12:22:50.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:50.049+0000 7f8ce37e6640 1 -- 172.21.15.60:0/2167575044 wait complete. 2024-03-04T12:22:50.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:50 smithi086 ceph-mon[36851]: Deploying daemon mds.c on smithi086 2024-03-04T12:22:50.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:50 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.86:6834/2934725061,v1:172.21.15.86:6835/2934725061] up:boot 2024-03-04T12:22:50.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:50 smithi086 ceph-mon[36851]: fsmap 6 up:standby 2024-03-04T12:22:50.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:50 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-04T12:22:50.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:50 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2167575044' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:50.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:50 smithi063 ceph-mon[36717]: Deploying daemon mds.c on smithi086 2024-03-04T12:22:50.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:50 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.86:6834/2934725061,v1:172.21.15.86:6835/2934725061] up:boot 2024-03-04T12:22:50.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:50 smithi063 ceph-mon[36717]: fsmap 6 up:standby 2024-03-04T12:22:50.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:50 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-04T12:22:50.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:50 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2167575044' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:50.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:50 smithi060 ceph-mon[32745]: Deploying daemon mds.c on smithi086 2024-03-04T12:22:50.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:50 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.86:6834/2934725061,v1:172.21.15.86:6835/2934725061] up:boot 2024-03-04T12:22:50.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:50 smithi060 ceph-mon[32745]: fsmap 6 up:standby 2024-03-04T12:22:50.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:50 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-04T12:22:50.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:50 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2167575044' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:51.508 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:22:51 smithi086 systemd[1]: Starting Ceph mds.c for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:22:51.847 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:51 smithi086 ceph-mon[36851]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:51.848 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:22:51 smithi086 podman[67826]: 2024-03-04 12:22:51.48630745 +0000 UTC m=+0.017599707 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:22:51.848 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:22:51 smithi086 podman[67826]: 2024-03-04 12:22:51.590799941 +0000 UTC m=+0.122092179 container create b34586918c54540155abd2ad19e3955c071007f655113f2e1e8841a7108b8b04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-04T12:22:51.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:51 smithi063 ceph-mon[36717]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:51.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:51 smithi060 ceph-mon[32745]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:52.066 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:22:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.176+0000 7f6abec4d640 1 -- 172.21.15.60:0/2199142091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab81025b0 msgr2=0x7f6ab8102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.176+0000 7f6abec4d640 1 --2- 172.21.15.60:0/2199142091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab81025b0 0x7f6ab8102a30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4009900 tx=0x7f6aa402bc30 comp rx=0 tx=0).stop 2024-03-04T12:22:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.176+0000 7f6abec4d640 1 -- 172.21.15.60:0/2199142091 shutdown_connections 2024-03-04T12:22:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.176+0000 7f6abec4d640 1 --2- 172.21.15.60:0/2199142091 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab8102f70 0x7f6ab8109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.176+0000 7f6abec4d640 1 --2- 172.21.15.60:0/2199142091 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab81025b0 0x7f6ab8102a30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.176+0000 7f6abec4d640 1 --2- 172.21.15.60:0/2199142091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab81013c0 0x7f6ab81017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.176+0000 7f6abec4d640 1 -- 172.21.15.60:0/2199142091 >> 172.21.15.60:0/2199142091 conn(0x7f6ab80fc950 msgr2=0x7f6ab80fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.176+0000 7f6abec4d640 1 -- 172.21.15.60:0/2199142091 shutdown_connections 2024-03-04T12:22:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.176+0000 7f6abec4d640 1 -- 172.21.15.60:0/2199142091 wait complete. 2024-03-04T12:22:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abec4d640 1 Processor -- start 2024-03-04T12:22:52.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abec4d640 1 -- start start 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abec4d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab81013c0 0x7f6ab8197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abec4d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab81025b0 0x7f6ab8197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abec4d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab8102f70 0x7f6ab819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abec4d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab810cfe0 con 0x7f6ab8102f70 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abec4d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab810ce60 con 0x7f6ab81025b0 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abec4d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab810d160 con 0x7f6ab81013c0 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abd1ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab8102f70 0x7f6ab819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abd1ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab8102f70 0x7f6ab819ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54576/0 (socket says 172.21.15.60:54576) 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abd1ea640 1 -- 172.21.15.60:0/3675425645 learned_addr learned my addr 172.21.15.60:0/3675425645 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6ab7fff640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab81025b0 0x7f6ab8197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abc9e9640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab81013c0 0x7f6ab8197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abd1ea640 1 -- 172.21.15.60:0/3675425645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab81013c0 msgr2=0x7f6ab8197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abd1ea640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab81013c0 0x7f6ab8197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abd1ea640 1 -- 172.21.15.60:0/3675425645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab81025b0 msgr2=0x7f6ab8197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:52.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abd1ea640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab81025b0 0x7f6ab8197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abd1ea640 1 -- 172.21.15.60:0/3675425645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab819f290 con 0x7f6ab8102f70 2024-03-04T12:22:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6abc9e9640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab81013c0 0x7f6ab8197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:22:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.177+0000 7f6ab7fff640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab81025b0 0x7f6ab8197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:22:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.178+0000 7f6abd1ea640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab8102f70 0x7f6ab819ec20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000eb10 tx=0x7f6ab000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.178+0000 7f6ab5ffb640 1 -- 172.21.15.60:0/3675425645 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0004b20 con 0x7f6ab8102f70 2024-03-04T12:22:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.178+0000 7f6abec4d640 1 -- 172.21.15.60:0/3675425645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab819f580 con 0x7f6ab8102f70 2024-03-04T12:22:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.178+0000 7f6ab5ffb640 1 -- 172.21.15.60:0/3675425645 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ab0004cc0 con 0x7f6ab8102f70 2024-03-04T12:22:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.178+0000 7f6ab5ffb640 1 -- 172.21.15.60:0/3675425645 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0010760 con 0x7f6ab8102f70 2024-03-04T12:22:52.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.178+0000 7f6abec4d640 1 -- 172.21.15.60:0/3675425645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab819fa50 con 0x7f6ab8102f70 2024-03-04T12:22:52.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.179+0000 7f6abec4d640 1 -- 172.21.15.60:0/3675425645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab8108680 con 0x7f6ab8102f70 2024-03-04T12:22:52.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.180+0000 7f6ab5ffb640 1 -- 172.21.15.60:0/3675425645 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f6ab0005000 con 0x7f6ab8102f70 2024-03-04T12:22:52.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.181+0000 7f6ab5ffb640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6aa80a1d80 0x7f6aa80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:52.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.181+0000 7f6ab5ffb640 1 -- 172.21.15.60:0/3675425645 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6ab00ce080 con 0x7f6ab8102f70 2024-03-04T12:22:52.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.181+0000 7f6abc9e9640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6aa80a1d80 0x7f6aa80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:52.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.181+0000 7f6abc9e9640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6aa80a1d80 0x7f6aa80a4240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8102410 tx=0x7f6aac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:52.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.185+0000 7f6ab5ffb640 1 -- 172.21.15.60:0/3675425645 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6ab0052d20 con 0x7f6ab8102f70 2024-03-04T12:22:52.348 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:22:51 smithi086 podman[67826]: 2024-03-04 12:22:51.851709009 +0000 UTC m=+0.383001274 container init b34586918c54540155abd2ad19e3955c071007f655113f2e1e8841a7108b8b04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:22:52.348 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:22:51 smithi086 podman[67826]: 2024-03-04 12:22:51.854290055 +0000 UTC m=+0.385582299 container start b34586918c54540155abd2ad19e3955c071007f655113f2e1e8841a7108b8b04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-04T12:22:52.348 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:22:51 smithi086 ceph-mds[67845]: -- 172.21.15.86:0/3803155425 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ab958a1a40 con 0x55ab95874800 2024-03-04T12:22:52.348 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:22:51 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: starting mds.c at 2024-03-04T12:22:52.348 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:22:51 smithi086 bash[67826]: b34586918c54540155abd2ad19e3955c071007f655113f2e1e8841a7108b8b04 2024-03-04T12:22:52.348 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:22:51 smithi086 systemd[1]: Started Ceph mds.c for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:22:52.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.404+0000 7f6abec4d640 1 -- 172.21.15.60:0/3675425645 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ab802ceb0 con 0x7f6ab8102f70 2024-03-04T12:22:52.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.405+0000 7f6ab5ffb640 1 -- 172.21.15.60:0/3675425645 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5271 (secure 0 0 0) 0x7f6ab009b950 con 0x7f6ab8102f70 2024-03-04T12:22:52.410 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:52.410 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-04T12:22:52.411 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 7 2024-03-04T12:22:52.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.406+0000 7f6abec4d640 1 -- 172.21.15.60:0/3675425645 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6aa80a1d80 msgr2=0x7f6aa80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.406+0000 7f6abec4d640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6aa80a1d80 0x7f6aa80a4240 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8102410 tx=0x7f6aac005f90 comp rx=0 tx=0).stop 2024-03-04T12:22:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.407+0000 7f6abec4d640 1 -- 172.21.15.60:0/3675425645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab8102f70 msgr2=0x7f6ab819ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.407+0000 7f6abec4d640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab8102f70 0x7f6ab819ec20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000eb10 tx=0x7f6ab000c2d0 comp rx=0 tx=0).stop 2024-03-04T12:22:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.407+0000 7f6abec4d640 1 -- 172.21.15.60:0/3675425645 shutdown_connections 2024-03-04T12:22:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.407+0000 7f6abec4d640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6aa80a1d80 0x7f6aa80a4240 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.407+0000 7f6abec4d640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ab8102f70 0x7f6ab819ec20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.407+0000 7f6abec4d640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ab81025b0 0x7f6ab8197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.407+0000 7f6abec4d640 1 --2- 172.21.15.60:0/3675425645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab81013c0 0x7f6ab8197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.407+0000 7f6abec4d640 1 -- 172.21.15.60:0/3675425645 >> 172.21.15.60:0/3675425645 conn(0x7f6ab80fc950 msgr2=0x7f6ab8105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.407+0000 7f6abec4d640 1 -- 172.21.15.60:0/3675425645 shutdown_connections 2024-03-04T12:22:52.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:52.407+0000 7f6abec4d640 1 -- 172.21.15.60:0/3675425645 wait complete. 2024-03-04T12:22:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:53 smithi060 ceph-mon[32745]: Deploying daemon mds.h on smithi063 2024-03-04T12:22:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:53 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3675425645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:53.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:53 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.86:6836/472598778,v1:172.21.15.86:6837/472598778] up:boot 2024-03-04T12:22:53.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:53 smithi060 ceph-mon[32745]: fsmap 7 up:standby 2024-03-04T12:22:53.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T12:22:53.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:53.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:53.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:53.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:53.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:53.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:53.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:53 smithi086 ceph-mon[36851]: Deploying daemon mds.h on smithi063 2024-03-04T12:22:53.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:53 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3675425645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:53.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:53 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.86:6836/472598778,v1:172.21.15.86:6837/472598778] up:boot 2024-03-04T12:22:53.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:53 smithi086 ceph-mon[36851]: fsmap 7 up:standby 2024-03-04T12:22:53.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T12:22:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:53 smithi063 ceph-mon[36717]: Deploying daemon mds.h on smithi063 2024-03-04T12:22:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:53 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3675425645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:53 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.86:6836/472598778,v1:172.21.15.86:6837/472598778] up:boot 2024-03-04T12:22:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:53 smithi063 ceph-mon[36717]: fsmap 7 up:standby 2024-03-04T12:22:53.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T12:22:54.341 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:54 smithi063 ceph-mon[36717]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:54.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:54 smithi086 ceph-mon[36851]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:54.424 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:22:54.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:54 smithi060 ceph-mon[32745]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:54.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.533+0000 7f904900b640 1 -- 172.21.15.60:0/3541305266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90440ffc30 msgr2=0x7f9044100050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:54.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.533+0000 7f904900b640 1 --2- 172.21.15.60:0/3541305266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90440ffc30 0x7f9044100050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9034009ca0 tx=0x7f903402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:22:54.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.533+0000 7f904900b640 1 -- 172.21.15.60:0/3541305266 shutdown_connections 2024-03-04T12:22:54.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.533+0000 7f904900b640 1 --2- 172.21.15.60:0/3541305266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9044100f30 0x7f90441013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:54.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.533+0000 7f904900b640 1 --2- 172.21.15.60:0/3541305266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9044100590 0x7f90441009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:54.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.533+0000 7f904900b640 1 --2- 172.21.15.60:0/3541305266 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f90440ffc30 0x7f9044100050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:54.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.533+0000 7f904900b640 1 -- 172.21.15.60:0/3541305266 >> 172.21.15.60:0/3541305266 conn(0x7f90440fb7d0 msgr2=0x7f90440fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:54.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.534+0000 7f904900b640 1 -- 172.21.15.60:0/3541305266 shutdown_connections 2024-03-04T12:22:54.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.534+0000 7f904900b640 1 -- 172.21.15.60:0/3541305266 wait complete. 2024-03-04T12:22:54.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.534+0000 7f904900b640 1 Processor -- start 2024-03-04T12:22:54.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.534+0000 7f904900b640 1 -- start start 2024-03-04T12:22:54.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904900b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90440ffc30 0x7f9044199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:54.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904900b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9044100590 0x7f9044199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:54.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904900b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9044100f30 0x7f90441a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904900b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904410f220 con 0x7f9044100f30 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904900b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904410f0a0 con 0x7f9044100590 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904900b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904410f3a0 con 0x7f90440ffc30 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9044100f30 0x7f90441a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9044100590 0x7f9044199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904359e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9044100f30 0x7f90441a0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54592/0 (socket says 172.21.15.60:54592) 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904359e640 1 -- 172.21.15.60:0/515571917 learned_addr learned my addr 172.21.15.60:0/515571917 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f9042d9d640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90440ffc30 0x7f9044199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904359e640 1 -- 172.21.15.60:0/515571917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90440ffc30 msgr2=0x7f9044199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904359e640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90440ffc30 0x7f9044199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904359e640 1 -- 172.21.15.60:0/515571917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9044100590 msgr2=0x7f9044199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904359e640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9044100590 0x7f9044199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904359e640 1 -- 172.21.15.60:0/515571917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90441a1480 con 0x7f9044100f30 2024-03-04T12:22:54.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904259c640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9044100590 0x7f9044199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:22:54.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904359e640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9044100f30 0x7f90441a0e10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f903c009fd0 tx=0x7f903c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:54.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f902bfff640 1 -- 172.21.15.60:0/515571917 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c011830 con 0x7f9044100f30 2024-03-04T12:22:54.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f904900b640 1 -- 172.21.15.60:0/515571917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90441a1770 con 0x7f9044100f30 2024-03-04T12:22:54.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.535+0000 7f902bfff640 1 -- 172.21.15.60:0/515571917 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f903c016070 con 0x7f9044100f30 2024-03-04T12:22:54.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.536+0000 7f902bfff640 1 -- 172.21.15.60:0/515571917 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903c00f2e0 con 0x7f9044100f30 2024-03-04T12:22:54.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.536+0000 7f904900b640 1 -- 172.21.15.60:0/515571917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90441a1c70 con 0x7f9044100f30 2024-03-04T12:22:54.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.536+0000 7f904900b640 1 -- 172.21.15.60:0/515571917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904410ab10 con 0x7f9044100f30 2024-03-04T12:22:54.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.539+0000 7f902bfff640 1 -- 172.21.15.60:0/515571917 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f903c0119d0 con 0x7f9044100f30 2024-03-04T12:22:54.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.539+0000 7f902bfff640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f902c0a1d80 0x7f902c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:54.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.540+0000 7f9042d9d640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f902c0a1d80 0x7f902c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:54.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.540+0000 7f902bfff640 1 -- 172.21.15.60:0/515571917 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f903c0cd7a0 con 0x7f9044100f30 2024-03-04T12:22:54.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.540+0000 7f9042d9d640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f902c0a1d80 0x7f902c0a4240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9034006010 tx=0x7f9034032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:54.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.543+0000 7f902bfff640 1 -- 172.21.15.60:0/515571917 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f903c09afd0 con 0x7f9044100f30 2024-03-04T12:22:54.601 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:22:54 smithi063 systemd[1]: Starting Ceph mds.h for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:22:54.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.765+0000 7f904900b640 1 -- 172.21.15.60:0/515571917 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f904402ceb0 con 0x7f9044100f30 2024-03-04T12:22:54.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.765+0000 7f902bfff640 1 -- 172.21.15.60:0/515571917 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6059 (secure 0 0 0) 0x7f903c09a8d0 con 0x7f9044100f30 2024-03-04T12:22:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:54.771 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-04T12:22:54.772 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 8 2024-03-04T12:22:54.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.768+0000 7f904900b640 1 -- 172.21.15.60:0/515571917 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f902c0a1d80 msgr2=0x7f902c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:54.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.768+0000 7f904900b640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f902c0a1d80 0x7f902c0a4240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9034006010 tx=0x7f9034032590 comp rx=0 tx=0).stop 2024-03-04T12:22:54.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.768+0000 7f904900b640 1 -- 172.21.15.60:0/515571917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9044100f30 msgr2=0x7f90441a0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:54.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.768+0000 7f904900b640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9044100f30 0x7f90441a0e10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f903c009fd0 tx=0x7f903c00b8e0 comp rx=0 tx=0).stop 2024-03-04T12:22:54.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.768+0000 7f904900b640 1 -- 172.21.15.60:0/515571917 shutdown_connections 2024-03-04T12:22:54.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.768+0000 7f904900b640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f902c0a1d80 0x7f902c0a4240 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:54.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.768+0000 7f904900b640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9044100f30 0x7f90441a0e10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:54.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.768+0000 7f904900b640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9044100590 0x7f9044199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:54.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.768+0000 7f904900b640 1 --2- 172.21.15.60:0/515571917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90440ffc30 0x7f9044199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:54.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.768+0000 7f904900b640 1 -- 172.21.15.60:0/515571917 >> 172.21.15.60:0/515571917 conn(0x7f90440fb7d0 msgr2=0x7f90440fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:54.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.768+0000 7f904900b640 1 -- 172.21.15.60:0/515571917 shutdown_connections 2024-03-04T12:22:54.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:54.768+0000 7f904900b640 1 -- 172.21.15.60:0/515571917 wait complete. 2024-03-04T12:22:55.040 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:22:54 smithi063 podman[67084]: 2024-03-04 12:22:54.665699438 +0000 UTC m=+0.017998505 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:22:55.040 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:22:54 smithi063 podman[67084]: 2024-03-04 12:22:54.782056082 +0000 UTC m=+0.134355125 container create 6f810dfaa3daab6877bd52272a98aa0fc0f5dea907ed555c455ae0dd4851f0e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-04T12:22:55.040 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:22:55 smithi063 podman[67084]: 2024-03-04 12:22:55.035205749 +0000 UTC m=+0.387504818 container init 6f810dfaa3daab6877bd52272a98aa0fc0f5dea907ed555c455ae0dd4851f0e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:22:55.347 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:55 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/515571917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:55.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:55 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/515571917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:55.351 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:22:55 smithi063 podman[67084]: 2024-03-04 12:22:55.037883712 +0000 UTC m=+0.390182757 container start 6f810dfaa3daab6877bd52272a98aa0fc0f5dea907ed555c455ae0dd4851f0e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:22:55.351 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:22:55 smithi063 ceph-mds[67103]: -- 172.21.15.63:0/424110657 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564e5bd59a40 con 0x564e5bd2cc00 2024-03-04T12:22:55.351 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:22:55 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: starting mds.h at 2024-03-04T12:22:55.351 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:22:55 smithi063 bash[67084]: 6f810dfaa3daab6877bd52272a98aa0fc0f5dea907ed555c455ae0dd4851f0e8 2024-03-04T12:22:55.351 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:22:55 smithi063 systemd[1]: Started Ceph mds.h for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:22:55.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:55 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/515571917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:56.186 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:56 smithi086 ceph-mon[36851]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:56.186 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:56.187 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:56.187 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:56.187 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:56.187 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:56.187 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:56 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:56.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:56 smithi063 ceph-mon[36717]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:56.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:56.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:56.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:56.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:56.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:56.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:56 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:56.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:56 smithi060 ceph-mon[32745]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:56.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:56.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:56.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:56.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:56.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:56.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:56 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:56.786 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:22:56.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.895+0000 7f0dfd953640 1 -- 172.21.15.60:0/671641590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df81070a0 msgr2=0x7f0df8109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:56.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.895+0000 7f0dfd953640 1 --2- 172.21.15.60:0/671641590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df81070a0 0x7f0df8109490 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009ca0 tx=0x7f0df002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:22:56.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.895+0000 7f0dfd953640 1 -- 172.21.15.60:0/671641590 shutdown_connections 2024-03-04T12:22:56.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.895+0000 7f0dfd953640 1 --2- 172.21.15.60:0/671641590 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df81070a0 0x7f0df8109490 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:56.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.895+0000 7f0dfd953640 1 --2- 172.21.15.60:0/671641590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0df8104770 0x7f0df8106b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:56.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.895+0000 7f0dfd953640 1 --2- 172.21.15.60:0/671641590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8101e40 0x7f0df8104230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:56.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.895+0000 7f0dfd953640 1 -- 172.21.15.60:0/671641590 >> 172.21.15.60:0/671641590 conn(0x7f0df80fb7f0 msgr2=0x7f0df80fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.895+0000 7f0dfd953640 1 -- 172.21.15.60:0/671641590 shutdown_connections 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.895+0000 7f0dfd953640 1 -- 172.21.15.60:0/671641590 wait complete. 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0dfd953640 1 Processor -- start 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0dfd953640 1 -- start start 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0dfd953640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8101e40 0x7f0df8197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0dfd953640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df8104770 0x7f0df8197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0dfd953640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0df81070a0 0x7f0df819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0dfd953640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df810cff0 con 0x7f0df8104770 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0dfd953640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df810ce70 con 0x7f0df8101e40 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0dfd953640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df810d170 con 0x7f0df81070a0 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0deffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df8104770 0x7f0df8197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0deffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df8104770 0x7f0df8197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37566/0 (socket says 172.21.15.60:37566) 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0deffff640 1 -- 172.21.15.60:0/2648937826 learned_addr learned my addr 172.21.15.60:0/2648937826 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:56.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0df6ffd640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8101e40 0x7f0df8197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0df77fe640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0df81070a0 0x7f0df819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0deffff640 1 -- 172.21.15.60:0/2648937826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0df81070a0 msgr2=0x7f0df819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0deffff640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0df81070a0 0x7f0df819ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0deffff640 1 -- 172.21.15.60:0/2648937826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8101e40 msgr2=0x7f0df8197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0deffff640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8101e40 0x7f0df8197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.896+0000 7f0deffff640 1 -- 172.21.15.60:0/2648937826 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df819f220 con 0x7f0df8104770 2024-03-04T12:22:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.897+0000 7f0deffff640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df8104770 0x7f0df8197b40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0de000d740 tx=0x7f0de000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.897+0000 7f0dfc951640 1 -- 172.21.15.60:0/2648937826 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de0017070 con 0x7f0df8104770 2024-03-04T12:22:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.897+0000 7f0dfd953640 1 -- 172.21.15.60:0/2648937826 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df819f510 con 0x7f0df8104770 2024-03-04T12:22:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.897+0000 7f0dfc951640 1 -- 172.21.15.60:0/2648937826 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0de000de50 con 0x7f0df8104770 2024-03-04T12:22:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.897+0000 7f0dfc951640 1 -- 172.21.15.60:0/2648937826 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de0004e00 con 0x7f0df8104770 2024-03-04T12:22:56.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.897+0000 7f0dfd953640 1 -- 172.21.15.60:0/2648937826 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df819fa60 con 0x7f0df8104770 2024-03-04T12:22:56.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.898+0000 7f0dfd953640 1 -- 172.21.15.60:0/2648937826 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df80fd2e0 con 0x7f0df8104770 2024-03-04T12:22:56.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.899+0000 7f0dfc951640 1 -- 172.21.15.60:0/2648937826 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f0de0004fa0 con 0x7f0df8104770 2024-03-04T12:22:56.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.900+0000 7f0dfc951640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0de40a1d80 0x7f0de40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:56.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.900+0000 7f0df6ffd640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0de40a1d80 0x7f0de40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:56.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.900+0000 7f0dfc951640 1 -- 172.21.15.60:0/2648937826 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0de00cd350 con 0x7f0df8104770 2024-03-04T12:22:56.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.900+0000 7f0df6ffd640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0de40a1d80 0x7f0de40a4240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009bd0 tx=0x7f0de8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:56.910 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:56.904+0000 7f0dfc951640 1 -- 172.21.15.60:0/2648937826 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0de009ab80 con 0x7f0df8104770 2024-03-04T12:22:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.122+0000 7f0dfd953640 1 -- 172.21.15.60:0/2648937826 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0df8061f30 con 0x7f0df8104770 2024-03-04T12:22:57.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.123+0000 7f0dfc951640 1 -- 172.21.15.60:0/2648937826 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6850 (secure 0 0 0) 0x7f0de009a480 con 0x7f0df8104770 2024-03-04T12:22:57.129 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:57.129 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-04T12:22:57.129 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 9 2024-03-04T12:22:57.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.125+0000 7f0dfd953640 1 -- 172.21.15.60:0/2648937826 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0de40a1d80 msgr2=0x7f0de40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.125+0000 7f0dfd953640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0de40a1d80 0x7f0de40a4240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009bd0 tx=0x7f0de8005f90 comp rx=0 tx=0).stop 2024-03-04T12:22:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.125+0000 7f0dfd953640 1 -- 172.21.15.60:0/2648937826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df8104770 msgr2=0x7f0df8197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.125+0000 7f0dfd953640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df8104770 0x7f0df8197b40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0de000d740 tx=0x7f0de000dc00 comp rx=0 tx=0).stop 2024-03-04T12:22:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.125+0000 7f0dfd953640 1 -- 172.21.15.60:0/2648937826 shutdown_connections 2024-03-04T12:22:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.125+0000 7f0dfd953640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0de40a1d80 0x7f0de40a4240 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.125+0000 7f0dfd953640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0df81070a0 0x7f0df819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.125+0000 7f0dfd953640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0df8104770 0x7f0df8197b40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.125+0000 7f0dfd953640 1 --2- 172.21.15.60:0/2648937826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0df8101e40 0x7f0df8197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.125+0000 7f0dfd953640 1 -- 172.21.15.60:0/2648937826 >> 172.21.15.60:0/2648937826 conn(0x7f0df80fb7f0 msgr2=0x7f0df81026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.125+0000 7f0dfd953640 1 -- 172.21.15.60:0/2648937826 shutdown_connections 2024-03-04T12:22:57.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:57.125+0000 7f0dfd953640 1 -- 172.21.15.60:0/2648937826 wait complete. 2024-03-04T12:22:57.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:57 smithi086 ceph-mon[36851]: Deploying daemon mds.i on smithi086 2024-03-04T12:22:57.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:57 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] up:boot 2024-03-04T12:22:57.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:57 smithi086 ceph-mon[36851]: fsmap 8 up:standby 2024-03-04T12:22:57.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:57 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-04T12:22:57.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:57 smithi063 ceph-mon[36717]: Deploying daemon mds.i on smithi086 2024-03-04T12:22:57.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:57 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] up:boot 2024-03-04T12:22:57.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:57 smithi063 ceph-mon[36717]: fsmap 8 up:standby 2024-03-04T12:22:57.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:57 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-04T12:22:57.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:57 smithi060 ceph-mon[32745]: Deploying daemon mds.i on smithi086 2024-03-04T12:22:57.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:57 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] up:boot 2024-03-04T12:22:57.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:57 smithi060 ceph-mon[32745]: fsmap 8 up:standby 2024-03-04T12:22:57.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:57 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-04T12:22:58.088 INFO:journalctl@ceph.mds.i.smithi086.stdout:Mar 04 12:22:57 smithi086 systemd[1]: Started Ceph mds.i for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:22:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:58 smithi086 ceph-mon[36851]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:58 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2648937826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:58 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:58 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:58 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:58 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:58 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:58 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:58 smithi063 ceph-mon[36717]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:58 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2648937826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:58 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:58 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:58 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:58 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:58 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:58 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:58 smithi060 ceph-mon[32745]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:22:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:58 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2648937826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:22:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:58 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:58 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:58 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:22:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:58 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:22:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:58 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:22:58.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:58 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:22:59.143 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:22:59.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.246+0000 7fea49481640 1 -- 172.21.15.60:0/2774468966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea44102f10 msgr2=0x7fea441097a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.246+0000 7fea49481640 1 --2- 172.21.15.60:0/2774468966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea44102f10 0x7fea441097a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fea34009ca0 tx=0x7fea3402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 -- 172.21.15.60:0/2774468966 shutdown_connections 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 --2- 172.21.15.60:0/2774468966 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea44102f10 0x7fea441097a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 --2- 172.21.15.60:0/2774468966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea44102550 0x7fea441029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 --2- 172.21.15.60:0/2774468966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea44101360 0x7fea44101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 -- 172.21.15.60:0/2774468966 >> 172.21.15.60:0/2774468966 conn(0x7fea440fc910 msgr2=0x7fea440fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 -- 172.21.15.60:0/2774468966 shutdown_connections 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 -- 172.21.15.60:0/2774468966 wait complete. 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 Processor -- start 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 -- start start 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea44101360 0x7fea44197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea44102550 0x7fea44197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea44102f10 0x7fea4419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:59.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4410cff0 con 0x7fea44102f10 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4410ce70 con 0x7fea44102550 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.247+0000 7fea49481640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea4410d170 con 0x7fea44101360 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea437fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea44102f10 0x7fea4419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea437fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea44102f10 0x7fea4419ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37584/0 (socket says 172.21.15.60:37584) 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea437fe640 1 -- 172.21.15.60:0/947422639 learned_addr learned my addr 172.21.15.60:0/947422639 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea427fc640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea44102550 0x7fea44197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea42ffd640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea44101360 0x7fea44197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea437fe640 1 -- 172.21.15.60:0/947422639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea44101360 msgr2=0x7fea44197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea437fe640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea44101360 0x7fea44197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea437fe640 1 -- 172.21.15.60:0/947422639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea44102550 msgr2=0x7fea44197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea437fe640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea44102550 0x7fea44197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea437fe640 1 -- 172.21.15.60:0/947422639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4419f230 con 0x7fea44102f10 2024-03-04T12:22:59.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea42ffd640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea44101360 0x7fea44197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:22:59.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea437fe640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea44102f10 0x7fea4419ebc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fea34037040 tx=0x7fea34032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:59.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea3bfff640 1 -- 172.21.15.60:0/947422639 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea34041020 con 0x7fea44102f10 2024-03-04T12:22:59.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea3bfff640 1 -- 172.21.15.60:0/947422639 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea34034d90 con 0x7fea44102f10 2024-03-04T12:22:59.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea49481640 1 -- 172.21.15.60:0/947422639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea4419f4c0 con 0x7fea44102f10 2024-03-04T12:22:59.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea3bfff640 1 -- 172.21.15.60:0/947422639 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea34039040 con 0x7fea44102f10 2024-03-04T12:22:59.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.248+0000 7fea49481640 1 -- 172.21.15.60:0/947422639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4419f9b0 con 0x7fea44102f10 2024-03-04T12:22:59.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.249+0000 7fea49481640 1 -- 172.21.15.60:0/947422639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea44108620 con 0x7fea44102f10 2024-03-04T12:22:59.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.250+0000 7fea3bfff640 1 -- 172.21.15.60:0/947422639 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fea34044020 con 0x7fea44102f10 2024-03-04T12:22:59.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.251+0000 7fea3bfff640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fea300a1d80 0x7fea300a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:22:59.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.251+0000 7fea42ffd640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fea300a1d80 0x7fea300a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:22:59.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.251+0000 7fea3bfff640 1 -- 172.21.15.60:0/947422639 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fea340ef940 con 0x7fea44102f10 2024-03-04T12:22:59.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.251+0000 7fea42ffd640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fea300a1d80 0x7fea300a4240 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fea441023b0 tx=0x7fea3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:22:59.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.255+0000 7fea3bfff640 1 -- 172.21.15.60:0/947422639 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fea3404b020 con 0x7fea44102f10 2024-03-04T12:22:59.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:59 smithi063 ceph-mon[36717]: Deploying daemon mds.g on smithi060 2024-03-04T12:22:59.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:59 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] up:boot 2024-03-04T12:22:59.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:59 smithi063 ceph-mon[36717]: fsmap 9 up:standby 2024-03-04T12:22:59.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:22:59 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-04T12:22:59.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:59 smithi060 ceph-mon[32745]: Deploying daemon mds.g on smithi060 2024-03-04T12:22:59.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:59 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] up:boot 2024-03-04T12:22:59.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:59 smithi060 ceph-mon[32745]: fsmap 9 up:standby 2024-03-04T12:22:59.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:22:59 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-04T12:22:59.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.476+0000 7fea49481640 1 -- 172.21.15.60:0/947422639 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea44061f30 con 0x7fea44102f10 2024-03-04T12:22:59.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.476+0000 7fea3bfff640 1 -- 172.21.15.60:0/947422639 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7643 (secure 0 0 0) 0x7fea340bd170 con 0x7fea44102f10 2024-03-04T12:22:59.482 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:22:59.482 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-04T12:22:59.482 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 10 2024-03-04T12:22:59.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.478+0000 7fea49481640 1 -- 172.21.15.60:0/947422639 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fea300a1d80 msgr2=0x7fea300a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:59.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.478+0000 7fea49481640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fea300a1d80 0x7fea300a4240 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fea441023b0 tx=0x7fea3c005f90 comp rx=0 tx=0).stop 2024-03-04T12:22:59.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.478+0000 7fea49481640 1 -- 172.21.15.60:0/947422639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea44102f10 msgr2=0x7fea4419ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:22:59.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.478+0000 7fea49481640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea44102f10 0x7fea4419ebc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fea34037040 tx=0x7fea34032f20 comp rx=0 tx=0).stop 2024-03-04T12:22:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.478+0000 7fea49481640 1 -- 172.21.15.60:0/947422639 shutdown_connections 2024-03-04T12:22:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.478+0000 7fea49481640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fea300a1d80 0x7fea300a4240 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.478+0000 7fea49481640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fea44102f10 0x7fea4419ebc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.478+0000 7fea49481640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea44102550 0x7fea44197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.478+0000 7fea49481640 1 --2- 172.21.15.60:0/947422639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea44101360 0x7fea44197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:22:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.478+0000 7fea49481640 1 -- 172.21.15.60:0/947422639 >> 172.21.15.60:0/947422639 conn(0x7fea440fc910 msgr2=0x7fea441057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:22:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.478+0000 7fea49481640 1 -- 172.21.15.60:0/947422639 shutdown_connections 2024-03-04T12:22:59.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:22:59.478+0000 7fea49481640 1 -- 172.21.15.60:0/947422639 wait complete. 2024-03-04T12:22:59.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:59 smithi086 ceph-mon[36851]: Deploying daemon mds.g on smithi060 2024-03-04T12:22:59.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:59 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] up:boot 2024-03-04T12:22:59.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:59 smithi086 ceph-mon[36851]: fsmap 9 up:standby 2024-03-04T12:22:59.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:22:59 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-04T12:23:00.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:00 smithi060 ceph-mon[32745]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:00.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:00 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/947422639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:00.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:00 smithi063 ceph-mon[36717]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:00.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:00 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/947422639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:00.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:00 smithi086 ceph-mon[36851]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:00.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:00 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/947422639' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:00.928 INFO:journalctl@ceph.mds.g.smithi060.stdout:Mar 04 12:23:00 smithi060 systemd[1]: Started Ceph mds.g for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:23:01.496 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:23:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.599+0000 7f6a77227640 1 -- 172.21.15.60:0/1652963071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a70101de0 msgr2=0x7f6a701041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.599+0000 7f6a77227640 1 --2- 172.21.15.60:0/1652963071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a70101de0 0x7f6a701041d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6a68009ca0 tx=0x7f6a6802c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.599+0000 7f6a77227640 1 -- 172.21.15.60:0/1652963071 shutdown_connections 2024-03-04T12:23:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.599+0000 7f6a77227640 1 --2- 172.21.15.60:0/1652963071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a70107040 0x7f6a70109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.599+0000 7f6a77227640 1 --2- 172.21.15.60:0/1652963071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a70104710 0x7f6a70106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.599+0000 7f6a77227640 1 --2- 172.21.15.60:0/1652963071 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a70101de0 0x7f6a701041d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.599+0000 7f6a77227640 1 -- 172.21.15.60:0/1652963071 >> 172.21.15.60:0/1652963071 conn(0x7f6a700fb7d0 msgr2=0x7f6a700fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.599+0000 7f6a77227640 1 -- 172.21.15.60:0/1652963071 shutdown_connections 2024-03-04T12:23:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.599+0000 7f6a77227640 1 -- 172.21.15.60:0/1652963071 wait complete. 2024-03-04T12:23:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.599+0000 7f6a77227640 1 Processor -- start 2024-03-04T12:23:01.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.599+0000 7f6a77227640 1 -- start start 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.599+0000 7f6a77227640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a70101de0 0x7f6a70197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a77227640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a70104710 0x7f6a70197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a77227640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a70107040 0x7f6a7019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a77227640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7010cff0 con 0x7f6a70107040 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a77227640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7010ce70 con 0x7f6a70104710 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a77227640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7010d170 con 0x7f6a70101de0 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a757c4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a70107040 0x7f6a7019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a757c4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a70107040 0x7f6a7019ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37612/0 (socket says 172.21.15.60:37612) 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a757c4640 1 -- 172.21.15.60:0/3232750489 learned_addr learned my addr 172.21.15.60:0/3232750489 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a6ffff640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a70104710 0x7f6a70197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a74fc3640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a70101de0 0x7f6a70197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a757c4640 1 -- 172.21.15.60:0/3232750489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a70101de0 msgr2=0x7f6a70197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a757c4640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a70101de0 0x7f6a70197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:01.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a757c4640 1 -- 172.21.15.60:0/3232750489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a70104710 msgr2=0x7f6a70197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a757c4640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a70104710 0x7f6a70197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a757c4640 1 -- 172.21.15.60:0/3232750489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7019f260 con 0x7f6a70107040 2024-03-04T12:23:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a6ffff640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a70104710 0x7f6a70197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a757c4640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a70107040 0x7f6a7019ebf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6a6000e970 tx=0x7f6a6000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a6dffb640 1 -- 172.21.15.60:0/3232750489 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6000ccc0 con 0x7f6a70107040 2024-03-04T12:23:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a6dffb640 1 -- 172.21.15.60:0/3232750489 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a6000ce60 con 0x7f6a70107040 2024-03-04T12:23:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a77227640 1 -- 172.21.15.60:0/3232750489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7019f550 con 0x7f6a70107040 2024-03-04T12:23:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a6dffb640 1 -- 172.21.15.60:0/3232750489 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60010490 con 0x7f6a70107040 2024-03-04T12:23:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.600+0000 7f6a77227640 1 -- 172.21.15.60:0/3232750489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7019faa0 con 0x7f6a70107040 2024-03-04T12:23:01.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.601+0000 7f6a77227640 1 -- 172.21.15.60:0/3232750489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a700fd2c0 con 0x7f6a70107040 2024-03-04T12:23:01.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.602+0000 7f6a6dffb640 1 -- 172.21.15.60:0/3232750489 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f6a60005020 con 0x7f6a70107040 2024-03-04T12:23:01.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.603+0000 7f6a6dffb640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6a580a1e00 0x7f6a580a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:01.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.603+0000 7f6a6dffb640 1 -- 172.21.15.60:0/3232750489 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6a600cde60 con 0x7f6a70107040 2024-03-04T12:23:01.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.603+0000 7f6a74fc3640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6a580a1e00 0x7f6a580a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:01.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.603+0000 7f6a74fc3640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6a580a1e00 0x7f6a580a42c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6a68006010 tx=0x7f6a680093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:01.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.608+0000 7f6a6dffb640 1 -- 172.21.15.60:0/3232750489 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6a6009b690 con 0x7f6a70107040 2024-03-04T12:23:01.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.835+0000 7f6a77227640 1 -- 172.21.15.60:0/3232750489 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a70061f30 con 0x7f6a70107040 2024-03-04T12:23:01.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.836+0000 7f6a6dffb640 1 -- 172.21.15.60:0/3232750489 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8435 (secure 0 0 0) 0x7f6a6009af90 con 0x7f6a70107040 2024-03-04T12:23:01.842 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:01.842 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-04T12:23:01.842 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 11 2024-03-04T12:23:01.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.838+0000 7f6a77227640 1 -- 172.21.15.60:0/3232750489 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6a580a1e00 msgr2=0x7f6a580a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:01.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.838+0000 7f6a77227640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6a580a1e00 0x7f6a580a42c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6a68006010 tx=0x7f6a680093f0 comp rx=0 tx=0).stop 2024-03-04T12:23:01.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.838+0000 7f6a77227640 1 -- 172.21.15.60:0/3232750489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a70107040 msgr2=0x7f6a7019ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:01.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.838+0000 7f6a77227640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a70107040 0x7f6a7019ebf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6a6000e970 tx=0x7f6a6000ed70 comp rx=0 tx=0).stop 2024-03-04T12:23:01.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.838+0000 7f6a77227640 1 -- 172.21.15.60:0/3232750489 shutdown_connections 2024-03-04T12:23:01.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.838+0000 7f6a77227640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6a580a1e00 0x7f6a580a42c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:01.844 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.838+0000 7f6a77227640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a70107040 0x7f6a7019ebf0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:01.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.838+0000 7f6a77227640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a70104710 0x7f6a70197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:01.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.838+0000 7f6a77227640 1 --2- 172.21.15.60:0/3232750489 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a70101de0 0x7f6a70197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:01.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.838+0000 7f6a77227640 1 -- 172.21.15.60:0/3232750489 >> 172.21.15.60:0/3232750489 conn(0x7f6a700fb7d0 msgr2=0x7f6a70102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:01.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.838+0000 7f6a77227640 1 -- 172.21.15.60:0/3232750489 shutdown_connections 2024-03-04T12:23:01.845 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:01.838+0000 7f6a77227640 1 -- 172.21.15.60:0/3232750489 wait complete. 2024-03-04T12:23:01.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:01 smithi060 ceph-mon[32745]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:01.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:01 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:01.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:01 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:01.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:01 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:01.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:01 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:23:01.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:01 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:23:01.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:01 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:23:01.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:01 smithi060 ceph-mon[32745]: Deploying daemon mds.a on smithi060 2024-03-04T12:23:01.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:01 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.60:6838/2180757903,v1:172.21.15.60:6839/2180757903] up:boot 2024-03-04T12:23:01.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:01 smithi060 ceph-mon[32745]: fsmap 10 up:standby 2024-03-04T12:23:01.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:01 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T12:23:02.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:01 smithi086 ceph-mon[36851]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:02.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:01 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:02.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:01 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:02.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:01 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:02.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:01 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:23:02.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:01 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:23:02.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:01 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:23:02.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:01 smithi086 ceph-mon[36851]: Deploying daemon mds.a on smithi060 2024-03-04T12:23:02.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:01 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.60:6838/2180757903,v1:172.21.15.60:6839/2180757903] up:boot 2024-03-04T12:23:02.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:01 smithi086 ceph-mon[36851]: fsmap 10 up:standby 2024-03-04T12:23:02.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:01 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T12:23:02.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:01 smithi063 ceph-mon[36717]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:02.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:01 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:02.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:01 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:02.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:01 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:02.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:01 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:23:02.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:01 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:23:02.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:01 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:23:02.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:01 smithi063 ceph-mon[36717]: Deploying daemon mds.a on smithi060 2024-03-04T12:23:02.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:01 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.60:6838/2180757903,v1:172.21.15.60:6839/2180757903] up:boot 2024-03-04T12:23:02.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:01 smithi063 ceph-mon[36717]: fsmap 10 up:standby 2024-03-04T12:23:02.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:01 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T12:23:02.967 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:02 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3232750489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:02.968 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:23:02 smithi060 systemd[1]: Starting Ceph mds.a for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:23:03.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:02 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3232750489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:03.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:02 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3232750489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:03.251 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:23:03 smithi060 podman[75208]: 2024-03-04 12:23:03.021004625 +0000 UTC m=+0.018145408 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:23:03.251 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:23:03 smithi060 podman[75208]: 2024-03-04 12:23:03.246351303 +0000 UTC m=+0.243492084 container create 5b43f6f536a382f17cc9d07216325579b2c6d63e06057afda41b7c1f7d798996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:23:03.856 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:23:03.881 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:03 smithi060 ceph-mon[32745]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:03.881 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:03.882 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:03.882 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:03.882 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:23:03.882 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:23:03.882 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:03 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:23:03.882 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:23:03 smithi060 podman[75208]: 2024-03-04 12:23:03.507432735 +0000 UTC m=+0.504573530 container init 5b43f6f536a382f17cc9d07216325579b2c6d63e06057afda41b7c1f7d798996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux ) 2024-03-04T12:23:03.882 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:23:03 smithi060 podman[75208]: 2024-03-04 12:23:03.510768873 +0000 UTC m=+0.507909664 container start 5b43f6f536a382f17cc9d07216325579b2c6d63e06057afda41b7c1f7d798996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-04T12:23:03.882 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:23:03 smithi060 ceph-mds[75237]: -- 172.21.15.60:0/2969873469 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55df25d77a40 con 0x55df25d4a800 2024-03-04T12:23:03.882 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:23:03 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a[75233]: starting mds.a at 2024-03-04T12:23:03.882 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:23:03 smithi060 bash[75208]: 5b43f6f536a382f17cc9d07216325579b2c6d63e06057afda41b7c1f7d798996 2024-03-04T12:23:03.882 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:23:03 smithi060 systemd[1]: Started Ceph mds.a for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:23:03.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.957+0000 7fa973f7e640 1 -- 172.21.15.60:0/787574278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa96c101de0 msgr2=0x7fa96c1041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:03.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.957+0000 7fa973f7e640 1 --2- 172.21.15.60:0/787574278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa96c101de0 0x7fa96c1041d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa968009900 tx=0x7fa96802bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:03.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.957+0000 7fa973f7e640 1 -- 172.21.15.60:0/787574278 shutdown_connections 2024-03-04T12:23:03.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.957+0000 7fa973f7e640 1 --2- 172.21.15.60:0/787574278 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa96c107040 0x7fa96c109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:03.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.957+0000 7fa973f7e640 1 --2- 172.21.15.60:0/787574278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96c104710 0x7fa96c106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.957+0000 7fa973f7e640 1 --2- 172.21.15.60:0/787574278 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa96c101de0 0x7fa96c1041d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.957+0000 7fa973f7e640 1 -- 172.21.15.60:0/787574278 >> 172.21.15.60:0/787574278 conn(0x7fa96c0fb7f0 msgr2=0x7fa96c0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.957+0000 7fa973f7e640 1 -- 172.21.15.60:0/787574278 shutdown_connections 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.957+0000 7fa973f7e640 1 -- 172.21.15.60:0/787574278 wait complete. 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa973f7e640 1 Processor -- start 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa973f7e640 1 -- start start 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa973f7e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa96c101de0 0x7fa96c1976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa973f7e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96c104710 0x7fa96c197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa973f7e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa96c107040 0x7fa96c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa973f7e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96c10cff0 con 0x7fa96c101de0 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa973f7e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96c10ce70 con 0x7fa96c107040 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa973f7e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96c10d170 con 0x7fa96c104710 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa971d1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa96c101de0 0x7fa96c1976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:03.966 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa971d1a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa96c101de0 0x7fa96c1976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37654/0 (socket says 172.21.15.60:37654) 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa971d1a640 1 -- 172.21.15.60:0/2701194025 learned_addr learned my addr 172.21.15.60:0/2701194025 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa97251b640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa96c107040 0x7fa96c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa971519640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96c104710 0x7fa96c197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa971d1a640 1 -- 172.21.15.60:0/2701194025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96c104710 msgr2=0x7fa96c197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa971d1a640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96c104710 0x7fa96c197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa971d1a640 1 -- 172.21.15.60:0/2701194025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa96c107040 msgr2=0x7fa96c19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.958+0000 7fa971d1a640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa96c107040 0x7fa96c19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.959+0000 7fa971d1a640 1 -- 172.21.15.60:0/2701194025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c19f2e0 con 0x7fa96c101de0 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.959+0000 7fa971d1a640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa96c101de0 0x7fa96c1976c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa968009230 tx=0x7fa968005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.959+0000 7fa962ffd640 1 -- 172.21.15.60:0/2701194025 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968041070 con 0x7fa96c101de0 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.959+0000 7fa973f7e640 1 -- 172.21.15.60:0/2701194025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96c19f570 con 0x7fa96c101de0 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.959+0000 7fa962ffd640 1 -- 172.21.15.60:0/2701194025 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9680054e0 con 0x7fa96c101de0 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.959+0000 7fa962ffd640 1 -- 172.21.15.60:0/2701194025 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968034440 con 0x7fa96c101de0 2024-03-04T12:23:03.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.959+0000 7fa973f7e640 1 -- 172.21.15.60:0/2701194025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96c19fa60 con 0x7fa96c101de0 2024-03-04T12:23:03.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.960+0000 7fa973f7e640 1 -- 172.21.15.60:0/2701194025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa96c0fd2e0 con 0x7fa96c101de0 2024-03-04T12:23:03.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.961+0000 7fa962ffd640 1 -- 172.21.15.60:0/2701194025 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fa968044070 con 0x7fa96c101de0 2024-03-04T12:23:03.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.961+0000 7fa962ffd640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa9580a1d80 0x7fa9580a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:03.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.962+0000 7fa971519640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa9580a1d80 0x7fa9580a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:03.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.962+0000 7fa962ffd640 1 -- 172.21.15.60:0/2701194025 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa9680ef5f0 con 0x7fa96c101de0 2024-03-04T12:23:03.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.962+0000 7fa971519640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa9580a1d80 0x7fa9580a4240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa95c005fd0 tx=0x7fa95c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:03.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:03.966+0000 7fa962ffd640 1 -- 172.21.15.60:0/2701194025 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa9680bcec0 con 0x7fa96c101de0 2024-03-04T12:23:04.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:03 smithi086 ceph-mon[36851]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:04.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:04.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:04.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:04.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:23:04.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:23:04.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:03 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:23:04.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:03 smithi063 ceph-mon[36717]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:04.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:04.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:04.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:04.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-04T12:23:04.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-04T12:23:04.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:03 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:23:04.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.190+0000 7fa973f7e640 1 -- 172.21.15.60:0/2701194025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa96c02d0d0 con 0x7fa96c101de0 2024-03-04T12:23:04.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.190+0000 7fa962ffd640 1 -- 172.21.15.60:0/2701194025 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9227 (secure 0 0 0) 0x7fa9680bc7c0 con 0x7fa96c101de0 2024-03-04T12:23:04.196 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:04.196 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-04T12:23:04.196 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 12 2024-03-04T12:23:04.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.192+0000 7fa973f7e640 1 -- 172.21.15.60:0/2701194025 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa9580a1d80 msgr2=0x7fa9580a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:04.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.192+0000 7fa973f7e640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa9580a1d80 0x7fa9580a4240 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa95c005fd0 tx=0x7fa95c005850 comp rx=0 tx=0).stop 2024-03-04T12:23:04.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.192+0000 7fa973f7e640 1 -- 172.21.15.60:0/2701194025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa96c101de0 msgr2=0x7fa96c1976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:04.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.192+0000 7fa973f7e640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa96c101de0 0x7fa96c1976c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa968009230 tx=0x7fa968005dd0 comp rx=0 tx=0).stop 2024-03-04T12:23:04.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.193+0000 7fa973f7e640 1 -- 172.21.15.60:0/2701194025 shutdown_connections 2024-03-04T12:23:04.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.193+0000 7fa973f7e640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa9580a1d80 0x7fa9580a4240 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:04.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.193+0000 7fa973f7e640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa96c107040 0x7fa96c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:04.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.193+0000 7fa973f7e640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96c104710 0x7fa96c197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:04.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.193+0000 7fa973f7e640 1 --2- 172.21.15.60:0/2701194025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa96c101de0 0x7fa96c1976c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:04.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.193+0000 7fa973f7e640 1 -- 172.21.15.60:0/2701194025 >> 172.21.15.60:0/2701194025 conn(0x7fa96c0fb7f0 msgr2=0x7fa96c102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:04.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.193+0000 7fa973f7e640 1 -- 172.21.15.60:0/2701194025 shutdown_connections 2024-03-04T12:23:04.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:04.193+0000 7fa973f7e640 1 -- 172.21.15.60:0/2701194025 wait complete. 2024-03-04T12:23:04.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:04 smithi063 ceph-mon[36717]: Deploying daemon mds.b on smithi063 2024-03-04T12:23:04.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:04 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.60:6840/4175833227,v1:172.21.15.60:6841/4175833227] up:boot 2024-03-04T12:23:04.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:04 smithi063 ceph-mon[36717]: fsmap 11 up:standby 2024-03-04T12:23:04.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-04T12:23:04.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:04 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2701194025' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:05.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:04 smithi086 ceph-mon[36851]: Deploying daemon mds.b on smithi063 2024-03-04T12:23:05.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:04 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.60:6840/4175833227,v1:172.21.15.60:6841/4175833227] up:boot 2024-03-04T12:23:05.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:04 smithi086 ceph-mon[36851]: fsmap 11 up:standby 2024-03-04T12:23:05.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-04T12:23:05.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:04 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2701194025' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:05.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:04 smithi060 ceph-mon[32745]: Deploying daemon mds.b on smithi063 2024-03-04T12:23:05.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:04 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.60:6840/4175833227,v1:172.21.15.60:6841/4175833227] up:boot 2024-03-04T12:23:05.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:04 smithi060 ceph-mon[32745]: fsmap 11 up:standby 2024-03-04T12:23:05.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-04T12:23:05.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:04 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2701194025' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:06.083 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:05 smithi063 ceph-mon[36717]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:06.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:05 smithi086 ceph-mon[36851]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:06.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:05 smithi060 ceph-mon[32745]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:06.210 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:23:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.319+0000 7f1fca1c6640 1 -- 172.21.15.60:0/2143697369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc41051a0 msgr2=0x7f1fc410ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.319+0000 7f1fca1c6640 1 --2- 172.21.15.60:0/2143697369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc41051a0 0x7f1fc410ba30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc009900 tx=0x7f1fbc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.319+0000 7f1fca1c6640 1 -- 172.21.15.60:0/2143697369 shutdown_connections 2024-03-04T12:23:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.319+0000 7f1fca1c6640 1 --2- 172.21.15.60:0/2143697369 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc41051a0 0x7f1fc410ba30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.319+0000 7f1fca1c6640 1 --2- 172.21.15.60:0/2143697369 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fc41047e0 0x7f1fc4104c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.319+0000 7f1fca1c6640 1 --2- 172.21.15.60:0/2143697369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fc41035f0 0x7f1fc4103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.319+0000 7f1fca1c6640 1 -- 172.21.15.60:0/2143697369 >> 172.21.15.60:0/2143697369 conn(0x7f1fc40feb40 msgr2=0x7f1fc4100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:06.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.319+0000 7f1fca1c6640 1 -- 172.21.15.60:0/2143697369 shutdown_connections 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.319+0000 7f1fca1c6640 1 -- 172.21.15.60:0/2143697369 wait complete. 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.320+0000 7f1fca1c6640 1 Processor -- start 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.320+0000 7f1fca1c6640 1 -- start start 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.320+0000 7f1fca1c6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fc41035f0 0x7f1fc4199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.320+0000 7f1fca1c6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc41047e0 0x7f1fc4199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.320+0000 7f1fca1c6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fc41051a0 0x7f1fc41a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.320+0000 7f1fca1c6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc410f210 con 0x7f1fc41047e0 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.320+0000 7f1fca1c6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc410f090 con 0x7f1fc41051a0 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.320+0000 7f1fca1c6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc410f390 con 0x7f1fc41035f0 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.320+0000 7f1fc2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc41047e0 0x7f1fc4199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.320+0000 7f1fc2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc41047e0 0x7f1fc4199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54664/0 (socket says 172.21.15.60:54664) 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.320+0000 7f1fc2ffd640 1 -- 172.21.15.60:0/3989617030 learned_addr learned my addr 172.21.15.60:0/3989617030 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fc3fff640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fc41051a0 0x7f1fc41a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:06.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fc37fe640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fc41035f0 0x7f1fc4199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:06.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fc2ffd640 1 -- 172.21.15.60:0/3989617030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fc41035f0 msgr2=0x7f1fc4199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:06.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fc2ffd640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fc41035f0 0x7f1fc4199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:06.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fc2ffd640 1 -- 172.21.15.60:0/3989617030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fc41051a0 msgr2=0x7f1fc41a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:06.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fc2ffd640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fc41051a0 0x7f1fc41a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:06.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fc2ffd640 1 -- 172.21.15.60:0/3989617030 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc41a1350 con 0x7f1fc41047e0 2024-03-04T12:23:06.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fc2ffd640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc41047e0 0x7f1fc4199c70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800d950 tx=0x7f1fb800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:06.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fc0ff9640 1 -- 172.21.15.60:0/3989617030 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8017030 con 0x7f1fc41047e0 2024-03-04T12:23:06.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fc0ff9640 1 -- 172.21.15.60:0/3989617030 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fb8004020 con 0x7f1fc41047e0 2024-03-04T12:23:06.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fca1c6640 1 -- 172.21.15.60:0/3989617030 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc41a1640 con 0x7f1fc41047e0 2024-03-04T12:23:06.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fc0ff9640 1 -- 172.21.15.60:0/3989617030 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb80047a0 con 0x7f1fc41047e0 2024-03-04T12:23:06.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.321+0000 7f1fca1c6640 1 -- 172.21.15.60:0/3989617030 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc41a1be0 con 0x7f1fc41047e0 2024-03-04T12:23:06.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.322+0000 7f1fca1c6640 1 -- 172.21.15.60:0/3989617030 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc410a8b0 con 0x7f1fc41047e0 2024-03-04T12:23:06.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.323+0000 7f1fc0ff9640 1 -- 172.21.15.60:0/3989617030 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f1fb80041d0 con 0x7f1fc41047e0 2024-03-04T12:23:06.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.324+0000 7f1fc0ff9640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1fac0a1e00 0x7f1fac0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:06.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.324+0000 7f1fc37fe640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1fac0a1e00 0x7f1fac0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:06.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.324+0000 7f1fc0ff9640 1 -- 172.21.15.60:0/3989617030 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1fb80cd850 con 0x7f1fc41047e0 2024-03-04T12:23:06.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.324+0000 7f1fc37fe640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1fac0a1e00 0x7f1fac0a42c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4104640 tx=0x7f1fb4007600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:06.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.328+0000 7f1fc0ff9640 1 -- 172.21.15.60:0/3989617030 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1fb809b1a0 con 0x7f1fc41047e0 2024-03-04T12:23:06.334 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:23:06 smithi063 podman[67395]: 2024-03-04 12:23:06.329078177 +0000 UTC m=+0.147320174 container create e613f71e49fc41ffb30cd24caca856c9c99ba4317b9c4e653a0594960cf29a76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:23:06.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.548+0000 7f1fca1c6640 1 -- 172.21.15.60:0/3989617030 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fc402d120 con 0x7f1fc41047e0 2024-03-04T12:23:06.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.548+0000 7f1fc0ff9640 1 -- 172.21.15.60:0/3989617030 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9227 (secure 0 0 0) 0x7f1fb809aaa0 con 0x7f1fc41047e0 2024-03-04T12:23:06.554 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:06.554 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-04T12:23:06.555 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 12 2024-03-04T12:23:06.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.550+0000 7f1fca1c6640 1 -- 172.21.15.60:0/3989617030 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1fac0a1e00 msgr2=0x7f1fac0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:06.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.550+0000 7f1fca1c6640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1fac0a1e00 0x7f1fac0a42c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4104640 tx=0x7f1fb4007600 comp rx=0 tx=0).stop 2024-03-04T12:23:06.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.550+0000 7f1fca1c6640 1 -- 172.21.15.60:0/3989617030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc41047e0 msgr2=0x7f1fc4199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:06.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.550+0000 7f1fca1c6640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc41047e0 0x7f1fc4199c70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800d950 tx=0x7f1fb800de10 comp rx=0 tx=0).stop 2024-03-04T12:23:06.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.550+0000 7f1fca1c6640 1 -- 172.21.15.60:0/3989617030 shutdown_connections 2024-03-04T12:23:06.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.550+0000 7f1fca1c6640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1fac0a1e00 0x7f1fac0a42c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:06.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.550+0000 7f1fca1c6640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1fc41051a0 0x7f1fc41a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:06.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.550+0000 7f1fca1c6640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1fc41047e0 0x7f1fc4199c70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:06.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.550+0000 7f1fca1c6640 1 --2- 172.21.15.60:0/3989617030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fc41035f0 0x7f1fc4199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:06.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.550+0000 7f1fca1c6640 1 -- 172.21.15.60:0/3989617030 >> 172.21.15.60:0/3989617030 conn(0x7f1fc40feb40 msgr2=0x7f1fc4107a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:06.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.551+0000 7f1fca1c6640 1 -- 172.21.15.60:0/3989617030 shutdown_connections 2024-03-04T12:23:06.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:06.551+0000 7f1fca1c6640 1 -- 172.21.15.60:0/3989617030 wait complete. 2024-03-04T12:23:06.601 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:23:06 smithi063 podman[67395]: 2024-03-04 12:23:06.573652045 +0000 UTC m=+0.391894059 container init e613f71e49fc41ffb30cd24caca856c9c99ba4317b9c4e653a0594960cf29a76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-03-04T12:23:06.601 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:23:06 smithi063 podman[67395]: 2024-03-04 12:23:06.576438308 +0000 UTC m=+0.394680306 container start e613f71e49fc41ffb30cd24caca856c9c99ba4317b9c4e653a0594960cf29a76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-04T12:23:06.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:06 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3989617030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:06.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:06 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:06.928 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:06 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:07.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:06 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3989617030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:07.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:06 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:07.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:06 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:07.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:06 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3989617030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:07.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:06 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:07.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:06 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:07.101 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:23:06 smithi063 ceph-mds[67415]: -- 172.21.15.63:0/4047349032 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56033426fa40 con 0x560334242c00 2024-03-04T12:23:07.101 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:23:06 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b[67411]: starting mds.b at 2024-03-04T12:23:07.101 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:23:06 smithi063 bash[67395]: e613f71e49fc41ffb30cd24caca856c9c99ba4317b9c4e653a0594960cf29a76 2024-03-04T12:23:07.101 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:23:06 smithi063 systemd[1]: Started Ceph mds.b for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:23:08.091 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:07 smithi063 ceph-mon[36717]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:08.091 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:07 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] up:boot 2024-03-04T12:23:08.091 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:07 smithi063 ceph-mon[36717]: fsmap 12 up:standby 2024-03-04T12:23:08.091 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:07 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T12:23:08.091 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:07 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:08.091 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:07 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:08.091 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:07 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:08.091 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:07 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:23:08.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:07 smithi086 ceph-mon[36851]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:08.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:07 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] up:boot 2024-03-04T12:23:08.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:07 smithi086 ceph-mon[36851]: fsmap 12 up:standby 2024-03-04T12:23:08.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:07 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T12:23:08.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:07 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:08.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:07 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:08.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:07 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:08.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:07 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:23:08.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:07 smithi060 ceph-mon[32745]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:08.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:07 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] up:boot 2024-03-04T12:23:08.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:07 smithi060 ceph-mon[32745]: fsmap 12 up:standby 2024-03-04T12:23:08.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:07 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T12:23:08.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:07 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:08.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:07 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:08.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:07 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:08.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:07 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:23:08.569 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:23:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.674+0000 7f99e362b640 1 -- 172.21.15.60:0/2440190078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99dc1017f0 msgr2=0x7f99dc101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.674+0000 7f99e362b640 1 --2- 172.21.15.60:0/2440190078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99dc1017f0 0x7f99dc101c10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f99d0009900 tx=0x7f99d002bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.674+0000 7f99e362b640 1 -- 172.21.15.60:0/2440190078 shutdown_connections 2024-03-04T12:23:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.674+0000 7f99e362b640 1 --2- 172.21.15.60:0/2440190078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99dc100ee0 0x7f99dc108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.674+0000 7f99e362b640 1 --2- 172.21.15.60:0/2440190078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99dc100520 0x7f99dc1009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.674+0000 7f99e362b640 1 --2- 172.21.15.60:0/2440190078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99dc1017f0 0x7f99dc101c10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.674+0000 7f99e362b640 1 -- 172.21.15.60:0/2440190078 >> 172.21.15.60:0/2440190078 conn(0x7f99dc0faf50 msgr2=0x7f99dc0fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.674+0000 7f99e362b640 1 -- 172.21.15.60:0/2440190078 shutdown_connections 2024-03-04T12:23:08.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.674+0000 7f99e362b640 1 -- 172.21.15.60:0/2440190078 wait complete. 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e362b640 1 Processor -- start 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e362b640 1 -- start start 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e362b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99dc100520 0x7f99dc197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e362b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99dc100ee0 0x7f99dc197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e362b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99dc1017f0 0x7f99dc19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e362b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc10bec0 con 0x7f99dc100ee0 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e362b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc10bd40 con 0x7f99dc100520 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e362b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc10c040 con 0x7f99dc1017f0 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e0bc6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99dc100ee0 0x7f99dc197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e0bc6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99dc100ee0 0x7f99dc197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54682/0 (socket says 172.21.15.60:54682) 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e0bc6640 1 -- 172.21.15.60:0/1129230132 learned_addr learned my addr 172.21.15.60:0/1129230132 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e1bc8640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99dc1017f0 0x7f99dc19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:08.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e13c7640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99dc100520 0x7f99dc197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e0bc6640 1 -- 172.21.15.60:0/1129230132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99dc1017f0 msgr2=0x7f99dc19ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e0bc6640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99dc1017f0 0x7f99dc19ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e0bc6640 1 -- 172.21.15.60:0/1129230132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99dc100520 msgr2=0x7f99dc197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e0bc6640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99dc100520 0x7f99dc197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e0bc6640 1 -- 172.21.15.60:0/1129230132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99dc19f290 con 0x7f99dc100ee0 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e1bc8640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99dc1017f0 0x7f99dc19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99e0bc6640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99dc100ee0 0x7f99dc197bb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f99d800e9f0 tx=0x7f99d800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.675+0000 7f99ce7fc640 1 -- 172.21.15.60:0/1129230132 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d800cd80 con 0x7f99dc100ee0 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.676+0000 7f99e362b640 1 -- 172.21.15.60:0/1129230132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99dc19f580 con 0x7f99dc100ee0 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.676+0000 7f99ce7fc640 1 -- 172.21.15.60:0/1129230132 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99d8010910 con 0x7f99dc100ee0 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.676+0000 7f99ce7fc640 1 -- 172.21.15.60:0/1129230132 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d80188f0 con 0x7f99dc100ee0 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.676+0000 7f99e362b640 1 -- 172.21.15.60:0/1129230132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99dc19fb20 con 0x7f99dc100ee0 2024-03-04T12:23:08.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.677+0000 7f99e362b640 1 -- 172.21.15.60:0/1129230132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99dc19fd80 con 0x7f99dc100ee0 2024-03-04T12:23:08.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.678+0000 7f99ce7fc640 1 -- 172.21.15.60:0/1129230132 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f99d8010ac0 con 0x7f99dc100ee0 2024-03-04T12:23:08.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.679+0000 7f99ce7fc640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99c80a1d80 0x7f99c80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:08.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.679+0000 7f99e13c7640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99c80a1d80 0x7f99c80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:08.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.679+0000 7f99ce7fc640 1 -- 172.21.15.60:0/1129230132 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f99d80cdd50 con 0x7f99dc100ee0 2024-03-04T12:23:08.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.680+0000 7f99e13c7640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99c80a1d80 0x7f99c80a4240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f99d002efd0 tx=0x7f99d0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:08.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.684+0000 7f99ce7fc640 1 -- 172.21.15.60:0/1129230132 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f99d809af30 con 0x7f99dc100ee0 2024-03-04T12:23:08.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.908+0000 7f99e362b640 1 -- 172.21.15.60:0/1129230132 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99dc19fff0 con 0x7f99dc100ee0 2024-03-04T12:23:08.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.910+0000 7f99ce7fc640 1 -- 172.21.15.60:0/1129230132 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10019 (secure 0 0 0) 0x7f99d809efd0 con 0x7f99dc100ee0 2024-03-04T12:23:08.915 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:08.915 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24524,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-04T12:23:08.916 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 13 2024-03-04T12:23:08.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.912+0000 7f99e362b640 1 -- 172.21.15.60:0/1129230132 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99c80a1d80 msgr2=0x7f99c80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:08.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.912+0000 7f99e362b640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99c80a1d80 0x7f99c80a4240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f99d002efd0 tx=0x7f99d0000bc0 comp rx=0 tx=0).stop 2024-03-04T12:23:08.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.912+0000 7f99e362b640 1 -- 172.21.15.60:0/1129230132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99dc100ee0 msgr2=0x7f99dc197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:08.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.912+0000 7f99e362b640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99dc100ee0 0x7f99dc197bb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f99d800e9f0 tx=0x7f99d800eeb0 comp rx=0 tx=0).stop 2024-03-04T12:23:08.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.912+0000 7f99e362b640 1 -- 172.21.15.60:0/1129230132 shutdown_connections 2024-03-04T12:23:08.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.912+0000 7f99e362b640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99c80a1d80 0x7f99c80a4240 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:08.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.912+0000 7f99e362b640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99dc1017f0 0x7f99dc19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:08.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.912+0000 7f99e362b640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99dc100ee0 0x7f99dc197bb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:08.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.912+0000 7f99e362b640 1 --2- 172.21.15.60:0/1129230132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99dc100520 0x7f99dc197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:08.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.912+0000 7f99e362b640 1 -- 172.21.15.60:0/1129230132 >> 172.21.15.60:0/1129230132 conn(0x7f99dc0faf50 msgr2=0x7f99dc0fca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:08.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.912+0000 7f99e362b640 1 -- 172.21.15.60:0/1129230132 shutdown_connections 2024-03-04T12:23:08.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:08.912+0000 7f99e362b640 1 -- 172.21.15.60:0/1129230132 wait complete. 2024-03-04T12:23:08.929 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:23:09.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 -- 172.21.15.60:0/3150607542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb084100270 msgr2=0x7fb084100690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 --2- 172.21.15.60:0/3150607542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb084100270 0x7fb084100690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb080009900 tx=0x7fb08002bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 -- 172.21.15.60:0/3150607542 shutdown_connections 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 --2- 172.21.15.60:0/3150607542 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb084101e20 0x7fb0841086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 --2- 172.21.15.60:0/3150607542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb084101460 0x7fb0841018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 --2- 172.21.15.60:0/3150607542 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb084100270 0x7fb084100690 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 -- 172.21.15.60:0/3150607542 >> 172.21.15.60:0/3150607542 conn(0x7fb0840fb7c0 msgr2=0x7fb0840fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 -- 172.21.15.60:0/3150607542 shutdown_connections 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 -- 172.21.15.60:0/3150607542 wait complete. 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 Processor -- start 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 -- start start 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb084100270 0x7fb0841042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb084101460 0x7fb084102910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.027+0000 7fb08b87e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb084101e20 0x7fb084102e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb08b87e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08410bf00 con 0x7fb084101e20 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb08b87e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08410bd80 con 0x7fb084101460 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb08b87e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08410c080 con 0x7fb084100270 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb089e1b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb084101e20 0x7fb084102e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb089e1b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb084101e20 0x7fb084102e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54702/0 (socket says 172.21.15.60:54702) 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb08961a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb084100270 0x7fb0841042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb089e1b640 1 -- 172.21.15.60:0/3470753881 learned_addr learned my addr 172.21.15.60:0/3470753881 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb088e19640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb084101460 0x7fb084102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb089e1b640 1 -- 172.21.15.60:0/3470753881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb084100270 msgr2=0x7fb0841042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb089e1b640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb084100270 0x7fb0841042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb089e1b640 1 -- 172.21.15.60:0/3470753881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb084101460 msgr2=0x7fb084102910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb089e1b640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb084101460 0x7fb084102910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb089e1b640 1 -- 172.21.15.60:0/3470753881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0841036e0 con 0x7fb084101e20 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb088e19640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb084101460 0x7fb084102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:23:09.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb089e1b640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb084101e20 0x7fb084102e50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb07c00eb10 tx=0x7fb07c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:09.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb07a7fc640 1 -- 172.21.15.60:0/3470753881 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c004b20 con 0x7fb084101e20 2024-03-04T12:23:09.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb08b87e640 1 -- 172.21.15.60:0/3470753881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0841a17c0 con 0x7fb084101e20 2024-03-04T12:23:09.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb07a7fc640 1 -- 172.21.15.60:0/3470753881 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb07c004cc0 con 0x7fb084101e20 2024-03-04T12:23:09.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb07a7fc640 1 -- 172.21.15.60:0/3470753881 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb07c010760 con 0x7fb084101e20 2024-03-04T12:23:09.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.028+0000 7fb08b87e640 1 -- 172.21.15.60:0/3470753881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0841a1d10 con 0x7fb084101e20 2024-03-04T12:23:09.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.029+0000 7fb08b87e640 1 -- 172.21.15.60:0/3470753881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0841a1fd0 con 0x7fb084101e20 2024-03-04T12:23:09.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.031+0000 7fb07a7fc640 1 -- 172.21.15.60:0/3470753881 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fb07c005000 con 0x7fb084101e20 2024-03-04T12:23:09.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.032+0000 7fb07a7fc640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb0700a1d80 0x7fb0700a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.032+0000 7fb07a7fc640 1 -- 172.21.15.60:0/3470753881 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb07c0ce460 con 0x7fb084101e20 2024-03-04T12:23:09.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.032+0000 7fb08961a640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb0700a1d80 0x7fb0700a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.032+0000 7fb08961a640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb0700a1d80 0x7fb0700a4240 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb080000c00 tx=0x7fb080005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:09.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.035+0000 7fb07a7fc640 1 -- 172.21.15.60:0/3470753881 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb07c09bd30 con 0x7fb084101e20 2024-03-04T12:23:09.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.225+0000 7fb08b87e640 1 -- 172.21.15.60:0/3470753881 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0841a22c0 con 0x7fb084101e20 2024-03-04T12:23:09.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.226+0000 7fb07a7fc640 1 -- 172.21.15.60:0/3470753881 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18325 (secure 0 0 0) 0x7fb07c0b9b40 con 0x7fb084101e20 2024-03-04T12:23:09.232 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:09.232 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":81,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:22:20.316992+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:23:09.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.228+0000 7fb08b87e640 1 -- 172.21.15.60:0/3470753881 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb0700a1d80 msgr2=0x7fb0700a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.228+0000 7fb08b87e640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb0700a1d80 0x7fb0700a4240 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb080000c00 tx=0x7fb080005fd0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.228+0000 7fb08b87e640 1 -- 172.21.15.60:0/3470753881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb084101e20 msgr2=0x7fb084102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.228+0000 7fb08b87e640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb084101e20 0x7fb084102e50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb07c00eb10 tx=0x7fb07c00c2d0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.228+0000 7fb08b87e640 1 -- 172.21.15.60:0/3470753881 shutdown_connections 2024-03-04T12:23:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.228+0000 7fb08b87e640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb0700a1d80 0x7fb0700a4240 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.228+0000 7fb08b87e640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb084101e20 0x7fb084102e50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.228+0000 7fb08b87e640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb084101460 0x7fb084102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.228+0000 7fb08b87e640 1 --2- 172.21.15.60:0/3470753881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb084100270 0x7fb0841042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.228+0000 7fb08b87e640 1 -- 172.21.15.60:0/3470753881 >> 172.21.15.60:0/3470753881 conn(0x7fb0840fb7c0 msgr2=0x7fb0840694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.228+0000 7fb08b87e640 1 -- 172.21.15.60:0/3470753881 shutdown_connections 2024-03-04T12:23:09.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.228+0000 7fb08b87e640 1 -- 172.21.15.60:0/3470753881 wait complete. 2024-03-04T12:23:09.247 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T12:23:09.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:23:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.348+0000 7f84049cd640 1 -- 172.21.15.60:0/3572804847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8400101e00 msgr2=0x7f8400108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.348+0000 7f84049cd640 1 --2- 172.21.15.60:0/3572804847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8400101e00 0x7f8400108690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009ca0 tx=0x7f83f802c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.348+0000 7f84049cd640 1 -- 172.21.15.60:0/3572804847 shutdown_connections 2024-03-04T12:23:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.348+0000 7f84049cd640 1 --2- 172.21.15.60:0/3572804847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8400101e00 0x7f8400108690 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.348+0000 7f84049cd640 1 --2- 172.21.15.60:0/3572804847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400101440 0x7f84001018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.348+0000 7f84049cd640 1 --2- 172.21.15.60:0/3572804847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8400100250 0x7f8400100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.348+0000 7f84049cd640 1 -- 172.21.15.60:0/3572804847 >> 172.21.15.60:0/3572804847 conn(0x7f84000fb7e0 msgr2=0x7f84000fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.348+0000 7f84049cd640 1 -- 172.21.15.60:0/3572804847 shutdown_connections 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.348+0000 7f84049cd640 1 -- 172.21.15.60:0/3572804847 wait complete. 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f84049cd640 1 Processor -- start 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f84049cd640 1 -- start start 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f84049cd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8400100250 0x7f84001041e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f84049cd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400101440 0x7f8400102820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f84049cd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8400101e00 0x7f8400102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f84049cd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840010bee0 con 0x7f8400101e00 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f84049cd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840010bd60 con 0x7f8400100250 2024-03-04T12:23:09.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f84049cd640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840010c060 con 0x7f8400101440 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f83fed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8400101e00 0x7f8400102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f83fed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8400101e00 0x7f8400102d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54712/0 (socket says 172.21.15.60:54712) 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f83fed9d640 1 -- 172.21.15.60:0/1004434454 learned_addr learned my addr 172.21.15.60:0/1004434454 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f83fe59c640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8400100250 0x7f84001041e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f83f5d9b640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400101440 0x7f8400102820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f83fed9d640 1 -- 172.21.15.60:0/1004434454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400101440 msgr2=0x7f8400102820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f83fed9d640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400101440 0x7f8400102820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f83fed9d640 1 -- 172.21.15.60:0/1004434454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8400100250 msgr2=0x7f84001041e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.349+0000 7f83fed9d640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8400100250 0x7f84001041e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.350+0000 7f83fed9d640 1 -- 172.21.15.60:0/1004434454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84001035f0 con 0x7f8400101e00 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.350+0000 7f83f5d9b640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400101440 0x7f8400102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.350+0000 7f83fed9d640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8400101e00 0x7f8400102d60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f83f802c700 tx=0x7f83f8032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.350+0000 7f83f7fff640 1 -- 172.21.15.60:0/1004434454 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f8039040 con 0x7f8400101e00 2024-03-04T12:23:09.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.350+0000 7f83f7fff640 1 -- 172.21.15.60:0/1004434454 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f83f8044070 con 0x7f8400101e00 2024-03-04T12:23:09.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.350+0000 7f84049cd640 1 -- 172.21.15.60:0/1004434454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84001a16f0 con 0x7f8400101e00 2024-03-04T12:23:09.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.350+0000 7f83f7fff640 1 -- 172.21.15.60:0/1004434454 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f80042a0 con 0x7f8400101e00 2024-03-04T12:23:09.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.350+0000 7f84049cd640 1 -- 172.21.15.60:0/1004434454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84001a1c40 con 0x7f8400101e00 2024-03-04T12:23:09.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.351+0000 7f84049cd640 1 -- 172.21.15.60:0/1004434454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84001a1ea0 con 0x7f8400101e00 2024-03-04T12:23:09.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.353+0000 7f83f7fff640 1 -- 172.21.15.60:0/1004434454 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f83f8003970 con 0x7f8400101e00 2024-03-04T12:23:09.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.353+0000 7f83f7fff640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f83ec0a1d80 0x7f83ec0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.353+0000 7f83fe59c640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f83ec0a1d80 0x7f83ec0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.353+0000 7f83f7fff640 1 -- 172.21.15.60:0/1004434454 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f83f80f04a0 con 0x7f8400101e00 2024-03-04T12:23:09.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.354+0000 7f83fe59c640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f83ec0a1d80 0x7f83ec0a4240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f84001012a0 tx=0x7f83f000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:09.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.357+0000 7f83f7fff640 1 -- 172.21.15.60:0/1004434454 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f83f80bde00 con 0x7f8400101e00 2024-03-04T12:23:09.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.542+0000 7f84049cd640 1 -- 172.21.15.60:0/1004434454 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f84001a2070 con 0x7f8400101e00 2024-03-04T12:23:09.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.542+0000 7f83f7fff640 1 -- 172.21.15.60:0/1004434454 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f83f80bd700 con 0x7f8400101e00 2024-03-04T12:23:09.548 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:23:09.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.545+0000 7f84049cd640 1 -- 172.21.15.60:0/1004434454 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f83ec0a1d80 msgr2=0x7f83ec0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.545+0000 7f84049cd640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f83ec0a1d80 0x7f83ec0a4240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f84001012a0 tx=0x7f83f000b390 comp rx=0 tx=0).stop 2024-03-04T12:23:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.545+0000 7f84049cd640 1 -- 172.21.15.60:0/1004434454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8400101e00 msgr2=0x7f8400102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.545+0000 7f84049cd640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8400101e00 0x7f8400102d60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f83f802c700 tx=0x7f83f8032f20 comp rx=0 tx=0).stop 2024-03-04T12:23:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.545+0000 7f84049cd640 1 -- 172.21.15.60:0/1004434454 shutdown_connections 2024-03-04T12:23:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.545+0000 7f84049cd640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f83ec0a1d80 0x7f83ec0a4240 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.545+0000 7f84049cd640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8400101e00 0x7f8400102d60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.545+0000 7f84049cd640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8400101440 0x7f8400102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.545+0000 7f84049cd640 1 --2- 172.21.15.60:0/1004434454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8400100250 0x7f84001041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.545+0000 7f84049cd640 1 -- 172.21.15.60:0/1004434454 >> 172.21.15.60:0/1004434454 conn(0x7f84000fb7e0 msgr2=0x7f84000fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.545+0000 7f84049cd640 1 -- 172.21.15.60:0/1004434454 shutdown_connections 2024-03-04T12:23:09.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.545+0000 7f84049cd640 1 -- 172.21.15.60:0/1004434454 wait complete. 2024-03-04T12:23:09.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-04T12:23:09.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.661+0000 7f8da4545640 1 -- 172.21.15.60:0/625263915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d9c101450 msgr2=0x7f8d9c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.661+0000 7f8da4545640 1 --2- 172.21.15.60:0/625263915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d9c101450 0x7f8d9c1018d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c009900 tx=0x7f8d8c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.661+0000 7f8da4545640 1 -- 172.21.15.60:0/625263915 shutdown_connections 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.661+0000 7f8da4545640 1 --2- 172.21.15.60:0/625263915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d9c101e10 0x7f8d9c1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.661+0000 7f8da4545640 1 --2- 172.21.15.60:0/625263915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d9c101450 0x7f8d9c1018d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.661+0000 7f8da4545640 1 --2- 172.21.15.60:0/625263915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d9c100260 0x7f8d9c100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.661+0000 7f8da4545640 1 -- 172.21.15.60:0/625263915 >> 172.21.15.60:0/625263915 conn(0x7f8d9c0fb7d0 msgr2=0x7f8d9c0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.661+0000 7f8da4545640 1 -- 172.21.15.60:0/625263915 shutdown_connections 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.661+0000 7f8da4545640 1 -- 172.21.15.60:0/625263915 wait complete. 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.661+0000 7f8da4545640 1 Processor -- start 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.661+0000 7f8da4545640 1 -- start start 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da4545640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d9c100260 0x7f8d9c199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da4545640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d9c101450 0x7f8d9c199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da4545640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d9c101e10 0x7f8d9c1a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da4545640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c10bf40 con 0x7f8d9c101450 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da4545640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c10bdc0 con 0x7f8d9c101e10 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da4545640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c10c0c0 con 0x7f8d9c100260 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da1ae0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d9c101450 0x7f8d9c199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da1ae0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d9c101450 0x7f8d9c199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54732/0 (socket says 172.21.15.60:54732) 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da1ae0640 1 -- 172.21.15.60:0/2517873593 learned_addr learned my addr 172.21.15.60:0/2517873593 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da22e1640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d9c100260 0x7f8d9c199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da2ae2640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d9c101e10 0x7f8d9c1a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da22e1640 1 -- 172.21.15.60:0/2517873593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d9c101e10 msgr2=0x7f8d9c1a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da22e1640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d9c101e10 0x7f8d9c1a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da22e1640 1 -- 172.21.15.60:0/2517873593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d9c101450 msgr2=0x7f8d9c199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da22e1640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d9c101450 0x7f8d9c199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da22e1640 1 -- 172.21.15.60:0/2517873593 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c1a1440 con 0x7f8d9c100260 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da1ae0640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d9c101450 0x7f8d9c199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:23:09.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da2ae2640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d9c101e10 0x7f8d9c1a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:23:09.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.662+0000 7f8da22e1640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d9c100260 0x7f8d9c199820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800e9c0 tx=0x7f8d9800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:09.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.663+0000 7f8d937fe640 1 -- 172.21.15.60:0/2517873593 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9800ccc0 con 0x7f8d9c100260 2024-03-04T12:23:09.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.663+0000 7f8da4545640 1 -- 172.21.15.60:0/2517873593 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9c1a1730 con 0x7f8d9c100260 2024-03-04T12:23:09.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.663+0000 7f8d937fe640 1 -- 172.21.15.60:0/2517873593 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d980035f0 con 0x7f8d9c100260 2024-03-04T12:23:09.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.663+0000 7f8d937fe640 1 -- 172.21.15.60:0/2517873593 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9800ce60 con 0x7f8d9c100260 2024-03-04T12:23:09.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.663+0000 7f8da4545640 1 -- 172.21.15.60:0/2517873593 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9c1a1c50 con 0x7f8d9c100260 2024-03-04T12:23:09.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.664+0000 7f8da4545640 1 -- 172.21.15.60:0/2517873593 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9c1a1f10 con 0x7f8d9c100260 2024-03-04T12:23:09.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.665+0000 7f8d937fe640 1 -- 172.21.15.60:0/2517873593 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f8d980037a0 con 0x7f8d9c100260 2024-03-04T12:23:09.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.666+0000 7f8d937fe640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8d880a1d80 0x7f8d880a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.666+0000 7f8d937fe640 1 -- 172.21.15.60:0/2517873593 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8d980cdee0 con 0x7f8d9c100260 2024-03-04T12:23:09.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.666+0000 7f8da1ae0640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8d880a1d80 0x7f8d880a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.666+0000 7f8da1ae0640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8d880a1d80 0x7f8d880a4240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c02c8e0 tx=0x7f8d8c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:09.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.671+0000 7f8d937fe640 1 -- 172.21.15.60:0/2517873593 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8d9809b710 con 0x7f8d9c100260 2024-03-04T12:23:09.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:09.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1129230132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3470753881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:23:09.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:09 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1004434454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:23:09.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.855+0000 7f8da4545640 1 -- 172.21.15.60:0/2517873593 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8d9c1a20e0 con 0x7f8d9c100260 2024-03-04T12:23:09.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.855+0000 7f8d937fe640 1 -- 172.21.15.60:0/2517873593 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8d9809b010 con 0x7f8d9c100260 2024-03-04T12:23:09.861 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:09.861 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2024-03-04T12:23:09.861 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:23:09.861 INFO:teuthology.orchestra.run.smithi060.stdout: "poolnum": 1, 2024-03-04T12:23:09.861 INFO:teuthology.orchestra.run.smithi060.stdout: "poolname": ".mgr" 2024-03-04T12:23:09.861 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:23:09.862 INFO:teuthology.orchestra.run.smithi060.stdout:] 2024-03-04T12:23:09.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.859+0000 7f8da4545640 1 -- 172.21.15.60:0/2517873593 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8d880a1d80 msgr2=0x7f8d880a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.859+0000 7f8da4545640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8d880a1d80 0x7f8d880a4240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c02c8e0 tx=0x7f8d8c02df90 comp rx=0 tx=0).stop 2024-03-04T12:23:09.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.859+0000 7f8da4545640 1 -- 172.21.15.60:0/2517873593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d9c100260 msgr2=0x7f8d9c199820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.859+0000 7f8da4545640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d9c100260 0x7f8d9c199820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800e9c0 tx=0x7f8d9800ee80 comp rx=0 tx=0).stop 2024-03-04T12:23:09.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.859+0000 7f8da4545640 1 -- 172.21.15.60:0/2517873593 shutdown_connections 2024-03-04T12:23:09.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.859+0000 7f8da4545640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8d880a1d80 0x7f8d880a4240 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.859+0000 7f8da4545640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d9c101e10 0x7f8d9c1a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.859+0000 7f8da4545640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8d9c101450 0x7f8d9c199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.859+0000 7f8da4545640 1 --2- 172.21.15.60:0/2517873593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d9c100260 0x7f8d9c199820 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.859+0000 7f8da4545640 1 -- 172.21.15.60:0/2517873593 >> 172.21.15.60:0/2517873593 conn(0x7f8d9c0fb7d0 msgr2=0x7f8d9c0fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:09.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.859+0000 7f8da4545640 1 -- 172.21.15.60:0/2517873593 shutdown_connections 2024-03-04T12:23:09.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.859+0000 7f8da4545640 1 -- 172.21.15.60:0/2517873593 wait complete. 2024-03-04T12:23:09.876 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:09.876 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.876 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.876 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.876 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.876 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.876 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.876 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1129230132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:09.876 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.876 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3470753881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:09.877 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:23:09.877 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:23:09.877 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:09.877 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:23:09.877 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:09 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1004434454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:23:09.878 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-04T12:23:09.878 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-04T12:23:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.989+0000 7fd098c81640 1 -- 172.21.15.60:0/1620324758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0941025e0 msgr2=0x7fd094102a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.989+0000 7fd098c81640 1 --2- 172.21.15.60:0/1620324758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0941025e0 0x7fd094102a60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd084009930 tx=0x7fd08402bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.989+0000 7fd098c81640 1 -- 172.21.15.60:0/1620324758 shutdown_connections 2024-03-04T12:23:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.989+0000 7fd098c81640 1 --2- 172.21.15.60:0/1620324758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd094102fa0 0x7fd094109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.989+0000 7fd098c81640 1 --2- 172.21.15.60:0/1620324758 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0941025e0 0x7fd094102a60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.989+0000 7fd098c81640 1 --2- 172.21.15.60:0/1620324758 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0940ff610 0x7fd0940ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.989+0000 7fd098c81640 1 -- 172.21.15.60:0/1620324758 >> 172.21.15.60:0/1620324758 conn(0x7fd0940faf40 msgr2=0x7fd0940fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.989+0000 7fd098c81640 1 -- 172.21.15.60:0/1620324758 shutdown_connections 2024-03-04T12:23:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.989+0000 7fd098c81640 1 -- 172.21.15.60:0/1620324758 wait complete. 2024-03-04T12:23:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.990+0000 7fd098c81640 1 Processor -- start 2024-03-04T12:23:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.990+0000 7fd098c81640 1 -- start start 2024-03-04T12:23:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.990+0000 7fd098c81640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0940ff610 0x7fd094101f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.990+0000 7fd098c81640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0941025e0 0x7fd094100590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.990+0000 7fd098c81640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd094102fa0 0x7fd094100ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.990+0000 7fd098c81640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09410d000 con 0x7fd0941025e0 2024-03-04T12:23:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.990+0000 7fd098c81640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09410ce80 con 0x7fd0940ff610 2024-03-04T12:23:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.990+0000 7fd098c81640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09410d180 con 0x7fd094102fa0 2024-03-04T12:23:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.990+0000 7fd091d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0941025e0 0x7fd094100590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.990+0000 7fd091d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0941025e0 0x7fd094100590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54740/0 (socket says 172.21.15.60:54740) 2024-03-04T12:23:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.990+0000 7fd091d9b640 1 -- 172.21.15.60:0/2782166676 learned_addr learned my addr 172.21.15.60:0/2782166676 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.990+0000 7fd09259c640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0940ff610 0x7fd094101f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.991+0000 7fd092d9d640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd094102fa0 0x7fd094100ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.991+0000 7fd091d9b640 1 -- 172.21.15.60:0/2782166676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd094102fa0 msgr2=0x7fd094100ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.991+0000 7fd091d9b640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd094102fa0 0x7fd094100ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.991+0000 7fd091d9b640 1 -- 172.21.15.60:0/2782166676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0940ff610 msgr2=0x7fd094101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.991+0000 7fd091d9b640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0940ff610 0x7fd094101f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.991+0000 7fd091d9b640 1 -- 172.21.15.60:0/2782166676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd094101360 con 0x7fd0941025e0 2024-03-04T12:23:09.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.991+0000 7fd091d9b640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0941025e0 0x7fd094100590 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd084000c00 tx=0x7fd08402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.991+0000 7fd0837fe640 1 -- 172.21.15.60:0/2782166676 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084041070 con 0x7fd0941025e0 2024-03-04T12:23:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.991+0000 7fd098c81640 1 -- 172.21.15.60:0/2782166676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0941a1780 con 0x7fd0941025e0 2024-03-04T12:23:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.991+0000 7fd0837fe640 1 -- 172.21.15.60:0/2782166676 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd084004450 con 0x7fd0941025e0 2024-03-04T12:23:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.991+0000 7fd0837fe640 1 -- 172.21.15.60:0/2782166676 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084034440 con 0x7fd0941025e0 2024-03-04T12:23:10.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.991+0000 7fd098c81640 1 -- 172.21.15.60:0/2782166676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0941a1d20 con 0x7fd0941025e0 2024-03-04T12:23:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.992+0000 7fd098c81640 1 -- 172.21.15.60:0/2782166676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0941087e0 con 0x7fd0941025e0 2024-03-04T12:23:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.993+0000 7fd0837fe640 1 -- 172.21.15.60:0/2782166676 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fd084044070 con 0x7fd0941025e0 2024-03-04T12:23:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.994+0000 7fd0837fe640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd07c0a1d80 0x7fd07c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.994+0000 7fd0837fe640 1 -- 172.21.15.60:0/2782166676 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd0840ef430 con 0x7fd0941025e0 2024-03-04T12:23:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.994+0000 7fd09259c640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd07c0a1d80 0x7fd07c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:10.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:09.995+0000 7fd09259c640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd07c0a1d80 0x7fd07c0a4240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd094068400 tx=0x7fd088005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:10.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.000+0000 7fd0837fe640 1 -- 172.21.15.60:0/2782166676 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd0840bcd00 con 0x7fd0941025e0 2024-03-04T12:23:10.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:10.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:10.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:10.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:10.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:10.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:10.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:10.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1129230132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:10.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:10.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3470753881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:10.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:23:10.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:23:10.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:10.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:23:10.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:09 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1004434454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:23:10.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.191+0000 7fd098c81640 1 -- 172.21.15.60:0/2782166676 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd09402ceb0 con 0x7fd0941025e0 2024-03-04T12:23:10.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.875+0000 7fd0837fe640 1 -- 172.21.15.60:0/2782166676 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd0840bc600 con 0x7fd0941025e0 2024-03-04T12:23:10.881 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' created 2024-03-04T12:23:10.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.879+0000 7fd098c81640 1 -- 172.21.15.60:0/2782166676 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd07c0a1d80 msgr2=0x7fd07c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:10.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.879+0000 7fd098c81640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd07c0a1d80 0x7fd07c0a4240 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd094068400 tx=0x7fd088005f90 comp rx=0 tx=0).stop 2024-03-04T12:23:10.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.879+0000 7fd098c81640 1 -- 172.21.15.60:0/2782166676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0941025e0 msgr2=0x7fd094100590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:10.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.879+0000 7fd098c81640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0941025e0 0x7fd094100590 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd084000c00 tx=0x7fd08402c650 comp rx=0 tx=0).stop 2024-03-04T12:23:10.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.880+0000 7fd098c81640 1 -- 172.21.15.60:0/2782166676 shutdown_connections 2024-03-04T12:23:10.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.880+0000 7fd098c81640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd07c0a1d80 0x7fd07c0a4240 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:10.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.880+0000 7fd098c81640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd094102fa0 0x7fd094100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:10.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.880+0000 7fd098c81640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd0941025e0 0x7fd094100590 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:10.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.880+0000 7fd098c81640 1 --2- 172.21.15.60:0/2782166676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd0940ff610 0x7fd094101f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:10.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.880+0000 7fd098c81640 1 -- 172.21.15.60:0/2782166676 >> 172.21.15.60:0/2782166676 conn(0x7fd0940faf40 msgr2=0x7fd0941059b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:10.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.880+0000 7fd098c81640 1 -- 172.21.15.60:0/2782166676 shutdown_connections 2024-03-04T12:23:10.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:10.880+0000 7fd098c81640 1 -- 172.21.15.60:0/2782166676 wait complete. 2024-03-04T12:23:10.900 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:10 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2517873593' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-04T12:23:10.900 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:10 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2782166676' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-04T12:23:10.902 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-04T12:23:11.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.017+0000 7f8610c8e640 1 -- 172.21.15.60:0/1651599454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f860c101df0 msgr2=0x7f860c108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:11.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.017+0000 7f8610c8e640 1 --2- 172.21.15.60:0/1651599454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f860c101df0 0x7f860c108680 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8604009ca0 tx=0x7f860402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:11.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.018+0000 7f8610c8e640 1 -- 172.21.15.60:0/1651599454 shutdown_connections 2024-03-04T12:23:11.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.018+0000 7f8610c8e640 1 --2- 172.21.15.60:0/1651599454 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f860c101df0 0x7f860c108680 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:11.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.018+0000 7f8610c8e640 1 --2- 172.21.15.60:0/1651599454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c101430 0x7f860c1018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:11.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.018+0000 7f8610c8e640 1 --2- 172.21.15.60:0/1651599454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f860c100240 0x7f860c100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:11.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.018+0000 7f8610c8e640 1 -- 172.21.15.60:0/1651599454 >> 172.21.15.60:0/1651599454 conn(0x7f860c0fb770 msgr2=0x7f860c0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:11.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.018+0000 7f8610c8e640 1 -- 172.21.15.60:0/1651599454 shutdown_connections 2024-03-04T12:23:11.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.018+0000 7f8610c8e640 1 -- 172.21.15.60:0/1651599454 wait complete. 2024-03-04T12:23:11.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.018+0000 7f8610c8e640 1 Processor -- start 2024-03-04T12:23:11.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.018+0000 7f8610c8e640 1 -- start start 2024-03-04T12:23:11.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8610c8e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f860c100240 0x7f860c197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:11.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8610c8e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f860c101430 0x7f860c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8610c8e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c101df0 0x7f860c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8610c8e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c10bed0 con 0x7f860c101430 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8610c8e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c10bd50 con 0x7f860c100240 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8610c8e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c10c050 con 0x7f860c101df0 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8609d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f860c101430 0x7f860c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8609d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f860c101430 0x7f860c197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54748/0 (socket says 172.21.15.60:54748) 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8609d9b640 1 -- 172.21.15.60:0/2716894134 learned_addr learned my addr 172.21.15.60:0/2716894134 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f860a59c640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f860c100240 0x7f860c197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f860ad9d640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c101df0 0x7f860c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8609d9b640 1 -- 172.21.15.60:0/2716894134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c101df0 msgr2=0x7f860c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8609d9b640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c101df0 0x7f860c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8609d9b640 1 -- 172.21.15.60:0/2716894134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f860c100240 msgr2=0x7f860c197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8609d9b640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f860c100240 0x7f860c197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:11.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8609d9b640 1 -- 172.21.15.60:0/2716894134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c19f200 con 0x7f860c101430 2024-03-04T12:23:11.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f860ad9d640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c101df0 0x7f860c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:23:11.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f860a59c640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f860c100240 0x7f860c197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:23:11.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.019+0000 7f8609d9b640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f860c101430 0x7f860c197b70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f860000d770 tx=0x7f860000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:11.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.023+0000 7f85fb7fe640 1 -- 172.21.15.60:0/2716894134 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600017070 con 0x7f860c101430 2024-03-04T12:23:11.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.023+0000 7f8610c8e640 1 -- 172.21.15.60:0/2716894134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860c19f4f0 con 0x7f860c101430 2024-03-04T12:23:11.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.023+0000 7f85fb7fe640 1 -- 172.21.15.60:0/2716894134 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8600005500 con 0x7f860c101430 2024-03-04T12:23:11.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.023+0000 7f85fb7fe640 1 -- 172.21.15.60:0/2716894134 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600004e00 con 0x7f860c101430 2024-03-04T12:23:11.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.023+0000 7f8610c8e640 1 -- 172.21.15.60:0/2716894134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860c19fa40 con 0x7f860c101430 2024-03-04T12:23:11.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.025+0000 7f8610c8e640 1 -- 172.21.15.60:0/2716894134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860c19fca0 con 0x7f860c101430 2024-03-04T12:23:11.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.025+0000 7f85fb7fe640 1 -- 172.21.15.60:0/2716894134 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f8600004fa0 con 0x7f860c101430 2024-03-04T12:23:11.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.026+0000 7f85fb7fe640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f85f40a1d80 0x7f85f40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:11.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.026+0000 7f860a59c640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f85f40a1d80 0x7f85f40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:11.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.026+0000 7f85fb7fe640 1 -- 172.21.15.60:0/2716894134 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f86000cd390 con 0x7f860c101430 2024-03-04T12:23:11.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.027+0000 7f860a59c640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f85f40a1d80 0x7f85f40a4240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f85fc005fd0 tx=0x7f85fc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:11.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.031+0000 7f85fb7fe640 1 -- 172.21.15.60:0/2716894134 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f860009a4a0 con 0x7f860c101430 2024-03-04T12:23:11.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:10 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2517873593' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-04T12:23:11.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:10 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2782166676' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-04T12:23:11.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:10 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2517873593' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-04T12:23:11.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:10 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2782166676' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-04T12:23:11.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.226+0000 7f8610c8e640 1 -- 172.21.15.60:0/2716894134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f860c1a0360 con 0x7f860c101430 2024-03-04T12:23:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.877+0000 7f85fb7fe640 1 -- 172.21.15.60:0/2716894134 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) v1 ==== 171+0+0 (secure 0 0 0) 0x7f860009e540 con 0x7f860c101430 2024-03-04T12:23:11.883 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' created 2024-03-04T12:23:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.881+0000 7f8610c8e640 1 -- 172.21.15.60:0/2716894134 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f85f40a1d80 msgr2=0x7f85f40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.881+0000 7f8610c8e640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f85f40a1d80 0x7f85f40a4240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f85fc005fd0 tx=0x7f85fc00b410 comp rx=0 tx=0).stop 2024-03-04T12:23:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.881+0000 7f8610c8e640 1 -- 172.21.15.60:0/2716894134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f860c101430 msgr2=0x7f860c197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.881+0000 7f8610c8e640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f860c101430 0x7f860c197b70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f860000d770 tx=0x7f860000dc30 comp rx=0 tx=0).stop 2024-03-04T12:23:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.881+0000 7f8610c8e640 1 -- 172.21.15.60:0/2716894134 shutdown_connections 2024-03-04T12:23:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.881+0000 7f8610c8e640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f85f40a1d80 0x7f85f40a4240 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.881+0000 7f8610c8e640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860c101df0 0x7f860c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.881+0000 7f8610c8e640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f860c101430 0x7f860c197b70 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f860000d770 tx=0x7f860000dc30 comp rx=0 tx=0).stop 2024-03-04T12:23:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.881+0000 7f8610c8e640 1 --2- 172.21.15.60:0/2716894134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f860c100240 0x7f860c197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.881+0000 7f8610c8e640 1 -- 172.21.15.60:0/2716894134 >> 172.21.15.60:0/2716894134 conn(0x7f860c0fb770 msgr2=0x7f860c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.881+0000 7f8610c8e640 1 -- 172.21.15.60:0/2716894134 shutdown_connections 2024-03-04T12:23:11.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:11.881+0000 7f8610c8e640 1 -- 172.21.15.60:0/2716894134 wait complete. 2024-03-04T12:23:11.909 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-04T12:23:12.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.012+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3196453450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc14100080 msgr2=0x7fcc141004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:12.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.012+0000 7fcc1a5a2640 1 --2- 172.21.15.60:0/3196453450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc14100080 0x7fcc141004a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcc10009900 tx=0x7fcc1002bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:12.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.012+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3196453450 shutdown_connections 2024-03-04T12:23:12.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.012+0000 7fcc1a5a2640 1 --2- 172.21.15.60:0/3196453450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc14101c30 0x7fcc141084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.012+0000 7fcc1a5a2640 1 --2- 172.21.15.60:0/3196453450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc14101270 0x7fcc141016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.012+0000 7fcc1a5a2640 1 --2- 172.21.15.60:0/3196453450 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc14100080 0x7fcc141004a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.012+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3196453450 >> 172.21.15.60:0/3196453450 conn(0x7fcc140fb610 msgr2=0x7fcc140fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3196453450 shutdown_connections 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3196453450 wait complete. 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc1a5a2640 1 Processor -- start 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc1a5a2640 1 -- start start 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc1a5a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc14100080 0x7fcc14196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc1a5a2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc14101270 0x7fcc141969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc1a5a2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc14101c30 0x7fcc1419da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc1a5a2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1410bdc0 con 0x7fcc14100080 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc1a5a2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1410bc40 con 0x7fcc14101270 2024-03-04T12:23:12.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc1a5a2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1410bf40 con 0x7fcc14101c30 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc195a0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc14100080 0x7fcc14196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc195a0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc14100080 0x7fcc14196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54776/0 (socket says 172.21.15.60:54776) 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.013+0000 7fcc195a0640 1 -- 172.21.15.60:0/3889005868 learned_addr learned my addr 172.21.15.60:0/3889005868 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc19da1640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc14101c30 0x7fcc1419da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc18d9f640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc14101270 0x7fcc141969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc195a0640 1 -- 172.21.15.60:0/3889005868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc14101c30 msgr2=0x7fcc1419da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc195a0640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc14101c30 0x7fcc1419da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc195a0640 1 -- 172.21.15.60:0/3889005868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc14101270 msgr2=0x7fcc141969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc195a0640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc14101270 0x7fcc141969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc195a0640 1 -- 172.21.15.60:0/3889005868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1419e080 con 0x7fcc14100080 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc18d9f640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc14101270 0x7fcc141969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc195a0640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc14100080 0x7fcc14196460 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcc10000c00 tx=0x7fcc1002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:12.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc0a7fc640 1 -- 172.21.15.60:0/3889005868 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10043020 con 0x7fcc14100080 2024-03-04T12:23:12.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc0a7fc640 1 -- 172.21.15.60:0/3889005868 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc100054e0 con 0x7fcc14100080 2024-03-04T12:23:12.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3889005868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1419e310 con 0x7fcc14100080 2024-03-04T12:23:12.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc0a7fc640 1 -- 172.21.15.60:0/3889005868 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10003ce0 con 0x7fcc14100080 2024-03-04T12:23:12.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.014+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3889005868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1419e830 con 0x7fcc14100080 2024-03-04T12:23:12.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.015+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3889005868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc141073a0 con 0x7fcc14100080 2024-03-04T12:23:12.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.016+0000 7fcc0a7fc640 1 -- 172.21.15.60:0/3889005868 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fcc10046020 con 0x7fcc14100080 2024-03-04T12:23:12.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.017+0000 7fcc0a7fc640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcc000a1d80 0x7fcc000a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:12.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.017+0000 7fcc18d9f640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcc000a1d80 0x7fcc000a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:12.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.017+0000 7fcc0a7fc640 1 -- 172.21.15.60:0/3889005868 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fcc100f02b0 con 0x7fcc14100080 2024-03-04T12:23:12.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.017+0000 7fcc18d9f640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcc000a1d80 0x7fcc000a4240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcc04006fd0 tx=0x7fcc04008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:12.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.020+0000 7fcc0a7fc640 1 -- 172.21.15.60:0/3889005868 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcc1004d020 con 0x7fcc14100080 2024-03-04T12:23:12.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:12.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2782166676' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-04T12:23:12.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: osdmap e82: 12 total, 12 up, 12 in 2024-03-04T12:23:12.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:23:12.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:23:12.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:11 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2716894134' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-04T12:23:12.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.240+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3889005868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcc1402cf50 con 0x7fcc14100080 2024-03-04T12:23:12.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:12.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2782166676' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-04T12:23:12.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: osdmap e82: 12 total, 12 up, 12 in 2024-03-04T12:23:12.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:23:12.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:23:12.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:11 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2716894134' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-04T12:23:12.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:12.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2782166676' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-04T12:23:12.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: osdmap e82: 12 total, 12 up, 12 in 2024-03-04T12:23:12.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:23:12.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:23:12.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:23:12.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:11 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2716894134' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-04T12:23:12.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.901+0000 7fcc0a7fc640 1 -- 172.21.15.60:0/3889005868 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fcc100bd890 con 0x7fcc14100080 2024-03-04T12:23:12.907 INFO:teuthology.orchestra.run.smithi060.stderr:new fs with metadata pool 2 and data pool 3 2024-03-04T12:23:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.906+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3889005868 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcc000a1d80 msgr2=0x7fcc000a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.906+0000 7fcc1a5a2640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcc000a1d80 0x7fcc000a4240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fcc04006fd0 tx=0x7fcc04008040 comp rx=0 tx=0).stop 2024-03-04T12:23:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.907+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3889005868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc14100080 msgr2=0x7fcc14196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.907+0000 7fcc1a5a2640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc14100080 0x7fcc14196460 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fcc10000c00 tx=0x7fcc1002c6c0 comp rx=0 tx=0).stop 2024-03-04T12:23:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.907+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3889005868 shutdown_connections 2024-03-04T12:23:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.907+0000 7fcc1a5a2640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcc000a1d80 0x7fcc000a4240 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.907+0000 7fcc1a5a2640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc14101c30 0x7fcc1419da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.907+0000 7fcc1a5a2640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc14101270 0x7fcc141969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.907+0000 7fcc1a5a2640 1 --2- 172.21.15.60:0/3889005868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc14100080 0x7fcc14196460 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.907+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3889005868 >> 172.21.15.60:0/3889005868 conn(0x7fcc140fb610 msgr2=0x7fcc14104570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:12.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.907+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3889005868 shutdown_connections 2024-03-04T12:23:12.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:12.908+0000 7fcc1a5a2640 1 -- 172.21.15.60:0/3889005868 wait complete. 2024-03-04T12:23:12.937 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-04T12:23:12.937 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-04T12:23:13.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 -- 172.21.15.60:0/1660837857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec88107030 msgr2=0x7fec88109480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 --2- 172.21.15.60:0/1660837857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec88107030 0x7fec88109480 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fec80009ca0 tx=0x7fec8002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 -- 172.21.15.60:0/1660837857 shutdown_connections 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 --2- 172.21.15.60:0/1660837857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec88107030 0x7fec88109480 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 --2- 172.21.15.60:0/1660837857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec88104700 0x7fec88106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 --2- 172.21.15.60:0/1660837857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec88101dd0 0x7fec881041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 -- 172.21.15.60:0/1660837857 >> 172.21.15.60:0/1660837857 conn(0x7fec880fb7c0 msgr2=0x7fec880fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 -- 172.21.15.60:0/1660837857 shutdown_connections 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 -- 172.21.15.60:0/1660837857 wait complete. 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 Processor -- start 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 -- start start 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec88101dd0 0x7fec88197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec88104700 0x7fec88197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec88107030 0x7fec8819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:13.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8810cfe0 con 0x7fec88107030 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8810ce60 con 0x7fec88104700 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.045+0000 7fec904e8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8810d160 con 0x7fec88101dd0 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec8ea85640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec88107030 0x7fec8819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec8ea85640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec88107030 0x7fec8819ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54796/0 (socket says 172.21.15.60:54796) 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec8ea85640 1 -- 172.21.15.60:0/3834069117 learned_addr learned my addr 172.21.15.60:0/3834069117 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec8e284640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec88101dd0 0x7fec88197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec8ea85640 1 -- 172.21.15.60:0/3834069117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec88101dd0 msgr2=0x7fec88197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec8da83640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec88104700 0x7fec88197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec8ea85640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec88101dd0 0x7fec88197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec8ea85640 1 -- 172.21.15.60:0/3834069117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec88104700 msgr2=0x7fec88197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec8ea85640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec88104700 0x7fec88197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec8ea85640 1 -- 172.21.15.60:0/3834069117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8819f220 con 0x7fec88107030 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec8ea85640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec88107030 0x7fec8819ebb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fec80037040 tx=0x7fec80032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:13.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec7f7fe640 1 -- 172.21.15.60:0/3834069117 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80041020 con 0x7fec88107030 2024-03-04T12:23:13.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec7f7fe640 1 -- 172.21.15.60:0/3834069117 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec80034d90 con 0x7fec88107030 2024-03-04T12:23:13.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec904e8640 1 -- 172.21.15.60:0/3834069117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8819f4b0 con 0x7fec88107030 2024-03-04T12:23:13.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec7f7fe640 1 -- 172.21.15.60:0/3834069117 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80039040 con 0x7fec88107030 2024-03-04T12:23:13.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.046+0000 7fec904e8640 1 -- 172.21.15.60:0/3834069117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8819f9a0 con 0x7fec88107030 2024-03-04T12:23:13.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.047+0000 7fec904e8640 1 -- 172.21.15.60:0/3834069117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec880fd2b0 con 0x7fec88107030 2024-03-04T12:23:13.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.048+0000 7fec7f7fe640 1 -- 172.21.15.60:0/3834069117 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fec80044020 con 0x7fec88107030 2024-03-04T12:23:13.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.049+0000 7fec7f7fe640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fec740a1d80 0x7fec740a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:13.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.049+0000 7fec8e284640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fec740a1d80 0x7fec740a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:13.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.049+0000 7fec7f7fe640 1 -- 172.21.15.60:0/3834069117 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fec800efc90 con 0x7fec88107030 2024-03-04T12:23:13.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.049+0000 7fec8e284640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fec740a1d80 0x7fec740a4240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec840058c0 tx=0x7fec8400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:13.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.053+0000 7fec7f7fe640 1 -- 172.21.15.60:0/3834069117 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec8004b020 con 0x7fec88107030 2024-03-04T12:23:13.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:12 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2716894134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-04T12:23:13.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:12 smithi060 ceph-mon[32745]: osdmap e83: 12 total, 12 up, 12 in 2024-03-04T12:23:13.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:12 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3889005868' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-04T12:23:13.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:12 smithi060 ceph-mon[32745]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-04T12:23:13.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.245+0000 7fec904e8640 1 -- 172.21.15.60:0/3834069117 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fec88061f30 con 0x7fec88107030 2024-03-04T12:23:13.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:12 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2716894134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-04T12:23:13.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:12 smithi086 ceph-mon[36851]: osdmap e83: 12 total, 12 up, 12 in 2024-03-04T12:23:13.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:12 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3889005868' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-04T12:23:13.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:12 smithi086 ceph-mon[36851]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-04T12:23:13.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:12 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2716894134' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-04T12:23:13.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:12 smithi063 ceph-mon[36717]: osdmap e83: 12 total, 12 up, 12 in 2024-03-04T12:23:13.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:12 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3889005868' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-04T12:23:13.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:12 smithi063 ceph-mon[36717]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-04T12:23:13.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.891+0000 7fec7f7fe640 1 -- 172.21.15.60:0/3834069117 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) v1 ==== 157+0+0 (secure 0 0 0) 0x7fec800bd170 con 0x7fec88107030 2024-03-04T12:23:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.894+0000 7fec904e8640 1 -- 172.21.15.60:0/3834069117 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fec740a1d80 msgr2=0x7fec740a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:13.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.895+0000 7fec904e8640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fec740a1d80 0x7fec740a4240 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec840058c0 tx=0x7fec8400a2d0 comp rx=0 tx=0).stop 2024-03-04T12:23:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.895+0000 7fec904e8640 1 -- 172.21.15.60:0/3834069117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec88107030 msgr2=0x7fec8819ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.895+0000 7fec904e8640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec88107030 0x7fec8819ebb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fec80037040 tx=0x7fec80032f20 comp rx=0 tx=0).stop 2024-03-04T12:23:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.895+0000 7fec904e8640 1 -- 172.21.15.60:0/3834069117 shutdown_connections 2024-03-04T12:23:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.895+0000 7fec904e8640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fec740a1d80 0x7fec740a4240 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.895+0000 7fec904e8640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fec88107030 0x7fec8819ebb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.895+0000 7fec904e8640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec88104700 0x7fec88197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.895+0000 7fec904e8640 1 --2- 172.21.15.60:0/3834069117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec88101dd0 0x7fec88197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.895+0000 7fec904e8640 1 -- 172.21.15.60:0/3834069117 >> 172.21.15.60:0/3834069117 conn(0x7fec880fb7c0 msgr2=0x7fec88102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.895+0000 7fec904e8640 1 -- 172.21.15.60:0/3834069117 shutdown_connections 2024-03-04T12:23:13.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:13.895+0000 7fec904e8640 1 -- 172.21.15.60:0/3834069117 wait complete. 2024-03-04T12:23:13.915 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-04T12:23:14.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.026+0000 7fb403032640 1 -- 172.21.15.60:0/474666485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc104820 msgr2=0x7fb3fc106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:14.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.026+0000 7fb403032640 1 --2- 172.21.15.60:0/474666485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc104820 0x7fb3fc106c10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8009900 tx=0x7fb3e802bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:14.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.026+0000 7fb403032640 1 -- 172.21.15.60:0/474666485 shutdown_connections 2024-03-04T12:23:14.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.026+0000 7fb403032640 1 --2- 172.21.15.60:0/474666485 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3fc107150 0x7fb3fc109540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:14.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.026+0000 7fb403032640 1 --2- 172.21.15.60:0/474666485 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc104820 0x7fb3fc106c10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:14.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.026+0000 7fb403032640 1 --2- 172.21.15.60:0/474666485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc101ef0 0x7fb3fc1042e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:14.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.026+0000 7fb403032640 1 -- 172.21.15.60:0/474666485 >> 172.21.15.60:0/474666485 conn(0x7fb3fc0fb840 msgr2=0x7fb3fc0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:14.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.026+0000 7fb403032640 1 -- 172.21.15.60:0/474666485 shutdown_connections 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.026+0000 7fb403032640 1 -- 172.21.15.60:0/474666485 wait complete. 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb403032640 1 Processor -- start 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb403032640 1 -- start start 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb403032640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc101ef0 0x7fb3fc197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb403032640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3fc104820 0x7fb3fc197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb403032640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc107150 0x7fb3fc19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb403032640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc10d010 con 0x7fb3fc101ef0 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb403032640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc10ce90 con 0x7fb3fc104820 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb403032640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3fc10d190 con 0x7fb3fc107150 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb400dce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc101ef0 0x7fb3fc197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb400dce640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc101ef0 0x7fb3fc197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:54820/0 (socket says 172.21.15.60:54820) 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb400dce640 1 -- 172.21.15.60:0/1804015947 learned_addr learned my addr 172.21.15.60:0/1804015947 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb3fbfff640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3fc104820 0x7fb3fc197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb400dce640 1 -- 172.21.15.60:0/1804015947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc107150 msgr2=0x7fb3fc19ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb4015cf640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc107150 0x7fb3fc19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb400dce640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc107150 0x7fb3fc19ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:14.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb400dce640 1 -- 172.21.15.60:0/1804015947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3fc104820 msgr2=0x7fb3fc197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb400dce640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3fc104820 0x7fb3fc197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb400dce640 1 -- 172.21.15.60:0/1804015947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3fc19f290 con 0x7fb3fc101ef0 2024-03-04T12:23:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb4015cf640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc107150 0x7fb3fc19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:23:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb3fbfff640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3fc104820 0x7fb3fc197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.027+0000 7fb400dce640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc101ef0 0x7fb3fc197670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000d770 tx=0x7fb3f000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.028+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/1804015947 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0017070 con 0x7fb3fc101ef0 2024-03-04T12:23:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.028+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/1804015947 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3f0005500 con 0x7fb3fc101ef0 2024-03-04T12:23:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.028+0000 7fb403032640 1 -- 172.21.15.60:0/1804015947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3fc19f580 con 0x7fb3fc101ef0 2024-03-04T12:23:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.028+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/1804015947 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0004e00 con 0x7fb3fc101ef0 2024-03-04T12:23:14.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.028+0000 7fb403032640 1 -- 172.21.15.60:0/1804015947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3fc19faa0 con 0x7fb3fc101ef0 2024-03-04T12:23:14.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.029+0000 7fb403032640 1 -- 172.21.15.60:0/1804015947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc0fd330 con 0x7fb3fc101ef0 2024-03-04T12:23:14.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.030+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/1804015947 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fb3f0004fa0 con 0x7fb3fc101ef0 2024-03-04T12:23:14.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.030+0000 7fb3f9ffb640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb3ec0a1e00 0x7fb3ec0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:14.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.031+0000 7fb3fbfff640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb3ec0a1e00 0x7fb3ec0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:14.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.031+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/1804015947 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb3f00cd840 con 0x7fb3fc101ef0 2024-03-04T12:23:14.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.031+0000 7fb3fbfff640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb3ec0a1e00 0x7fb3ec0a42c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb3e802c7b0 tx=0x7fb3e8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:14.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.035+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/1804015947 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb3f009ac40 con 0x7fb3fc101ef0 2024-03-04T12:23:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:13 smithi060 ceph-mon[32745]: pgmap v287: 97 pgs: 96 unknown, 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:13 smithi060 ceph-mon[32745]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T12:23:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:13 smithi060 ceph-mon[32745]: osdmap e84: 12 total, 12 up, 12 in 2024-03-04T12:23:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:13 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3889005868' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-04T12:23:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:13 smithi060 ceph-mon[32745]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-04T12:23:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:13 smithi060 ceph-mon[32745]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-04T12:23:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:13 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3834069117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-04T12:23:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:13 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3834069117' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-04T12:23:14.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:13 smithi060 ceph-mon[32745]: osdmap e85: 12 total, 12 up, 12 in 2024-03-04T12:23:14.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:14.217+0000 7fb403032640 1 -- 172.21.15.60:0/1804015947 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb3fc02ceb0 con 0x7fb3fc101ef0 2024-03-04T12:23:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:13 smithi086 ceph-mon[36851]: pgmap v287: 97 pgs: 96 unknown, 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:13 smithi086 ceph-mon[36851]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T12:23:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:13 smithi086 ceph-mon[36851]: osdmap e84: 12 total, 12 up, 12 in 2024-03-04T12:23:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:13 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3889005868' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-04T12:23:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:13 smithi086 ceph-mon[36851]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-04T12:23:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:13 smithi086 ceph-mon[36851]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-04T12:23:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:13 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3834069117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-04T12:23:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:13 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3834069117' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-04T12:23:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:13 smithi086 ceph-mon[36851]: osdmap e85: 12 total, 12 up, 12 in 2024-03-04T12:23:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:13 smithi063 ceph-mon[36717]: pgmap v287: 97 pgs: 96 unknown, 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:23:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:13 smithi063 ceph-mon[36717]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T12:23:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:13 smithi063 ceph-mon[36717]: osdmap e84: 12 total, 12 up, 12 in 2024-03-04T12:23:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:13 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3889005868' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-04T12:23:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:13 smithi063 ceph-mon[36717]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-04T12:23:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:13 smithi063 ceph-mon[36717]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-04T12:23:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:13 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3834069117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-04T12:23:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:13 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3834069117' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-04T12:23:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:13 smithi063 ceph-mon[36717]: osdmap e85: 12 total, 12 up, 12 in 2024-03-04T12:23:15.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:14 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] up:active 2024-03-04T12:23:15.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:14 smithi060 ceph-mon[32745]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:15.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:14 smithi060 ceph-mon[32745]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:15.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:14 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1804015947' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-04T12:23:15.249 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:14 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] up:active 2024-03-04T12:23:15.249 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:14 smithi086 ceph-mon[36851]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:15.249 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:14 smithi086 ceph-mon[36851]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:15.249 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:14 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1804015947' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-04T12:23:15.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:14 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] up:active 2024-03-04T12:23:15.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:14 smithi063 ceph-mon[36717]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:15.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:14 smithi063 ceph-mon[36717]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:15.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:14 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1804015947' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-04T12:23:15.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.919+0000 7fb3f9ffb640 1 -- 172.21.15.60:0/1804015947 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) v1 ==== 228+0+0 (secure 0 0 0) 0x7fb3f009a540 con 0x7fb3fc101ef0 2024-03-04T12:23:15.925 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data_ec' created 2024-03-04T12:23:15.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.924+0000 7fb403032640 1 -- 172.21.15.60:0/1804015947 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb3ec0a1e00 msgr2=0x7fb3ec0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:15.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.924+0000 7fb403032640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb3ec0a1e00 0x7fb3ec0a42c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb3e802c7b0 tx=0x7fb3e8005fd0 comp rx=0 tx=0).stop 2024-03-04T12:23:15.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.924+0000 7fb403032640 1 -- 172.21.15.60:0/1804015947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc101ef0 msgr2=0x7fb3fc197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:15.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.924+0000 7fb403032640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc101ef0 0x7fb3fc197670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000d770 tx=0x7fb3f000dc30 comp rx=0 tx=0).stop 2024-03-04T12:23:15.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.924+0000 7fb403032640 1 -- 172.21.15.60:0/1804015947 shutdown_connections 2024-03-04T12:23:15.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.924+0000 7fb403032640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb3ec0a1e00 0x7fb3ec0a42c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:15.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.924+0000 7fb403032640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3fc107150 0x7fb3fc19ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:15.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.924+0000 7fb403032640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3fc104820 0x7fb3fc197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:15.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.924+0000 7fb403032640 1 --2- 172.21.15.60:0/1804015947 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb3fc101ef0 0x7fb3fc197670 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:15.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.924+0000 7fb403032640 1 -- 172.21.15.60:0/1804015947 >> 172.21.15.60:0/1804015947 conn(0x7fb3fc0fb840 msgr2=0x7fb3fc102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:15.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.924+0000 7fb403032640 1 -- 172.21.15.60:0/1804015947 shutdown_connections 2024-03-04T12:23:15.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:15.924+0000 7fb403032640 1 -- 172.21.15.60:0/1804015947 wait complete. 2024-03-04T12:23:15.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-04T12:23:16.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.055+0000 7f1b7928c640 1 -- 172.21.15.60:0/3221635862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b74101e00 msgr2=0x7f1b74108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:16.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.055+0000 7f1b7928c640 1 --2- 172.21.15.60:0/3221635862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b74101e00 0x7f1b74108690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c009ca0 tx=0x7f1b6c02c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:16.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.055+0000 7f1b7928c640 1 -- 172.21.15.60:0/3221635862 shutdown_connections 2024-03-04T12:23:16.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.055+0000 7f1b7928c640 1 --2- 172.21.15.60:0/3221635862 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b74101e00 0x7f1b74108690 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:16.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.055+0000 7f1b7928c640 1 --2- 172.21.15.60:0/3221635862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b74101440 0x7f1b741018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:16.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.055+0000 7f1b7928c640 1 --2- 172.21.15.60:0/3221635862 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b74100250 0x7f1b74100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:16.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.055+0000 7f1b7928c640 1 -- 172.21.15.60:0/3221635862 >> 172.21.15.60:0/3221635862 conn(0x7f1b740fb7c0 msgr2=0x7f1b740fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:16.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.055+0000 7f1b7928c640 1 -- 172.21.15.60:0/3221635862 shutdown_connections 2024-03-04T12:23:16.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.055+0000 7f1b7928c640 1 -- 172.21.15.60:0/3221635862 wait complete. 2024-03-04T12:23:16.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b7928c640 1 Processor -- start 2024-03-04T12:23:16.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b7928c640 1 -- start start 2024-03-04T12:23:16.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b7928c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b74100250 0x7f1b7406a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:16.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b7928c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b74101440 0x7f1b7419c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:16.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b7928c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b74101e00 0x7f1b7419e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b7928c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7410bee0 con 0x7f1b74100250 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b7928c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7410bd60 con 0x7f1b74101e00 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b7928c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7410c060 con 0x7f1b74101440 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b72ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b74100250 0x7f1b7406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b72ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b74100250 0x7f1b7406a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58306/0 (socket says 172.21.15.60:58306) 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b72ffd640 1 -- 172.21.15.60:0/407860969 learned_addr learned my addr 172.21.15.60:0/407860969 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b72ffd640 1 -- 172.21.15.60:0/407860969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b74101440 msgr2=0x7f1b7419c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b737fe640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b74101e00 0x7f1b7419e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b727fc640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b74101440 0x7f1b7419c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b72ffd640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b74101440 0x7f1b7419c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b72ffd640 1 -- 172.21.15.60:0/407860969 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b74101e00 msgr2=0x7f1b7419e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b72ffd640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b74101e00 0x7f1b7419e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.056+0000 7f1b72ffd640 1 -- 172.21.15.60:0/407860969 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7419f280 con 0x7f1b74100250 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.057+0000 7f1b72ffd640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b74100250 0x7f1b7406a540 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1b6400e9c0 tx=0x7f1b6400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:16.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.057+0000 7f1b5bfff640 1 -- 172.21.15.60:0/407860969 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6400cd80 con 0x7f1b74100250 2024-03-04T12:23:16.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.057+0000 7f1b5bfff640 1 -- 172.21.15.60:0/407860969 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b640035f0 con 0x7f1b74100250 2024-03-04T12:23:16.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.057+0000 7f1b7928c640 1 -- 172.21.15.60:0/407860969 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7419f510 con 0x7f1b74100250 2024-03-04T12:23:16.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.057+0000 7f1b5bfff640 1 -- 172.21.15.60:0/407860969 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64010690 con 0x7f1b74100250 2024-03-04T12:23:16.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.057+0000 7f1b7928c640 1 -- 172.21.15.60:0/407860969 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7419fa60 con 0x7f1b74100250 2024-03-04T12:23:16.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.058+0000 7f1b7928c640 1 -- 172.21.15.60:0/407860969 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b74107570 con 0x7f1b74100250 2024-03-04T12:23:16.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.059+0000 7f1b5bfff640 1 -- 172.21.15.60:0/407860969 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f1b640037a0 con 0x7f1b74100250 2024-03-04T12:23:16.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.059+0000 7f1b5bfff640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1b5c0a1d80 0x7f1b5c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:16.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.060+0000 7f1b727fc640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1b5c0a1d80 0x7f1b5c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:16.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.060+0000 7f1b5bfff640 1 -- 172.21.15.60:0/407860969 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8770+0+0 (secure 0 0 0) 0x7f1b640ce620 con 0x7f1b74100250 2024-03-04T12:23:16.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.060+0000 7f1b727fc640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1b5c0a1d80 0x7f1b5c0a4240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1b7406a2b0 tx=0x7f1b68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:16.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.063+0000 7f1b5bfff640 1 -- 172.21.15.60:0/407860969 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1b6409b860 con 0x7f1b74100250 2024-03-04T12:23:16.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:15 smithi060 ceph-mon[32745]: pgmap v290: 97 pgs: 23 creating+peering, 44 active+clean, 30 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-04T12:23:16.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:15 smithi060 ceph-mon[32745]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-04T12:23:16.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:15 smithi060 ceph-mon[32745]: Cluster is now healthy 2024-03-04T12:23:16.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:15 smithi060 ceph-mon[32745]: osdmap e86: 12 total, 12 up, 12 in 2024-03-04T12:23:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.248+0000 7f1b7928c640 1 -- 172.21.15.60:0/407860969 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f1b74062040 con 0x7f1b74100250 2024-03-04T12:23:16.263 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:15 smithi086 ceph-mon[36851]: pgmap v290: 97 pgs: 23 creating+peering, 44 active+clean, 30 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-04T12:23:16.263 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:15 smithi086 ceph-mon[36851]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-04T12:23:16.264 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:15 smithi086 ceph-mon[36851]: Cluster is now healthy 2024-03-04T12:23:16.264 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:15 smithi086 ceph-mon[36851]: osdmap e86: 12 total, 12 up, 12 in 2024-03-04T12:23:16.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:15 smithi063 ceph-mon[36717]: pgmap v290: 97 pgs: 23 creating+peering, 44 active+clean, 30 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-04T12:23:16.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:15 smithi063 ceph-mon[36717]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-04T12:23:16.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:15 smithi063 ceph-mon[36717]: Cluster is now healthy 2024-03-04T12:23:16.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:15 smithi063 ceph-mon[36717]: osdmap e86: 12 total, 12 up, 12 in 2024-03-04T12:23:16.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.924+0000 7f1b5bfff640 1 -- 172.21.15.60:0/407860969 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1b6409b160 con 0x7f1b74100250 2024-03-04T12:23:16.930 INFO:teuthology.orchestra.run.smithi060.stderr:set pool 4 allow_ec_overwrites to true 2024-03-04T12:23:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.928+0000 7f1b7928c640 1 -- 172.21.15.60:0/407860969 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1b5c0a1d80 msgr2=0x7f1b5c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.928+0000 7f1b7928c640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1b5c0a1d80 0x7f1b5c0a4240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1b7406a2b0 tx=0x7f1b68005f90 comp rx=0 tx=0).stop 2024-03-04T12:23:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.928+0000 7f1b7928c640 1 -- 172.21.15.60:0/407860969 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b74100250 msgr2=0x7f1b7406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.928+0000 7f1b7928c640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b74100250 0x7f1b7406a540 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1b6400e9c0 tx=0x7f1b6400ee80 comp rx=0 tx=0).stop 2024-03-04T12:23:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.928+0000 7f1b7928c640 1 -- 172.21.15.60:0/407860969 shutdown_connections 2024-03-04T12:23:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.928+0000 7f1b7928c640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1b5c0a1d80 0x7f1b5c0a4240 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.928+0000 7f1b7928c640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b74101e00 0x7f1b7419e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.928+0000 7f1b7928c640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b74101440 0x7f1b7419c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.928+0000 7f1b7928c640 1 --2- 172.21.15.60:0/407860969 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1b74100250 0x7f1b7406a540 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.928+0000 7f1b7928c640 1 -- 172.21.15.60:0/407860969 >> 172.21.15.60:0/407860969 conn(0x7f1b740fb7c0 msgr2=0x7f1b74104740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.928+0000 7f1b7928c640 1 -- 172.21.15.60:0/407860969 shutdown_connections 2024-03-04T12:23:16.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:16.928+0000 7f1b7928c640 1 -- 172.21.15.60:0/407860969 wait complete. 2024-03-04T12:23:16.956 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-04T12:23:17.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.059+0000 7fdcb651b640 1 -- 172.21.15.60:0/1061732209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcb00ff3d0 msgr2=0x7fdcb00ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:17.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.059+0000 7fdcb651b640 1 --2- 172.21.15.60:0/1061732209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcb00ff3d0 0x7fdcb00ff850 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009900 tx=0x7fdca402bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.059+0000 7fdcb651b640 1 -- 172.21.15.60:0/1061732209 shutdown_connections 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.059+0000 7fdcb651b640 1 --2- 172.21.15.60:0/1061732209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdcb00ffd90 0x7fdcb0100240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.059+0000 7fdcb651b640 1 --2- 172.21.15.60:0/1061732209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcb00ff3d0 0x7fdcb00ff850 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.059+0000 7fdcb651b640 1 --2- 172.21.15.60:0/1061732209 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcb01018e0 0x7fdcb0101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.059+0000 7fdcb651b640 1 -- 172.21.15.60:0/1061732209 >> 172.21.15.60:0/1061732209 conn(0x7fdcb00faf50 msgr2=0x7fdcb00fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.059+0000 7fdcb651b640 1 -- 172.21.15.60:0/1061732209 shutdown_connections 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.059+0000 7fdcb651b640 1 -- 172.21.15.60:0/1061732209 wait complete. 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.060+0000 7fdcb651b640 1 Processor -- start 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.060+0000 7fdcb651b640 1 -- start start 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.060+0000 7fdcb651b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcb00ff3d0 0x7fdcb0199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.060+0000 7fdcb651b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdcb00ffd90 0x7fdcb0199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.060+0000 7fdcb651b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcb01018e0 0x7fdcb01a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:17.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.060+0000 7fdcb651b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb010f1d0 con 0x7fdcb01018e0 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.060+0000 7fdcb651b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb010f050 con 0x7fdcb00ff3d0 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.060+0000 7fdcb651b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb010f350 con 0x7fdcb00ffd90 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.060+0000 7fdcb4ab8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcb01018e0 0x7fdcb01a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.060+0000 7fdcb4ab8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcb01018e0 0x7fdcb01a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58318/0 (socket says 172.21.15.60:58318) 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.060+0000 7fdcb4ab8640 1 -- 172.21.15.60:0/2910009386 learned_addr learned my addr 172.21.15.60:0/2910009386 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.061+0000 7fdcaf7fe640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdcb00ffd90 0x7fdcb0199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.061+0000 7fdcb4ab8640 1 -- 172.21.15.60:0/2910009386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdcb00ffd90 msgr2=0x7fdcb0199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.061+0000 7fdcb4ab8640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdcb00ffd90 0x7fdcb0199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.061+0000 7fdcb4ab8640 1 -- 172.21.15.60:0/2910009386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcb00ff3d0 msgr2=0x7fdcb0199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.061+0000 7fdcb4ab8640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcb00ff3d0 0x7fdcb0199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.061+0000 7fdcb4ab8640 1 -- 172.21.15.60:0/2910009386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb01a1470 con 0x7fdcb01018e0 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.061+0000 7fdcb4ab8640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcb01018e0 0x7fdcb01a0e00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdca800eb10 tx=0x7fdca800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.061+0000 7fdcad7fa640 1 -- 172.21.15.60:0/2910009386 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca8004b20 con 0x7fdcb01018e0 2024-03-04T12:23:17.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.061+0000 7fdcad7fa640 1 -- 172.21.15.60:0/2910009386 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdca8004cc0 con 0x7fdcb01018e0 2024-03-04T12:23:17.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.061+0000 7fdcb651b640 1 -- 172.21.15.60:0/2910009386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb01a1760 con 0x7fdcb01018e0 2024-03-04T12:23:17.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.061+0000 7fdcad7fa640 1 -- 172.21.15.60:0/2910009386 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca8010760 con 0x7fdcb01018e0 2024-03-04T12:23:17.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.061+0000 7fdcb651b640 1 -- 172.21.15.60:0/2910009386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb01a1c30 con 0x7fdcb01018e0 2024-03-04T12:23:17.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.062+0000 7fdcb651b640 1 -- 172.21.15.60:0/2910009386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb010b3a0 con 0x7fdcb01018e0 2024-03-04T12:23:17.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.063+0000 7fdcad7fa640 1 -- 172.21.15.60:0/2910009386 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fdca8005000 con 0x7fdcb01018e0 2024-03-04T12:23:17.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.063+0000 7fdcad7fa640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdc980a1d80 0x7fdc980a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:17.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.064+0000 7fdcad7fa640 1 -- 172.21.15.60:0/2910009386 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fdca80ce7f0 con 0x7fdcb01018e0 2024-03-04T12:23:17.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.064+0000 7fdcaffff640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdc980a1d80 0x7fdc980a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:17.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.064+0000 7fdcaffff640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdc980a1d80 0x7fdc980a4240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdca0005fd0 tx=0x7fdca0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:17.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.067+0000 7fdcad7fa640 1 -- 172.21.15.60:0/2910009386 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdca8052d20 con 0x7fdcb01018e0 2024-03-04T12:23:17.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:16 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1804015947' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-04T12:23:17.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:16 smithi063 ceph-mon[36717]: osdmap e87: 12 total, 12 up, 12 in 2024-03-04T12:23:17.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:16 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/407860969' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-04T12:23:17.101 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:23:16 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b[67411]: 2024-03-04T12:23:16.750+0000 7ff844c68700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-04T12:23:17.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.288+0000 7fdcb651b640 1 -- 172.21.15.60:0/2910009386 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fdcb002ceb0 con 0x7fdcb01018e0 2024-03-04T12:23:17.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:16 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1804015947' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-04T12:23:17.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:16 smithi086 ceph-mon[36851]: osdmap e87: 12 total, 12 up, 12 in 2024-03-04T12:23:17.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:16 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/407860969' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-04T12:23:17.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:16 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1804015947' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-04T12:23:17.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:16 smithi060 ceph-mon[32745]: osdmap e87: 12 total, 12 up, 12 in 2024-03-04T12:23:17.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:16 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/407860969' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-04T12:23:17.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.926+0000 7fdcad7fa640 1 -- 172.21.15.60:0/2910009386 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fdca809ba00 con 0x7fdcb01018e0 2024-03-04T12:23:17.932 INFO:teuthology.orchestra.run.smithi060.stderr:added data pool 4 to fsmap 2024-03-04T12:23:17.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.930+0000 7fdcb651b640 1 -- 172.21.15.60:0/2910009386 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdc980a1d80 msgr2=0x7fdc980a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.930+0000 7fdcb651b640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdc980a1d80 0x7fdc980a4240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdca0005fd0 tx=0x7fdca0005850 comp rx=0 tx=0).stop 2024-03-04T12:23:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.930+0000 7fdcb651b640 1 -- 172.21.15.60:0/2910009386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcb01018e0 msgr2=0x7fdcb01a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.930+0000 7fdcb651b640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcb01018e0 0x7fdcb01a0e00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdca800eb10 tx=0x7fdca800c2d0 comp rx=0 tx=0).stop 2024-03-04T12:23:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.930+0000 7fdcb651b640 1 -- 172.21.15.60:0/2910009386 shutdown_connections 2024-03-04T12:23:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.930+0000 7fdcb651b640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdc980a1d80 0x7fdc980a4240 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.930+0000 7fdcb651b640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdcb01018e0 0x7fdcb01a0e00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.930+0000 7fdcb651b640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdcb00ffd90 0x7fdcb0199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.930+0000 7fdcb651b640 1 --2- 172.21.15.60:0/2910009386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdcb00ff3d0 0x7fdcb0199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.930+0000 7fdcb651b640 1 -- 172.21.15.60:0/2910009386 >> 172.21.15.60:0/2910009386 conn(0x7fdcb00faf50 msgr2=0x7fdcb00fc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.930+0000 7fdcb651b640 1 -- 172.21.15.60:0/2910009386 shutdown_connections 2024-03-04T12:23:17.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:17.930+0000 7fdcb651b640 1 -- 172.21.15.60:0/2910009386 wait complete. 2024-03-04T12:23:17.950 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:23:18.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 -- 172.21.15.60:0/397269440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee28104710 msgr2=0x7fee28106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 --2- 172.21.15.60:0/397269440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee28104710 0x7fee28106b00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fee10009900 tx=0x7fee1002bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:18.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 -- 172.21.15.60:0/397269440 shutdown_connections 2024-03-04T12:23:18.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 --2- 172.21.15.60:0/397269440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee28107040 0x7fee28109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 --2- 172.21.15.60:0/397269440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee28104710 0x7fee28106b00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 --2- 172.21.15.60:0/397269440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee28101de0 0x7fee281041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 -- 172.21.15.60:0/397269440 >> 172.21.15.60:0/397269440 conn(0x7fee280fb7f0 msgr2=0x7fee280fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:18.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 -- 172.21.15.60:0/397269440 shutdown_connections 2024-03-04T12:23:18.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 -- 172.21.15.60:0/397269440 wait complete. 2024-03-04T12:23:18.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 Processor -- start 2024-03-04T12:23:18.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 -- start start 2024-03-04T12:23:18.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee28101de0 0x7fee28197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:18.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.053+0000 7fee2d759640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee28104710 0x7fee28197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:18.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee2d759640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee28107040 0x7fee2819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:18.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee2d759640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2810cff0 con 0x7fee28104710 2024-03-04T12:23:18.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee2d759640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2810ce70 con 0x7fee28107040 2024-03-04T12:23:18.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee2d759640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2810d170 con 0x7fee28101de0 2024-03-04T12:23:18.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee28104710 0x7fee28197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:18.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee28104710 0x7fee28197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58346/0 (socket says 172.21.15.60:58346) 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee1ffff640 1 -- 172.21.15.60:0/2773473437 learned_addr learned my addr 172.21.15.60:0/2773473437 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee26ffd640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee28101de0 0x7fee28197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee277fe640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee28107040 0x7fee2819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee1ffff640 1 -- 172.21.15.60:0/2773473437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee28101de0 msgr2=0x7fee28197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee1ffff640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee28101de0 0x7fee28197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee1ffff640 1 -- 172.21.15.60:0/2773473437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee28107040 msgr2=0x7fee2819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee1ffff640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee28107040 0x7fee2819ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee1ffff640 1 -- 172.21.15.60:0/2773473437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2819f230 con 0x7fee28104710 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee1ffff640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee28104710 0x7fee28197b50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fee10000c00 tx=0x7fee10005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee24ff9640 1 -- 172.21.15.60:0/2773473437 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee10041070 con 0x7fee28104710 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee2d759640 1 -- 172.21.15.60:0/2773473437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2819f4c0 con 0x7fee28104710 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee24ff9640 1 -- 172.21.15.60:0/2773473437 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee100044d0 con 0x7fee28104710 2024-03-04T12:23:18.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee24ff9640 1 -- 172.21.15.60:0/2773473437 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee10034440 con 0x7fee28104710 2024-03-04T12:23:18.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.054+0000 7fee2d759640 1 -- 172.21.15.60:0/2773473437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2819f9b0 con 0x7fee28104710 2024-03-04T12:23:18.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.055+0000 7fee2d759640 1 -- 172.21.15.60:0/2773473437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee280fd2e0 con 0x7fee28104710 2024-03-04T12:23:18.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.056+0000 7fee24ff9640 1 -- 172.21.15.60:0/2773473437 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fee10044070 con 0x7fee28104710 2024-03-04T12:23:18.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.057+0000 7fee24ff9640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fee140a1d80 0x7fee140a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:18.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.057+0000 7fee26ffd640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fee140a1d80 0x7fee140a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:18.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.057+0000 7fee24ff9640 1 -- 172.21.15.60:0/2773473437 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fee100efe80 con 0x7fee28104710 2024-03-04T12:23:18.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.057+0000 7fee26ffd640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fee140a1d80 0x7fee140a4240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fee18009b60 tx=0x7fee18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:18.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.061+0000 7fee24ff9640 1 -- 172.21.15.60:0/2773473437 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fee100bd070 con 0x7fee28104710 2024-03-04T12:23:18.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.281+0000 7fee2d759640 1 -- 172.21.15.60:0/2773473437 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee2802d070 con 0x7fee28104710 2024-03-04T12:23:18.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.281+0000 7fee24ff9640 1 -- 172.21.15.60:0/2773473437 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11262 (secure 0 0 0) 0x7fee100bc970 con 0x7fee28104710 2024-03-04T12:23:18.287 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:18.287 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:23:17.290093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T12:23:18.288 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2024-03-04T12:23:18.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.284+0000 7fee2d759640 1 -- 172.21.15.60:0/2773473437 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fee140a1d80 msgr2=0x7fee140a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.284+0000 7fee2d759640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fee140a1d80 0x7fee140a4240 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fee18009b60 tx=0x7fee18005f90 comp rx=0 tx=0).stop 2024-03-04T12:23:18.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.284+0000 7fee2d759640 1 -- 172.21.15.60:0/2773473437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee28104710 msgr2=0x7fee28197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.284+0000 7fee2d759640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee28104710 0x7fee28197b50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fee10000c00 tx=0x7fee10005d50 comp rx=0 tx=0).stop 2024-03-04T12:23:18.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.284+0000 7fee2d759640 1 -- 172.21.15.60:0/2773473437 shutdown_connections 2024-03-04T12:23:18.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.284+0000 7fee2d759640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fee140a1d80 0x7fee140a4240 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.284+0000 7fee2d759640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee28107040 0x7fee2819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.284+0000 7fee2d759640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee28104710 0x7fee28197b50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.284+0000 7fee2d759640 1 --2- 172.21.15.60:0/2773473437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee28101de0 0x7fee28197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.284+0000 7fee2d759640 1 -- 172.21.15.60:0/2773473437 >> 172.21.15.60:0/2773473437 conn(0x7fee280fb7f0 msgr2=0x7fee28102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:18.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.284+0000 7fee2d759640 1 -- 172.21.15.60:0/2773473437 shutdown_connections 2024-03-04T12:23:18.290 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.284+0000 7fee2d759640 1 -- 172.21.15.60:0/2773473437 wait complete. 2024-03-04T12:23:18.299 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:17 smithi060 ceph-mon[32745]: pgmap v293: 129 pgs: 32 unknown, 23 creating+peering, 74 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2024-03-04T12:23:18.299 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:17 smithi060 ceph-mon[32745]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T12:23:18.299 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:17 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/407860969' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-04T12:23:18.299 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:17 smithi060 ceph-mon[32745]: osdmap e88: 12 total, 12 up, 12 in 2024-03-04T12:23:18.299 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:17 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2910009386' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-04T12:23:18.301 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:23:18.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:17 smithi086 ceph-mon[36851]: pgmap v293: 129 pgs: 32 unknown, 23 creating+peering, 74 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2024-03-04T12:23:18.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:17 smithi086 ceph-mon[36851]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T12:23:18.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:17 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/407860969' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-04T12:23:18.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:17 smithi086 ceph-mon[36851]: osdmap e88: 12 total, 12 up, 12 in 2024-03-04T12:23:18.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:17 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2910009386' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-04T12:23:18.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:17 smithi063 ceph-mon[36717]: pgmap v293: 129 pgs: 32 unknown, 23 creating+peering, 74 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2024-03-04T12:23:18.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:17 smithi063 ceph-mon[36717]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T12:23:18.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:17 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/407860969' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-04T12:23:18.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:17 smithi063 ceph-mon[36717]: osdmap e88: 12 total, 12 up, 12 in 2024-03-04T12:23:18.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:17 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2910009386' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-04T12:23:18.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.397+0000 7f53ebcf1640 1 -- 172.21.15.60:0/1091367501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53e4100260 msgr2=0x7f53e4100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.397+0000 7f53ebcf1640 1 --2- 172.21.15.60:0/1091367501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53e4100260 0x7f53e4100680 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f53e0009900 tx=0x7f53e002bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.397+0000 7f53ebcf1640 1 -- 172.21.15.60:0/1091367501 shutdown_connections 2024-03-04T12:23:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.397+0000 7f53ebcf1640 1 --2- 172.21.15.60:0/1091367501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53e4101e10 0x7f53e41086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.397+0000 7f53ebcf1640 1 --2- 172.21.15.60:0/1091367501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53e4101450 0x7f53e41018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.397+0000 7f53ebcf1640 1 --2- 172.21.15.60:0/1091367501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53e4100260 0x7f53e4100680 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.397+0000 7f53ebcf1640 1 -- 172.21.15.60:0/1091367501 >> 172.21.15.60:0/1091367501 conn(0x7f53e40fb7d0 msgr2=0x7f53e40fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.397+0000 7f53ebcf1640 1 -- 172.21.15.60:0/1091367501 shutdown_connections 2024-03-04T12:23:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.397+0000 7f53ebcf1640 1 -- 172.21.15.60:0/1091367501 wait complete. 2024-03-04T12:23:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53ebcf1640 1 Processor -- start 2024-03-04T12:23:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53ebcf1640 1 -- start start 2024-03-04T12:23:18.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53ebcf1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53e4100260 0x7f53e4069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53ebcf1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53e4101450 0x7f53e419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53ebcf1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53e4101e10 0x7f53e419ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53ebcf1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e410bf10 con 0x7f53e4100260 2024-03-04T12:23:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53ebcf1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e410bd90 con 0x7f53e4101450 2024-03-04T12:23:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53ebcf1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e410c090 con 0x7f53e4101e10 2024-03-04T12:23:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53e9a8d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53e4100260 0x7f53e4069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53e9a8d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53e4100260 0x7f53e4069140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58372/0 (socket says 172.21.15.60:58372) 2024-03-04T12:23:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53e9a8d640 1 -- 172.21.15.60:0/2345855211 learned_addr learned my addr 172.21.15.60:0/2345855211 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53e928c640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53e4101450 0x7f53e419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53ea28e640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53e4101e10 0x7f53e419ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:18.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53e9a8d640 1 -- 172.21.15.60:0/2345855211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53e4101e10 msgr2=0x7f53e419ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53e9a8d640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53e4101e10 0x7f53e419ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53e9a8d640 1 -- 172.21.15.60:0/2345855211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53e4101450 msgr2=0x7f53e419c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53e9a8d640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53e4101450 0x7f53e419c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53e9a8d640 1 -- 172.21.15.60:0/2345855211 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e419f2c0 con 0x7f53e4100260 2024-03-04T12:23:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53ea28e640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53e4101e10 0x7f53e419ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:23:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53e9a8d640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53e4100260 0x7f53e4069140 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f53e0000c00 tx=0x7f53e002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53daffd640 1 -- 172.21.15.60:0/2345855211 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e0043020 con 0x7f53e4100260 2024-03-04T12:23:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53daffd640 1 -- 172.21.15.60:0/2345855211 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f53e00054e0 con 0x7f53e4100260 2024-03-04T12:23:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53ebcf1640 1 -- 172.21.15.60:0/2345855211 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e419f550 con 0x7f53e4100260 2024-03-04T12:23:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.398+0000 7f53daffd640 1 -- 172.21.15.60:0/2345855211 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e0003ce0 con 0x7f53e4100260 2024-03-04T12:23:18.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.399+0000 7f53ebcf1640 1 -- 172.21.15.60:0/2345855211 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e419fa70 con 0x7f53e4100260 2024-03-04T12:23:18.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.399+0000 7f53ebcf1640 1 -- 172.21.15.60:0/2345855211 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e4107580 con 0x7f53e4100260 2024-03-04T12:23:18.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.400+0000 7f53daffd640 1 -- 172.21.15.60:0/2345855211 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f53e0046020 con 0x7f53e4100260 2024-03-04T12:23:18.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.401+0000 7f53daffd640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f53d00a1d80 0x7f53d00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:18.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.401+0000 7f53e928c640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f53d00a1d80 0x7f53d00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:18.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.401+0000 7f53daffd640 1 -- 172.21.15.60:0/2345855211 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f53e00f0160 con 0x7f53e4100260 2024-03-04T12:23:18.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.401+0000 7f53e928c640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f53d00a1d80 0x7f53d00a4240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f53e40696e0 tx=0x7f53d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:18.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.405+0000 7f53daffd640 1 -- 172.21.15.60:0/2345855211 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f53e004d020 con 0x7f53e4100260 2024-03-04T12:23:18.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.586+0000 7f53ebcf1640 1 -- 172.21.15.60:0/2345855211 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53e4061f60 con 0x7f53e4100260 2024-03-04T12:23:18.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.588+0000 7f53daffd640 1 -- 172.21.15.60:0/2345855211 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23997 (secure 0 0 0) 0x7f53e00bd350 con 0x7f53e4100260 2024-03-04T12:23:18.594 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:18.594 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":89,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:23:17.289925+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:23:18.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.590+0000 7f53ebcf1640 1 -- 172.21.15.60:0/2345855211 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f53d00a1d80 msgr2=0x7f53d00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.590+0000 7f53ebcf1640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f53d00a1d80 0x7f53d00a4240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f53e40696e0 tx=0x7f53d4005f90 comp rx=0 tx=0).stop 2024-03-04T12:23:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.590+0000 7f53ebcf1640 1 -- 172.21.15.60:0/2345855211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53e4100260 msgr2=0x7f53e4069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.590+0000 7f53ebcf1640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53e4100260 0x7f53e4069140 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f53e0000c00 tx=0x7f53e002c6c0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.590+0000 7f53ebcf1640 1 -- 172.21.15.60:0/2345855211 shutdown_connections 2024-03-04T12:23:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.590+0000 7f53ebcf1640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f53d00a1d80 0x7f53d00a4240 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.590+0000 7f53ebcf1640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f53e4101e10 0x7f53e419ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.590+0000 7f53ebcf1640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53e4101450 0x7f53e419c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.590+0000 7f53ebcf1640 1 --2- 172.21.15.60:0/2345855211 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f53e4100260 0x7f53e4069140 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.590+0000 7f53ebcf1640 1 -- 172.21.15.60:0/2345855211 >> 172.21.15.60:0/2345855211 conn(0x7f53e40fb7d0 msgr2=0x7f53e4104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.590+0000 7f53ebcf1640 1 -- 172.21.15.60:0/2345855211 shutdown_connections 2024-03-04T12:23:18.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.590+0000 7f53ebcf1640 1 -- 172.21.15.60:0/2345855211 wait complete. 2024-03-04T12:23:18.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:23:18.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.704+0000 7fe601940640 1 -- 172.21.15.60:0/2462978270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5fc100260 msgr2=0x7fe5fc100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.704+0000 7fe601940640 1 --2- 172.21.15.60:0/2462978270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5fc100260 0x7fe5fc100680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009900 tx=0x7fe5f402bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:18.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.704+0000 7fe601940640 1 -- 172.21.15.60:0/2462978270 shutdown_connections 2024-03-04T12:23:18.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.704+0000 7fe601940640 1 --2- 172.21.15.60:0/2462978270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5fc101e10 0x7fe5fc1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.704+0000 7fe601940640 1 --2- 172.21.15.60:0/2462978270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5fc101450 0x7fe5fc1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.704+0000 7fe601940640 1 --2- 172.21.15.60:0/2462978270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5fc100260 0x7fe5fc100680 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.704+0000 7fe601940640 1 -- 172.21.15.60:0/2462978270 >> 172.21.15.60:0/2462978270 conn(0x7fe5fc0fb7d0 msgr2=0x7fe5fc0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:18.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.704+0000 7fe601940640 1 -- 172.21.15.60:0/2462978270 shutdown_connections 2024-03-04T12:23:18.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.704+0000 7fe601940640 1 -- 172.21.15.60:0/2462978270 wait complete. 2024-03-04T12:23:18.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe601940640 1 Processor -- start 2024-03-04T12:23:18.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe601940640 1 -- start start 2024-03-04T12:23:18.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe601940640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5fc100260 0x7fe5fc069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:18.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe601940640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5fc101450 0x7fe5fc19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:18.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe601940640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5fc101e10 0x7fe5fc19ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:18.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe601940640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc10bf30 con 0x7fe5fc101450 2024-03-04T12:23:18.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe601940640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc10bdb0 con 0x7fe5fc100260 2024-03-04T12:23:18.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe601940640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc10c0b0 con 0x7fe5fc101e10 2024-03-04T12:23:18.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe5fa7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5fc101450 0x7fe5fc19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:18.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe5fa7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5fc101450 0x7fe5fc19c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58398/0 (socket says 172.21.15.60:58398) 2024-03-04T12:23:18.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe5fa7fc640 1 -- 172.21.15.60:0/4286161700 learned_addr learned my addr 172.21.15.60:0/4286161700 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:18.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe5faffd640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5fc100260 0x7fe5fc069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:18.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe5fb7fe640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5fc101e10 0x7fe5fc19ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:18.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe5fa7fc640 1 -- 172.21.15.60:0/4286161700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5fc101e10 msgr2=0x7fe5fc19ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.705+0000 7fe5fa7fc640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5fc101e10 0x7fe5fc19ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.706+0000 7fe5fa7fc640 1 -- 172.21.15.60:0/4286161700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5fc100260 msgr2=0x7fe5fc069140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.706+0000 7fe5fa7fc640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5fc100260 0x7fe5fc069140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.706+0000 7fe5fa7fc640 1 -- 172.21.15.60:0/4286161700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc19f2e0 con 0x7fe5fc101450 2024-03-04T12:23:18.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.706+0000 7fe5fa7fc640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5fc101450 0x7fe5fc19c430 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe5e400d950 tx=0x7fe5e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:18.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.706+0000 7fe60093e640 1 -- 172.21.15.60:0/4286161700 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e4017030 con 0x7fe5fc101450 2024-03-04T12:23:18.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.706+0000 7fe60093e640 1 -- 172.21.15.60:0/4286161700 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5e4004020 con 0x7fe5fc101450 2024-03-04T12:23:18.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.706+0000 7fe601940640 1 -- 172.21.15.60:0/4286161700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5fc19f5d0 con 0x7fe5fc101450 2024-03-04T12:23:18.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.706+0000 7fe60093e640 1 -- 172.21.15.60:0/4286161700 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5e40047a0 con 0x7fe5fc101450 2024-03-04T12:23:18.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.706+0000 7fe601940640 1 -- 172.21.15.60:0/4286161700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5fc19fb20 con 0x7fe5fc101450 2024-03-04T12:23:18.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.707+0000 7fe601940640 1 -- 172.21.15.60:0/4286161700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5fc107580 con 0x7fe5fc101450 2024-03-04T12:23:18.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.708+0000 7fe60093e640 1 -- 172.21.15.60:0/4286161700 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fe5e40041d0 con 0x7fe5fc101450 2024-03-04T12:23:18.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.709+0000 7fe60093e640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe5e80a1d80 0x7fe5e80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:18.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.709+0000 7fe5faffd640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe5e80a1d80 0x7fe5e80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:18.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.709+0000 7fe60093e640 1 -- 172.21.15.60:0/4286161700 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe5e40cda70 con 0x7fe5fc101450 2024-03-04T12:23:18.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.709+0000 7fe5faffd640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe5e80a1d80 0x7fe5e80a4240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4000c00 tx=0x7fe5f4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:18.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.713+0000 7fe60093e640 1 -- 172.21.15.60:0/4286161700 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe5e409ac60 con 0x7fe5fc101450 2024-03-04T12:23:18.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.896+0000 7fe601940640 1 -- 172.21.15.60:0/4286161700 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5fc061f30 con 0x7fe5fc101450 2024-03-04T12:23:18.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.897+0000 7fe60093e640 1 -- 172.21.15.60:0/4286161700 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23997 (secure 0 0 0) 0x7fe5e409a560 con 0x7fe5fc101450 2024-03-04T12:23:18.903 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:18.903 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":89,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:23:17.289925+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:23:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.899+0000 7fe601940640 1 -- 172.21.15.60:0/4286161700 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe5e80a1d80 msgr2=0x7fe5e80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.899+0000 7fe601940640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe5e80a1d80 0x7fe5e80a4240 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4000c00 tx=0x7fe5f4005fd0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.899+0000 7fe601940640 1 -- 172.21.15.60:0/4286161700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5fc101450 msgr2=0x7fe5fc19c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:18.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.899+0000 7fe601940640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5fc101450 0x7fe5fc19c430 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe5e400d950 tx=0x7fe5e400de10 comp rx=0 tx=0).stop 2024-03-04T12:23:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.899+0000 7fe601940640 1 -- 172.21.15.60:0/4286161700 shutdown_connections 2024-03-04T12:23:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.899+0000 7fe601940640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe5e80a1d80 0x7fe5e80a4240 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.899+0000 7fe601940640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5fc101e10 0x7fe5fc19ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.899+0000 7fe601940640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe5fc101450 0x7fe5fc19c430 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.899+0000 7fe601940640 1 --2- 172.21.15.60:0/4286161700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5fc100260 0x7fe5fc069140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.899+0000 7fe601940640 1 -- 172.21.15.60:0/4286161700 >> 172.21.15.60:0/4286161700 conn(0x7fe5fc0fb7d0 msgr2=0x7fe5fc104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.899+0000 7fe601940640 1 -- 172.21.15.60:0/4286161700 shutdown_connections 2024-03-04T12:23:18.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:18.899+0000 7fe601940640 1 -- 172.21.15.60:0/4286161700 wait complete. 2024-03-04T12:23:18.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:23:19.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.029+0000 7f9b8529a640 1 -- 172.21.15.60:0/1992171177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b801047d0 msgr2=0x7f9b80106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.029+0000 7f9b8529a640 1 --2- 172.21.15.60:0/1992171177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b801047d0 0x7f9b80106bc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9b68009900 tx=0x7f9b6802bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:19.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.029+0000 7f9b8529a640 1 -- 172.21.15.60:0/1992171177 shutdown_connections 2024-03-04T12:23:19.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.029+0000 7f9b8529a640 1 --2- 172.21.15.60:0/1992171177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b80107100 0x7f9b801094f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.029+0000 7f9b8529a640 1 --2- 172.21.15.60:0/1992171177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b801047d0 0x7f9b80106bc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.029+0000 7f9b8529a640 1 --2- 172.21.15.60:0/1992171177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b80101ea0 0x7f9b80104290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.029+0000 7f9b8529a640 1 -- 172.21.15.60:0/1992171177 >> 172.21.15.60:0/1992171177 conn(0x7f9b800fb7f0 msgr2=0x7f9b800fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.029+0000 7f9b8529a640 1 -- 172.21.15.60:0/1992171177 shutdown_connections 2024-03-04T12:23:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.029+0000 7f9b8529a640 1 -- 172.21.15.60:0/1992171177 wait complete. 2024-03-04T12:23:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.029+0000 7f9b8529a640 1 Processor -- start 2024-03-04T12:23:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.029+0000 7f9b8529a640 1 -- start start 2024-03-04T12:23:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b8529a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b80101ea0 0x7f9b80101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b8529a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b801047d0 0x7f9b800ffe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b8529a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b80107100 0x7f9b80100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b8529a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8010d040 con 0x7f9b801047d0 2024-03-04T12:23:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b8529a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8010cec0 con 0x7f9b80101ea0 2024-03-04T12:23:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b8529a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8010d1c0 con 0x7f9b80107100 2024-03-04T12:23:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b7e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b801047d0 0x7f9b800ffe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b7e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b801047d0 0x7f9b800ffe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58422/0 (socket says 172.21.15.60:58422) 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b7e7fc640 1 -- 172.21.15.60:0/1617237787 learned_addr learned my addr 172.21.15.60:0/1617237787 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b7effd640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b80101ea0 0x7f9b80101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b7f7fe640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b80107100 0x7f9b80100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b7e7fc640 1 -- 172.21.15.60:0/1617237787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b80107100 msgr2=0x7f9b80100390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b7e7fc640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b80107100 0x7f9b80100390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b7e7fc640 1 -- 172.21.15.60:0/1617237787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b80101ea0 msgr2=0x7f9b80101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b7e7fc640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b80101ea0 0x7f9b80101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b7e7fc640 1 -- 172.21.15.60:0/1617237787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b80100c20 con 0x7f9b801047d0 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b7f7fe640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b80107100 0x7f9b80100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b7e7fc640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b801047d0 0x7f9b800ffe50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9b68000c00 tx=0x7f9b68005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b77fff640 1 -- 172.21.15.60:0/1617237787 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b68041070 con 0x7f9b801047d0 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b77fff640 1 -- 172.21.15.60:0/1617237787 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b680044d0 con 0x7f9b801047d0 2024-03-04T12:23:19.039 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b8529a640 1 -- 172.21.15.60:0/1617237787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b801a17b0 con 0x7f9b801047d0 2024-03-04T12:23:19.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b77fff640 1 -- 172.21.15.60:0/1617237787 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b68034440 con 0x7f9b801047d0 2024-03-04T12:23:19.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.030+0000 7f9b8529a640 1 -- 172.21.15.60:0/1617237787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b801a1d50 con 0x7f9b801047d0 2024-03-04T12:23:19.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.031+0000 7f9b8529a640 1 -- 172.21.15.60:0/1617237787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b800fd2e0 con 0x7f9b801047d0 2024-03-04T12:23:19.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.033+0000 7f9b77fff640 1 -- 172.21.15.60:0/1617237787 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f9b68044070 con 0x7f9b801047d0 2024-03-04T12:23:19.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.033+0000 7f9b77fff640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9b6c0a1e00 0x7f9b6c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.034+0000 7f9b7effd640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9b6c0a1e00 0x7f9b6c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.034+0000 7f9b77fff640 1 -- 172.21.15.60:0/1617237787 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9b680efd80 con 0x7f9b801047d0 2024-03-04T12:23:19.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.034+0000 7f9b7effd640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9b6c0a1e00 0x7f9b6c0a42c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9b78009bd0 tx=0x7f9b78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:19.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.037+0000 7f9b77fff640 1 -- 172.21.15.60:0/1617237787 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9b680bcf70 con 0x7f9b801047d0 2024-03-04T12:23:19.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:18 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2910009386' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-04T12:23:19.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:18 smithi060 ceph-mon[32745]: osdmap e89: 12 total, 12 up, 12 in 2024-03-04T12:23:19.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:18 smithi060 ceph-mon[32745]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:19.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:18 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2773473437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:19.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:18 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2345855211' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:19.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:18 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/4286161700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:19.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.260+0000 7f9b8529a640 1 -- 172.21.15.60:0/1617237787 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b8002ceb0 con 0x7f9b801047d0 2024-03-04T12:23:19.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.260+0000 7f9b77fff640 1 -- 172.21.15.60:0/1617237787 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11262 (secure 0 0 0) 0x7f9b680bc870 con 0x7f9b801047d0 2024-03-04T12:23:19.267 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:19.267 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14643,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24407,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24463,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:23:17.290093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24524},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-04T12:23:19.267 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 17 2024-03-04T12:23:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.262+0000 7f9b8529a640 1 -- 172.21.15.60:0/1617237787 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9b6c0a1e00 msgr2=0x7f9b6c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.263+0000 7f9b8529a640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9b6c0a1e00 0x7f9b6c0a42c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9b78009bd0 tx=0x7f9b78005f90 comp rx=0 tx=0).stop 2024-03-04T12:23:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.263+0000 7f9b8529a640 1 -- 172.21.15.60:0/1617237787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b801047d0 msgr2=0x7f9b800ffe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.263+0000 7f9b8529a640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b801047d0 0x7f9b800ffe50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9b68000c00 tx=0x7f9b68005d50 comp rx=0 tx=0).stop 2024-03-04T12:23:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.263+0000 7f9b8529a640 1 -- 172.21.15.60:0/1617237787 shutdown_connections 2024-03-04T12:23:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.263+0000 7f9b8529a640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9b6c0a1e00 0x7f9b6c0a42c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.263+0000 7f9b8529a640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b80107100 0x7f9b80100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.263+0000 7f9b8529a640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b801047d0 0x7f9b800ffe50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.263+0000 7f9b8529a640 1 --2- 172.21.15.60:0/1617237787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b80101ea0 0x7f9b80101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.263+0000 7f9b8529a640 1 -- 172.21.15.60:0/1617237787 >> 172.21.15.60:0/1617237787 conn(0x7f9b800fb7f0 msgr2=0x7f9b80102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.263+0000 7f9b8529a640 1 -- 172.21.15.60:0/1617237787 shutdown_connections 2024-03-04T12:23:19.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.263+0000 7f9b8529a640 1 -- 172.21.15.60:0/1617237787 wait complete. 2024-03-04T12:23:19.279 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:23:19.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:18 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2910009386' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-04T12:23:19.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:18 smithi086 ceph-mon[36851]: osdmap e89: 12 total, 12 up, 12 in 2024-03-04T12:23:19.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:18 smithi086 ceph-mon[36851]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:19.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:18 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2773473437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:19.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:18 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2345855211' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:19.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:18 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/4286161700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:19.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:18 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2910009386' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-04T12:23:19.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:18 smithi063 ceph-mon[36717]: osdmap e89: 12 total, 12 up, 12 in 2024-03-04T12:23:19.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:18 smithi063 ceph-mon[36717]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:19.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:18 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2773473437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:19.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:18 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2345855211' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:19.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:18 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/4286161700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:19.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.377+0000 7f4ff5080640 1 -- 172.21.15.60:0/1942597535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff0100260 msgr2=0x7f4ff0100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.377+0000 7f4ff5080640 1 --2- 172.21.15.60:0/1942597535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff0100260 0x7f4ff0100680 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0009900 tx=0x7f4fe002bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:19.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.377+0000 7f4ff5080640 1 -- 172.21.15.60:0/1942597535 shutdown_connections 2024-03-04T12:23:19.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.377+0000 7f4ff5080640 1 --2- 172.21.15.60:0/1942597535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ff0101e10 0x7f4ff01086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.377+0000 7f4ff5080640 1 --2- 172.21.15.60:0/1942597535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ff0101450 0x7f4ff01018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.377+0000 7f4ff5080640 1 --2- 172.21.15.60:0/1942597535 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff0100260 0x7f4ff0100680 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.377+0000 7f4ff5080640 1 -- 172.21.15.60:0/1942597535 >> 172.21.15.60:0/1942597535 conn(0x7f4ff00fb7f0 msgr2=0x7f4ff00fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:19.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.377+0000 7f4ff5080640 1 -- 172.21.15.60:0/1942597535 shutdown_connections 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.377+0000 7f4ff5080640 1 -- 172.21.15.60:0/1942597535 wait complete. 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.377+0000 7f4ff5080640 1 Processor -- start 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4ff5080640 1 -- start start 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4ff5080640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ff0100260 0x7f4ff006a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4ff5080640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ff0101450 0x7f4ff019c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4ff5080640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff0101e10 0x7f4ff019ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4ff5080640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff010bef0 con 0x7f4ff0101e10 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4ff5080640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff010bd70 con 0x7f4ff0100260 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4ff5080640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff010c070 con 0x7f4ff0101450 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fef59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff0101e10 0x7f4ff019ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fef59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff0101e10 0x7f4ff019ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58434/0 (socket says 172.21.15.60:58434) 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fef59e640 1 -- 172.21.15.60:0/1756086914 learned_addr learned my addr 172.21.15.60:0/1756086914 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4feed9d640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ff0100260 0x7f4ff006a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fe7fff640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ff0101450 0x7f4ff019c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fef59e640 1 -- 172.21.15.60:0/1756086914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ff0101450 msgr2=0x7f4ff019c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fef59e640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ff0101450 0x7f4ff019c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fef59e640 1 -- 172.21.15.60:0/1756086914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ff0100260 msgr2=0x7f4ff006a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fef59e640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ff0100260 0x7f4ff006a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fef59e640 1 -- 172.21.15.60:0/1756086914 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff019f320 con 0x7f4ff0101e10 2024-03-04T12:23:19.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4feed9d640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ff0100260 0x7f4ff006a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:23:19.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fef59e640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff0101e10 0x7f4ff019ea10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800eb10 tx=0x7f4fe800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:19.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fecd99640 1 -- 172.21.15.60:0/1756086914 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8004b20 con 0x7f4ff0101e10 2024-03-04T12:23:19.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fecd99640 1 -- 172.21.15.60:0/1756086914 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4fe8004cc0 con 0x7f4ff0101e10 2024-03-04T12:23:19.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4ff5080640 1 -- 172.21.15.60:0/1756086914 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff019f5b0 con 0x7f4ff0101e10 2024-03-04T12:23:19.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.378+0000 7f4fecd99640 1 -- 172.21.15.60:0/1756086914 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8010760 con 0x7f4ff0101e10 2024-03-04T12:23:19.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.379+0000 7f4ff5080640 1 -- 172.21.15.60:0/1756086914 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff019fab0 con 0x7f4ff0101e10 2024-03-04T12:23:19.387 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.380+0000 7f4ff5080640 1 -- 172.21.15.60:0/1756086914 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff0107600 con 0x7f4ff0101e10 2024-03-04T12:23:19.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.381+0000 7f4fecd99640 1 -- 172.21.15.60:0/1756086914 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f4fe8005020 con 0x7f4ff0101e10 2024-03-04T12:23:19.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.381+0000 7f4fecd99640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4fdc0a1d80 0x7f4fdc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.381+0000 7f4feed9d640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4fdc0a1d80 0x7f4fdc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.381+0000 7f4fecd99640 1 -- 172.21.15.60:0/1756086914 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4fe80ce5a0 con 0x7f4ff0101e10 2024-03-04T12:23:19.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.382+0000 7f4feed9d640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4fdc0a1d80 0x7f4fdc0a4240 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0000c00 tx=0x7f4fe0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:19.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.386+0000 7f4fecd99640 1 -- 172.21.15.60:0/1756086914 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4fe809b790 con 0x7f4ff0101e10 2024-03-04T12:23:19.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.571+0000 7f4ff5080640 1 -- 172.21.15.60:0/1756086914 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ff0061f30 con 0x7f4ff0101e10 2024-03-04T12:23:19.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.572+0000 7f4fecd99640 1 -- 172.21.15.60:0/1756086914 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23997 (secure 0 0 0) 0x7f4fe809b090 con 0x7f4ff0101e10 2024-03-04T12:23:19.578 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:19.578 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":89,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:23:17.289925+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:23:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.574+0000 7f4ff5080640 1 -- 172.21.15.60:0/1756086914 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4fdc0a1d80 msgr2=0x7f4fdc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.574+0000 7f4ff5080640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4fdc0a1d80 0x7f4fdc0a4240 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0000c00 tx=0x7f4fe0005fd0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.574+0000 7f4ff5080640 1 -- 172.21.15.60:0/1756086914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff0101e10 msgr2=0x7f4ff019ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.574+0000 7f4ff5080640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff0101e10 0x7f4ff019ea10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800eb10 tx=0x7f4fe800c2d0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.574+0000 7f4ff5080640 1 -- 172.21.15.60:0/1756086914 shutdown_connections 2024-03-04T12:23:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.574+0000 7f4ff5080640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4fdc0a1d80 0x7f4fdc0a4240 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.574+0000 7f4ff5080640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4ff0101e10 0x7f4ff019ea10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.580 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.574+0000 7f4ff5080640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ff0101450 0x7f4ff019c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.574+0000 7f4ff5080640 1 --2- 172.21.15.60:0/1756086914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4ff0100260 0x7f4ff006a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.574+0000 7f4ff5080640 1 -- 172.21.15.60:0/1756086914 >> 172.21.15.60:0/1756086914 conn(0x7f4ff00fb7f0 msgr2=0x7f4ff01047d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:19.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.574+0000 7f4ff5080640 1 -- 172.21.15.60:0/1756086914 shutdown_connections 2024-03-04T12:23:19.581 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.574+0000 7f4ff5080640 1 -- 172.21.15.60:0/1756086914 wait complete. 2024-03-04T12:23:19.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-04T12:23:12.241967+0000', 'modified': '2024-03-04T12:23:17.290093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24524}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.63:6839/2728229139', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6838', 'nonce': 2728229139}, {'type': 'v1', 'addr': '172.21.15.63:6839', 'nonce': 2728229139}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-03-04T12:23:19.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-04T12:23:19.592 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-04T12:23:19.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.692+0000 7fd9d0d37640 1 -- 172.21.15.60:0/100924292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9cc101dd0 msgr2=0x7fd9cc1041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.692+0000 7fd9d0d37640 1 --2- 172.21.15.60:0/100924292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9cc101dd0 0x7fd9cc1041c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4009900 tx=0x7fd9c402bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:19.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.692+0000 7fd9d0d37640 1 -- 172.21.15.60:0/100924292 shutdown_connections 2024-03-04T12:23:19.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.692+0000 7fd9d0d37640 1 --2- 172.21.15.60:0/100924292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9cc107030 0x7fd9cc109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.692+0000 7fd9d0d37640 1 --2- 172.21.15.60:0/100924292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9cc104700 0x7fd9cc106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.692+0000 7fd9d0d37640 1 --2- 172.21.15.60:0/100924292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9cc101dd0 0x7fd9cc1041c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.692+0000 7fd9d0d37640 1 -- 172.21.15.60:0/100924292 >> 172.21.15.60:0/100924292 conn(0x7fd9cc0fb7c0 msgr2=0x7fd9cc0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:19.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.692+0000 7fd9d0d37640 1 -- 172.21.15.60:0/100924292 shutdown_connections 2024-03-04T12:23:19.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.692+0000 7fd9d0d37640 1 -- 172.21.15.60:0/100924292 wait complete. 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9d0d37640 1 Processor -- start 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9d0d37640 1 -- start start 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9d0d37640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9cc101dd0 0x7fd9cc1976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9d0d37640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9cc104700 0x7fd9cc197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9d0d37640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9cc107030 0x7fd9cc19ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9d0d37640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc10d030 con 0x7fd9cc107030 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9d0d37640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc10ceb0 con 0x7fd9cc101dd0 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9d0d37640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc10d1b0 con 0x7fd9cc104700 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9cad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9cc107030 0x7fd9cc19ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9cad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9cc107030 0x7fd9cc19ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58458/0 (socket says 172.21.15.60:58458) 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9cad9d640 1 -- 172.21.15.60:0/3474440408 learned_addr learned my addr 172.21.15.60:0/3474440408 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9ca59c640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9cc101dd0 0x7fd9cc1976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.693+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9cc104700 0x7fd9cc197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.694+0000 7fd9cad9d640 1 -- 172.21.15.60:0/3474440408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9cc104700 msgr2=0x7fd9cc197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.694+0000 7fd9cad9d640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9cc104700 0x7fd9cc197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.694+0000 7fd9cad9d640 1 -- 172.21.15.60:0/3474440408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9cc101dd0 msgr2=0x7fd9cc1976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.694+0000 7fd9cad9d640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9cc101dd0 0x7fd9cc1976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.694+0000 7fd9cad9d640 1 -- 172.21.15.60:0/3474440408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9cc19f2f0 con 0x7fd9cc107030 2024-03-04T12:23:19.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.694+0000 7fd9cad9d640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9cc107030 0x7fd9cc19ec80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00eaa0 tx=0x7fd9bc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:19.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.694+0000 7fd9c37fe640 1 -- 172.21.15.60:0/3474440408 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc00cdb0 con 0x7fd9cc107030 2024-03-04T12:23:19.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.694+0000 7fd9c37fe640 1 -- 172.21.15.60:0/3474440408 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9bc005020 con 0x7fd9cc107030 2024-03-04T12:23:19.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.694+0000 7fd9d0d37640 1 -- 172.21.15.60:0/3474440408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9cc19f5e0 con 0x7fd9cc107030 2024-03-04T12:23:19.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.694+0000 7fd9c37fe640 1 -- 172.21.15.60:0/3474440408 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc0116b0 con 0x7fd9cc107030 2024-03-04T12:23:19.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.694+0000 7fd9d0d37640 1 -- 172.21.15.60:0/3474440408 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9cc19fae0 con 0x7fd9cc107030 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.696+0000 7fd9c37fe640 1 -- 172.21.15.60:0/3474440408 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fd9bc005560 con 0x7fd9cc107030 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.696+0000 7fd9d39eb740 1 Processor -- start 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9d39eb740 1 -- start start 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9d39eb740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 0x55aae73bf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9d39eb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 0x55aae73c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9d39eb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73c4d60 0x55aae73c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9d39eb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aae73207c0 con 0x55aae73bfbf0 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9d39eb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aae72c9b50 con 0x55aae73c4d60 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9d39eb740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aae731c860 con 0x55aae73bf200 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9ca59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 0x55aae73c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9c37fe640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9b80a1d80 0x7fd9b80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9ca59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 0x55aae73c46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58472/0 (socket says 172.21.15.60:58472) 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9ca59c640 1 -- 172.21.15.60:0/3251924440 learned_addr learned my addr 172.21.15.60:0/3251924440 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:19.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/3251924440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 0x55aae73bf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/3251924440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73c4d60 0x55aae73c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9cad9d640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9b80a1d80 0x7fd9b80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9ca59c640 1 -- 172.21.15.60:0/3251924440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 msgr2=0x55aae73bf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9ca59c640 1 --2- 172.21.15.60:0/3251924440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 0x55aae73bf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9ca59c640 1 -- 172.21.15.60:0/3251924440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73c4d60 msgr2=0x55aae73c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9ca59c640 1 --2- 172.21.15.60:0/3251924440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73c4d60 0x55aae73c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9c37fe640 1 -- 172.21.15.60:0/3474440408 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd9bc0ce270 con 0x7fd9cc107030 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.697+0000 7fd9ca59c640 1 -- 172.21.15.60:0/3251924440 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aae72dd9a0 con 0x55aae73bfbf0 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.698+0000 7fd9ca59c640 1 --2- 172.21.15.60:0/3251924440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 0x55aae73c46f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4005ab0 tx=0x7fd9c4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1966b95e688d4c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.698+0000 7fd9cad9d640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9b80a1d80 0x7fd9b80a4240 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc09aa60 tx=0x7fd9bc06de20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.698+0000 7fd9c0ff9640 1 -- 172.21.15.60:0/3251924440 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4045070 con 0x55aae73bfbf0 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.698+0000 7fd9c0ff9640 1 -- 172.21.15.60:0/3251924440 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9c4004040 con 0x55aae73bfbf0 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.698+0000 7fd9c0ff9640 1 -- 172.21.15.60:0/3251924440 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4004eb0 con 0x55aae73bfbf0 2024-03-04T12:23:19.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.698+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3251924440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 msgr2=0x55aae73c46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.698+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/3251924440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 0x55aae73c46f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4005ab0 tx=0x7fd9c4005dd0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.698+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3251924440 shutdown_connections 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.698+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/3251924440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73c4d60 0x55aae73c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.698+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/3251924440 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 0x55aae73c46f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.698+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/3251924440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 0x55aae73bf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.698+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3251924440 >> 172.21.15.60:0/3251924440 conn(0x55aae73b6360 msgr2=0x55aae73b6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3251924440 shutdown_connections 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3251924440 wait complete. 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9d39eb740 1 -- start start 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9d39eb740 1 Processor -- start 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9d39eb740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 0x55aae73c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9d39eb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 0x55aae73c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9d39eb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73c4d60 0x55aae73d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9d39eb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aae73c7cb0 con 0x55aae73bfbf0 2024-03-04T12:23:19.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9d39eb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aae72dcf90 con 0x55aae73c4d60 2024-03-04T12:23:19.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9d39eb740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aae73c7e30 con 0x55aae73bf200 2024-03-04T12:23:19.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9d39eb740 10 client.0 _ll_register_callbacks cb 0x55aae72fed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:23:19.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9ca59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 0x55aae73c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9ca59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 0x55aae73c9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58488/0 (socket says 172.21.15.60:58488) 2024-03-04T12:23:19.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9ca59c640 1 -- 172.21.15.60:0/3616469359 learned_addr learned my addr 172.21.15.60:0/3616469359 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:19.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 0x55aae73c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.699+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73c4d60 0x55aae73d0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9ca59c640 1 -- 172.21.15.60:0/3616469359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 msgr2=0x55aae73c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:23:19.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9ca59c640 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 0x55aae73c96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9ca59c640 1 -- 172.21.15.60:0/3616469359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73c4d60 msgr2=0x55aae73d0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:23:19.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9ca59c640 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73c4d60 0x55aae73d0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9ca59c640 1 -- 172.21.15.60:0/3616469359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aae70352d0 con 0x55aae73bfbf0 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9ca59c640 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 0x55aae73c9c00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4036040 tx=0x7fd9c4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e58cfd3781d66f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9b1ffb640 1 -- 172.21.15.60:0/3616469359 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c404e070 con 0x55aae73bfbf0 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3616469359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9c4004040 con 0x55aae73bfbf0 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9b1ffb640 1 -- 172.21.15.60:0/3616469359 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd9c4004eb0 con 0x55aae73bfbf0 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9b1ffb640 1 -- 172.21.15.60:0/3616469359 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9c405a050 con 0x55aae73bfbf0 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9d39eb740 10 client.14769 fetch_fsmap learned FSMap version 17 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3616469359 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55aae70ff6b0 con 0x55aae73bfbf0 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9b1ffb640 1 -- 172.21.15.60:0/3616469359 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9c4004040 con 0x55aae73bfbf0 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9b1ffb640 1 -- 172.21.15.60:0/3616469359 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 17) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd9c4003b40 con 0x55aae73bfbf0 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9d39eb740 10 client.14769 fetch_fsmap finished waiting for FSMap version 17 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9d39eb740 10 client.14769 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9d39eb740 20 client.14769 populate_metadata read hostname 'smithi060' 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.700+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55aae73d5cb0 0x55aae73da130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.701+0000 7fd9d39eb740 4 client.14769 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:23:19.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.701+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3616469359 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aae732dd00 con 0x55aae73d5cb0 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.701+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55aae73d5cb0 0x55aae73da130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.701+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55aae73d5cb0 0x55aae73da130 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3befa97af72bbdb server_cookie=92731e863de80825 in_seq=0 out_seq=0 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.701+0000 7fd9b1ffb640 10 client.14769 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9b1ffb640 1 -- 172.21.15.60:0/3616469359 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55aae732dd00 con 0x55aae73d5cb0 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9b1ffb640 10 client.14769 handle_command_reply: tid=0 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9d39eb740 1 client.14769 shutdown 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3616469359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 msgr2=0x55aae73c9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 0x55aae73c9c00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4036040 tx=0x7fd9c4004270 comp rx=0 tx=0).stop 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3616469359 shutdown_connections 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55aae73d5cb0 0x55aae73da130 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73c4d60 0x55aae73d0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9b1ffb640 0 client.14769 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bfbf0 0x55aae73c9c00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/3616469359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 0x55aae73c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3616469359 >> 172.21.15.60:0/3616469359 conn(0x55aae73b6360 msgr2=0x55aae73b5640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:19.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3616469359 shutdown_connections 2024-03-04T12:23:19.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9d39eb740 1 -- 172.21.15.60:0/3616469359 wait complete. 2024-03-04T12:23:19.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.702+0000 7fd9d39eb740 20 client.14769 trim_cache size 0 max 16384 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9d39eb740 1 Processor -- start 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9d39eb740 1 -- start start 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9d39eb740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 0x55aae73c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9d39eb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73bfbf0 0x55aae73d0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9d39eb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73c4d60 0x55aae73de150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9d39eb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aae73d54d0 con 0x55aae73c4d60 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9d39eb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aae73d5b30 con 0x55aae73bfbf0 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9d39eb740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aae73d6190 con 0x55aae73bf200 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9c9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73c4d60 0x55aae73de150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9c9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73c4d60 0x55aae73de150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58498/0 (socket says 172.21.15.60:58498) 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9c9d9b640 1 -- 172.21.15.60:0/414977593 learned_addr learned my addr 172.21.15.60:0/414977593 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9ca59c640 1 --2- 172.21.15.60:0/414977593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73bfbf0 0x55aae73d0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9c9d9b640 1 -- 172.21.15.60:0/414977593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 msgr2=0x55aae73c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/414977593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 0x55aae73c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9c9d9b640 1 -- 172.21.15.60:0/414977593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73bfbf0 msgr2=0x55aae73d0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/414977593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73bfbf0 0x55aae73d0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.718+0000 7fd9c9d9b640 1 -- 172.21.15.60:0/414977593 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aae73bffd0 con 0x55aae73c4d60 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/414977593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73c4d60 0x55aae73de150 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4004700 tx=0x7fd9b400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79c9b7576d57e1e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9b3fff640 1 -- 172.21.15.60:0/414977593 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4011810 con 0x55aae73c4d60 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9b3fff640 1 -- 172.21.15.60:0/414977593 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9b4022110 con 0x55aae73c4d60 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9b3fff640 1 -- 172.21.15.60:0/414977593 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4020690 con 0x55aae73c4d60 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9d39eb740 1 -- 172.21.15.60:0/414977593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73c4d60 msgr2=0x55aae73de150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/414977593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73c4d60 0x55aae73de150 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4004700 tx=0x7fd9b400eb50 comp rx=0 tx=0).stop 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9d39eb740 1 -- 172.21.15.60:0/414977593 shutdown_connections 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/414977593 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73c4d60 0x55aae73de150 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/414977593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73bfbf0 0x55aae73d0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/414977593 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73bf200 0x55aae73c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9d39eb740 1 -- 172.21.15.60:0/414977593 >> 172.21.15.60:0/414977593 conn(0x55aae73b6360 msgr2=0x55aae73c92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9d39eb740 1 -- 172.21.15.60:0/414977593 shutdown_connections 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.719+0000 7fd9d39eb740 1 -- 172.21.15.60:0/414977593 wait complete. 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9d39eb740 1 -- start start 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9d39eb740 1 Processor -- start 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9d39eb740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bf200 0x55aae73d0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9d39eb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73bfbf0 0x55aae73cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9d39eb740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73c4d60 0x55aae73cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9d39eb740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aae73d6e50 con 0x55aae73bf200 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9d39eb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aae73d67f0 con 0x55aae73bfbf0 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9d39eb740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aae73d74b0 con 0x55aae73c4d60 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9d39eb740 10 client.0 _ll_register_callbacks cb 0x7fd9c402c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9c9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bf200 0x55aae73d0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9c9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bf200 0x55aae73d0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58500/0 (socket says 172.21.15.60:58500) 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9c9d9b640 1 -- 172.21.15.60:0/754354585 learned_addr learned my addr 172.21.15.60:0/754354585 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:19.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73c4d60 0x55aae73cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9ca59c640 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73bfbf0 0x55aae73cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9c9d9b640 1 -- 172.21.15.60:0/754354585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73c4d60 msgr2=0x55aae73cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73c4d60 0x55aae73cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9c9d9b640 1 -- 172.21.15.60:0/754354585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73bfbf0 msgr2=0x55aae73cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73bfbf0 0x55aae73cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9c9d9b640 1 -- 172.21.15.60:0/754354585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aae742be20 con 0x55aae73bf200 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9ca59c640 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73bfbf0 0x55aae73cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9c9d9b640 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bf200 0x55aae73d0c10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4005840 tx=0x7fd9b40133f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d4968fd6825278 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.720+0000 7fd9b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9b1ffb640 1 -- 172.21.15.60:0/754354585 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40143e0 con 0x55aae73bf200 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9b1ffb640 1 -- 172.21.15.60:0/754354585 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd9b4027050 con 0x55aae73bf200 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9d39eb740 1 -- 172.21.15.60:0/754354585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9b4022110 con 0x55aae73bf200 2024-03-04T12:23:19.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9b1ffb640 1 -- 172.21.15.60:0/754354585 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9b403b050 con 0x55aae73bf200 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9d39eb740 10 client.14775 fetch_fsmap learned FSMap version 17 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9d39eb740 1 -- 172.21.15.60:0/754354585 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55aae742d120 con 0x55aae73bf200 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9b1ffb640 1 -- 172.21.15.60:0/754354585 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9b4022110 con 0x55aae73bf200 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9b1ffb640 1 -- 172.21.15.60:0/754354585 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 17) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fd9b402d040 con 0x55aae73bf200 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9d39eb740 10 client.14775 fetch_fsmap finished waiting for FSMap version 17 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9d39eb740 10 client.14775 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9d39eb740 20 client.14775 populate_metadata read hostname 'smithi060' 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55aae74313e0 0x55aae742d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9d39eb740 4 client.14775 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9d39eb740 1 -- 172.21.15.60:0/754354585 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55aae7328050 con 0x55aae74313e0 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.721+0000 7fd9ca59c640 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55aae74313e0 0x55aae742d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9ca59c640 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55aae74313e0 0x55aae742d350 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be38baf28412f1e8 server_cookie=ff681833187d3182 in_seq=0 out_seq=0 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9b1ffb640 10 client.14775 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:23:19.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9b1ffb640 1 -- 172.21.15.60:0/754354585 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55aae7328050 con 0x55aae74313e0 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9b1ffb640 10 client.14775 handle_command_reply: tid=0 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d39eb740 1 client.14775 shutdown 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d39eb740 1 -- 172.21.15.60:0/754354585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bf200 msgr2=0x55aae73d0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bf200 0x55aae73d0c10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4005840 tx=0x7fd9b40133f0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d39eb740 1 -- 172.21.15.60:0/754354585 shutdown_connections 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55aae74313e0 0x55aae742d350 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55aae73c4d60 0x55aae73cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55aae73bfbf0 0x55aae73cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9b1ffb640 0 client.14775 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d39eb740 1 --2- 172.21.15.60:0/754354585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55aae73bf200 0x55aae73d0c10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d39eb740 1 -- 172.21.15.60:0/754354585 >> 172.21.15.60:0/754354585 conn(0x55aae73b6360 msgr2=0x55aae73de9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d39eb740 1 -- 172.21.15.60:0/754354585 shutdown_connections 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d39eb740 1 -- 172.21.15.60:0/754354585 wait complete. 2024-03-04T12:23:19.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d39eb740 20 client.14775 trim_cache size 0 max 16384 2024-03-04T12:23:19.735 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:23:19.735 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:23:19.735 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2024-03-04T12:23:19.735 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 24524, 2024-03-04T12:23:19.735 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-03-04T12:23:19.735 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-03-04T12:23:19.735 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-03-04T12:23:19.735 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 6.8182192160000001, 2024-03-04T12:23:19.735 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 17, 2024-03-04T12:23:19.735 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 89, 2024-03-04T12:23:19.735 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 88, 2024-03-04T12:23:19.735 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 13.119563105999999 2024-03-04T12:23:19.736 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:23:19.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d0d37640 1 -- 172.21.15.60:0/3474440408 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9b80a1d80 msgr2=0x7fd9b80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d0d37640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9b80a1d80 0x7fd9b80a4240 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc09aa60 tx=0x7fd9bc06de20 comp rx=0 tx=0).stop 2024-03-04T12:23:19.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d0d37640 1 -- 172.21.15.60:0/3474440408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9cc107030 msgr2=0x7fd9cc19ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:19.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.722+0000 7fd9d0d37640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9cc107030 0x7fd9cc19ec80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00eaa0 tx=0x7fd9bc00ef60 comp rx=0 tx=0).stop 2024-03-04T12:23:19.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.723+0000 7fd9d0d37640 1 -- 172.21.15.60:0/3474440408 shutdown_connections 2024-03-04T12:23:19.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.723+0000 7fd9d0d37640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9b80a1d80 0x7fd9b80a4240 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.723+0000 7fd9d0d37640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9cc107030 0x7fd9cc19ec80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.723+0000 7fd9d0d37640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9cc104700 0x7fd9cc197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.723+0000 7fd9d0d37640 1 --2- 172.21.15.60:0/3474440408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9cc101dd0 0x7fd9cc1976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:19.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.723+0000 7fd9d0d37640 1 -- 172.21.15.60:0/3474440408 >> 172.21.15.60:0/3474440408 conn(0x7fd9cc0fb7c0 msgr2=0x7fd9cc102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:19.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.723+0000 7fd9d0d37640 1 -- 172.21.15.60:0/3474440408 shutdown_connections 2024-03-04T12:23:19.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:19.723+0000 7fd9d0d37640 1 -- 172.21.15.60:0/3474440408 wait complete. 2024-03-04T12:23:19.739 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T12:23:19.739 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-04T12:23:19.739 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph fs ls 2024-03-04T12:23:19.984 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:20.219 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:19 smithi060 ceph-mon[32745]: pgmap v296: 129 pgs: 15 unknown, 114 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-03-04T12:23:20.219 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:19 smithi060 ceph-mon[32745]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T12:23:20.219 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:19 smithi060 ceph-mon[32745]: Cluster is now healthy 2024-03-04T12:23:20.219 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:19 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1617237787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:20.219 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:19 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1756086914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:20.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:19 smithi086 ceph-mon[36851]: pgmap v296: 129 pgs: 15 unknown, 114 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-03-04T12:23:20.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:19 smithi086 ceph-mon[36851]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T12:23:20.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:19 smithi086 ceph-mon[36851]: Cluster is now healthy 2024-03-04T12:23:20.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:19 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1617237787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:20.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:19 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1756086914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:20.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:19 smithi063 ceph-mon[36717]: pgmap v296: 129 pgs: 15 unknown, 114 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-03-04T12:23:20.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:19 smithi063 ceph-mon[36717]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T12:23:20.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:19 smithi063 ceph-mon[36717]: Cluster is now healthy 2024-03-04T12:23:20.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:19 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1617237787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:20.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:19 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1756086914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:21.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.825+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/866317298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74101120 msgr2=0x7fdb7410d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.825+0000 7fdb7bdbc700 1 --2- 172.21.15.60:0/866317298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74101120 0x7fdb7410d330 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c00b0a0 tx=0x7fdb6c00b3b0 comp rx=0 tx=0).stop 2024-03-04T12:23:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.825+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/866317298 shutdown_connections 2024-03-04T12:23:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.825+0000 7fdb7bdbc700 1 --2- 172.21.15.60:0/866317298 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74101120 0x7fdb7410d330 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.825+0000 7fdb7bdbc700 1 --2- 172.21.15.60:0/866317298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb74106560 0x7fdb74106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.825+0000 7fdb7bdbc700 1 --2- 172.21.15.60:0/866317298 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb74100760 0x7fdb74100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.825+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/866317298 >> 172.21.15.60:0/866317298 conn(0x7fdb74076080 msgr2=0x7fdb74076490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.825+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/866317298 shutdown_connections 2024-03-04T12:23:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.826+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/866317298 wait complete. 2024-03-04T12:23:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.826+0000 7fdb7bdbc700 1 Processor -- start 2024-03-04T12:23:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.826+0000 7fdb7bdbc700 1 -- start start 2024-03-04T12:23:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.826+0000 7fdb7bdbc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb74100760 0x7fdb740737d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:21.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.826+0000 7fdb7bdbc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74101120 0x7fdb7406e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.826+0000 7fdb7bdbc700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb74106560 0x7fdb7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.826+0000 7fdb7bdbc700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7410fff0 con 0x7fdb74101120 2024-03-04T12:23:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.826+0000 7fdb7bdbc700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7410fe70 con 0x7fdb74106560 2024-03-04T12:23:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.826+0000 7fdb7bdbc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb74110170 con 0x7fdb74100760 2024-03-04T12:23:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.826+0000 7fdb79357700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74101120 0x7fdb7406e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb79357700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74101120 0x7fdb7406e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58528/0 (socket says 172.21.15.60:58528) 2024-03-04T12:23:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb79357700 1 -- 172.21.15.60:0/3365557789 learned_addr learned my addr 172.21.15.60:0/3365557789 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb79b58700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb74100760 0x7fdb740737d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb7a359700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb74106560 0x7fdb7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb79357700 1 -- 172.21.15.60:0/3365557789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb74100760 msgr2=0x7fdb740737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:21.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb79357700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb74100760 0x7fdb740737d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb79357700 1 -- 172.21.15.60:0/3365557789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb74106560 msgr2=0x7fdb7406edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb79357700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb74106560 0x7fdb7406edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb79357700 1 -- 172.21.15.60:0/3365557789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7406f520 con 0x7fdb74101120 2024-03-04T12:23:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb79357700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74101120 0x7fdb7406e870 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000d950 tx=0x7fdb7000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb6affd700 1 -- 172.21.15.60:0/3365557789 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb70016070 con 0x7fdb74101120 2024-03-04T12:23:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb6affd700 1 -- 172.21.15.60:0/3365557789 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb70009e20 con 0x7fdb74101120 2024-03-04T12:23:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/3365557789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb741ad4b0 con 0x7fdb74101120 2024-03-04T12:23:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb6affd700 1 -- 172.21.15.60:0/3365557789 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7000f620 con 0x7fdb74101120 2024-03-04T12:23:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.827+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/3365557789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb741ad9e0 con 0x7fdb74101120 2024-03-04T12:23:21.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.828+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/3365557789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb74101bd0 con 0x7fdb74101120 2024-03-04T12:23:21.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.829+0000 7fdb6affd700 1 -- 172.21.15.60:0/3365557789 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fdb70019070 con 0x7fdb74101120 2024-03-04T12:23:21.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.830+0000 7fdb6affd700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdb600a1e80 0x7fdb600a4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:21.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.830+0000 7fdb79b58700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdb600a1e80 0x7fdb600a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:21.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.830+0000 7fdb6affd700 1 -- 172.21.15.60:0/3365557789 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fdb700cdb10 con 0x7fdb74101120 2024-03-04T12:23:21.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.830+0000 7fdb79b58700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdb600a1e80 0x7fdb600a4340 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdb64005950 tx=0x7fdb6400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:21.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:21.835+0000 7fdb6affd700 1 -- 172.21.15.60:0/3365557789 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdb7009aca0 con 0x7fdb74101120 2024-03-04T12:23:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.086+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/3365557789 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdb7406fd90 con 0x7fdb74101120 2024-03-04T12:23:22.092 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:21 smithi060 ceph-mon[32745]: pgmap v297: 129 pgs: 4 unknown, 125 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-03-04T12:23:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.086+0000 7fdb6affd700 1 -- 172.21.15.60:0/3365557789 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7fdb7009a5a0 con 0x7fdb74101120 2024-03-04T12:23:22.093 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-04T12:23:22.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.089+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/3365557789 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdb600a1e80 msgr2=0x7fdb600a4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:22.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.089+0000 7fdb7bdbc700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdb600a1e80 0x7fdb600a4340 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdb64005950 tx=0x7fdb6400a300 comp rx=0 tx=0).stop 2024-03-04T12:23:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.089+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/3365557789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74101120 msgr2=0x7fdb7406e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.089+0000 7fdb7bdbc700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74101120 0x7fdb7406e870 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000d950 tx=0x7fdb7000dc60 comp rx=0 tx=0).stop 2024-03-04T12:23:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.089+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/3365557789 shutdown_connections 2024-03-04T12:23:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.089+0000 7fdb7bdbc700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb74106560 0x7fdb7406edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.089+0000 7fdb7bdbc700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb74100760 0x7fdb740737d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.089+0000 7fdb7bdbc700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdb600a1e80 0x7fdb600a4340 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.089+0000 7fdb7bdbc700 1 --2- 172.21.15.60:0/3365557789 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdb74101120 0x7fdb7406e870 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.089+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/3365557789 >> 172.21.15.60:0/3365557789 conn(0x7fdb74076080 msgr2=0x7fdb740ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.089+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/3365557789 shutdown_connections 2024-03-04T12:23:22.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:22.089+0000 7fdb7bdbc700 1 -- 172.21.15.60:0/3365557789 wait complete. 2024-03-04T12:23:22.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:21 smithi086 ceph-mon[36851]: pgmap v297: 129 pgs: 4 unknown, 125 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-03-04T12:23:22.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:21 smithi063 ceph-mon[36717]: pgmap v297: 129 pgs: 4 unknown, 125 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-03-04T12:23:22.778 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-04T12:23:22.778 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T12:23:22.778 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2024-03-04T12:23:22.778 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-04T12:23:22.778 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-04T12:23:22.779 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-04T12:23:22.779 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T12:23:22.779 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T12:23:22.779 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-04T12:23:22.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:22.779 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2024-03-04T12:23:22.795 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T12:23:22.795 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2024-03-04T12:23:22.796 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T12:23:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 42177sec preferred_lft 42177sec 2024-03-04T12:23:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2024-03-04T12:23:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-03-04T12:23:22.797 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T12:23:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2024-03-04T12:23:22.797 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2024-03-04T12:23:22.797 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-04T12:23:22.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:22.798 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2024-03-04T12:23:22.798 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-03-04T12:23:22.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add name ceph-brx type bridge 2024-03-04T12:23:22.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip addr flush dev ceph-brx 2024-03-04T12:23:22.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set ceph-brx up 2024-03-04T12:23:22.798 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-04T12:23:22.798 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-03-04T12:23:22.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:22.949 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T12:23:22.975 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-03-04T12:23:22.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:22.977 DEBUG:teuthology.orchestra.run.smithi060:> route 2024-03-04T12:23:23.047 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2024-03-04T12:23:23.047 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T12:23:23.047 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T12:23:23.047 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T12:23:23.047 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T12:23:23.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:23.048 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2024-03-04T12:23:23.048 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-03-04T12:23:23.048 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T12:23:23.048 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T12:23:23.048 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T12:23:23.048 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-03-04T12:23:23.113 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:22 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3365557789' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T12:23:23.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:23.325 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-03-04T12:23:23.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:22 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3365557789' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T12:23:23.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:22 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3365557789' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T12:23:23.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:23.381 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list-id 2024-03-04T12:23:23.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:23.437 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2024-03-04T12:23:23.437 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-03-04T12:23:23.437 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T12:23:23.437 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-04T12:23:23.437 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-03-04T12:23:23.522 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-04T12:23:23.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:23.522 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2024-03-04T12:23:23.522 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-03-04T12:23:23.522 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-04T12:23:23.522 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-04T12:23:23.522 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-04T12:23:23.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-04T12:23:23.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-04T12:23:23.523 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-03-04T12:23:23.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:23.661 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2024-03-04T12:23:23.661 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-03-04T12:23:23.662 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2024-03-04T12:23:23.662 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set dev brx.0 master ceph-brx 2024-03-04T12:23:23.662 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-03-04T12:23:23.703 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-04T12:23:23.704 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T12:23:23.704 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-04T12:23:23.762 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-04T12:23:23.762 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T12:23:23.762 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-04T12:23:23.817 DEBUG:teuthology.orchestra.run.smithi060:> sudo modprobe fuse 2024-03-04T12:23:23.888 DEBUG:teuthology.orchestra.run.smithi060:> cat /proc/self/mounts | awk '{print $2}' 2024-03-04T12:23:23.942 INFO:teuthology.orchestra.run.smithi060.stdout:/proc 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/sys 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/dev 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/security 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/shm 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/pts 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/run 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/cgroup 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/pstore 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/bpf 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/ 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/selinux 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T12:23:23.943 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/mqueue 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/debug 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/hugepages 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/tracing 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/fuse/connections 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/config 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-sysctl.service 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/nfs/rpc_pipefs 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/run/user/1000 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/c4a102f7a095da0fde90b9779e9b871022f81313d36f134bf0674245bc178f3a/merged 2024-03-04T12:23:23.944 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/ad1d20e4fc7ef75db5a0c328b8375a6f0f6dc9d5c7e0262cb4f57f833af588a4/merged 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/run/user/0 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/f1672df3d5c296519b3fddac45de52ea36246890e702e22bc630df6935ee0053/merged 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/4823d1f2b2463e8cb8b607d307c56ebb2a70f0049b89af0e71e1c3dc3fe70bcd/merged 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/38f7ea4fd5c8e700ffaf733653e1db7571b477284edb93622a0d6228b3d79d93/merged 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/7fd89f1d2c09f40a834d2f3b2c468f7eb45921aa58710b1c9b95319dfa923d92/merged 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/0d6dd2b1404937a83a7be0fd54514cf4adb139ee463e96e5096ceb0c939e95d2/merged 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/9fa9c3f9a819e7061d67b66f4db2952ca0881eb29aa054ecfa66d1e2336d9e40/merged 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/e3e9b9657de03e125fb9ed16494a2dab386335c220b227376d70c8045371630b/merged 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/4a5bee2b063dd88be60df02286a771ad4ed9d56f3f562171b39abfdd882ffdb8/merged 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T12:23:23.945 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T12:23:23.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:23.946 DEBUG:teuthology.orchestra.run.smithi060:> ls /sys/fs/fuse/connections 2024-03-04T12:23:23.998 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-04T12:23:23.998 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-04T12:23:24.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo modprobe fuse 2024-03-04T12:23:24.067 DEBUG:teuthology.orchestra.run.smithi060:> cat /proc/self/mounts | awk '{print $2}' 2024-03-04T12:23:24.123 INFO:teuthology.orchestra.run.smithi060.stdout:/proc 2024-03-04T12:23:24.123 INFO:teuthology.orchestra.run.smithi060.stdout:/sys 2024-03-04T12:23:24.123 INFO:teuthology.orchestra.run.smithi060.stdout:/dev 2024-03-04T12:23:24.123 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/security 2024-03-04T12:23:24.123 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/shm 2024-03-04T12:23:24.123 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/pts 2024-03-04T12:23:24.123 INFO:teuthology.orchestra.run.smithi060.stdout:/run 2024-03-04T12:23:24.123 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/cgroup 2024-03-04T12:23:24.123 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/pstore 2024-03-04T12:23:24.124 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/bpf 2024-03-04T12:23:24.124 INFO:teuthology.orchestra.run.smithi060.stdout:/ 2024-03-04T12:23:24.124 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/selinux 2024-03-04T12:23:24.124 INFO:teuthology.orchestra.run.smithi060.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T12:23:24.124 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/mqueue 2024-03-04T12:23:24.124 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/debug 2024-03-04T12:23:24.124 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/hugepages 2024-03-04T12:23:24.124 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/tracing 2024-03-04T12:23:24.124 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-04T12:23:24.124 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/fuse/connections 2024-03-04T12:23:24.124 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/config 2024-03-04T12:23:24.124 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-sysctl.service 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/nfs/rpc_pipefs 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/run/user/1000 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/c4a102f7a095da0fde90b9779e9b871022f81313d36f134bf0674245bc178f3a/merged 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/ad1d20e4fc7ef75db5a0c328b8375a6f0f6dc9d5c7e0262cb4f57f833af588a4/merged 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/run/user/0 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/f1672df3d5c296519b3fddac45de52ea36246890e702e22bc630df6935ee0053/merged 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/4823d1f2b2463e8cb8b607d307c56ebb2a70f0049b89af0e71e1c3dc3fe70bcd/merged 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/38f7ea4fd5c8e700ffaf733653e1db7571b477284edb93622a0d6228b3d79d93/merged 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/7fd89f1d2c09f40a834d2f3b2c468f7eb45921aa58710b1c9b95319dfa923d92/merged 2024-03-04T12:23:24.125 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/0d6dd2b1404937a83a7be0fd54514cf4adb139ee463e96e5096ceb0c939e95d2/merged 2024-03-04T12:23:24.126 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/9fa9c3f9a819e7061d67b66f4db2952ca0881eb29aa054ecfa66d1e2336d9e40/merged 2024-03-04T12:23:24.126 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/e3e9b9657de03e125fb9ed16494a2dab386335c220b227376d70c8045371630b/merged 2024-03-04T12:23:24.126 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/4a5bee2b063dd88be60df02286a771ad4ed9d56f3f562171b39abfdd882ffdb8/merged 2024-03-04T12:23:24.126 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns 2024-03-04T12:23:24.126 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T12:23:24.126 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T12:23:24.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:24.126 DEBUG:teuthology.orchestra.run.smithi060:> ls /sys/fs/fuse/connections 2024-03-04T12:23:24.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:23 smithi060 ceph-mon[32745]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 2 op/s 2024-03-04T12:23:24.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi060.stderr:2024-03-04T12:23:24.182+0000 7f6e6ee3a480 -1 init, newargv = 0x5576716d2a70 newargc=15 2024-03-04T12:23:24.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi060.stderr:ceph-fuse[77485]: starting ceph client 2024-03-04T12:23:24.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi060.stderr:ceph-fuse[77485]: starting fuse 2024-03-04T12:23:24.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:23 smithi086 ceph-mon[36851]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 2 op/s 2024-03-04T12:23:24.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:23 smithi063 ceph-mon[36717]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 2 op/s 2024-03-04T12:23:25.281 DEBUG:teuthology.orchestra.run.smithi060:> sudo modprobe fuse 2024-03-04T12:23:25.315 DEBUG:teuthology.orchestra.run.smithi060:> cat /proc/self/mounts | awk '{print $2}' 2024-03-04T12:23:25.373 INFO:teuthology.orchestra.run.smithi060.stdout:/proc 2024-03-04T12:23:25.373 INFO:teuthology.orchestra.run.smithi060.stdout:/sys 2024-03-04T12:23:25.373 INFO:teuthology.orchestra.run.smithi060.stdout:/dev 2024-03-04T12:23:25.373 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/security 2024-03-04T12:23:25.373 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/shm 2024-03-04T12:23:25.373 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/pts 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/run 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/cgroup 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/pstore 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/bpf 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/ 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/selinux 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/mqueue 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/debug 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/dev/hugepages 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/tracing 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/fs/fuse/connections 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/sys/kernel/config 2024-03-04T12:23:25.374 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-sysctl.service 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/nfs/rpc_pipefs 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/run/user/1000 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/ceph 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/c4a102f7a095da0fde90b9779e9b871022f81313d36f134bf0674245bc178f3a/merged 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/ad1d20e4fc7ef75db5a0c328b8375a6f0f6dc9d5c7e0262cb4f57f833af588a4/merged 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/run/user/0 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/proc/sys/fs/binfmt_misc 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/f1672df3d5c296519b3fddac45de52ea36246890e702e22bc630df6935ee0053/merged 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/4823d1f2b2463e8cb8b607d307c56ebb2a70f0049b89af0e71e1c3dc3fe70bcd/merged 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/38f7ea4fd5c8e700ffaf733653e1db7571b477284edb93622a0d6228b3d79d93/merged 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/7fd89f1d2c09f40a834d2f3b2c468f7eb45921aa58710b1c9b95319dfa923d92/merged 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/0d6dd2b1404937a83a7be0fd54514cf4adb139ee463e96e5096ceb0c939e95d2/merged 2024-03-04T12:23:25.375 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/9fa9c3f9a819e7061d67b66f4db2952ca0881eb29aa054ecfa66d1e2336d9e40/merged 2024-03-04T12:23:25.376 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/e3e9b9657de03e125fb9ed16494a2dab386335c220b227376d70c8045371630b/merged 2024-03-04T12:23:25.376 INFO:teuthology.orchestra.run.smithi060.stdout:/var/lib/containers/storage/overlay/4a5bee2b063dd88be60df02286a771ad4ed9d56f3f562171b39abfdd882ffdb8/merged 2024-03-04T12:23:25.376 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns 2024-03-04T12:23:25.376 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T12:23:25.376 INFO:teuthology.orchestra.run.smithi060.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T12:23:25.376 INFO:teuthology.orchestra.run.smithi060.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-04T12:23:25.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:25.377 DEBUG:teuthology.orchestra.run.smithi060:> ls /sys/fs/fuse/connections 2024-03-04T12:23:25.434 INFO:teuthology.orchestra.run.smithi060.stdout:119 2024-03-04T12:23:25.435 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-04T12:23:25.435 DEBUG:teuthology.orchestra.run.smithi060:> sudo stdin-killer -- python3 -c ' 2024-03-04T12:23:25.435 DEBUG:teuthology.orchestra.run.smithi060:> import glob 2024-03-04T12:23:25.435 DEBUG:teuthology.orchestra.run.smithi060:> import re 2024-03-04T12:23:25.435 DEBUG:teuthology.orchestra.run.smithi060:> import os 2024-03-04T12:23:25.435 DEBUG:teuthology.orchestra.run.smithi060:> import subprocess 2024-03-04T12:23:25.435 DEBUG:teuthology.orchestra.run.smithi060:> 2024-03-04T12:23:25.435 DEBUG:teuthology.orchestra.run.smithi060:> def _find_admin_socket(client_name): 2024-03-04T12:23:25.435 DEBUG:teuthology.orchestra.run.smithi060:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-04T12:23:25.435 DEBUG:teuthology.orchestra.run.smithi060:> files = glob.glob(asok_path) 2024-03-04T12:23:25.435 DEBUG:teuthology.orchestra.run.smithi060:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> # Given a non-glob path, it better be there 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> if "*" not in asok_path: 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> assert(len(files) == 1) 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> return files[0] 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> for f in files: 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> contents = proc_f.read() 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> if mountpoint in contents: 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> return f 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> 2024-03-04T12:23:25.436 DEBUG:teuthology.orchestra.run.smithi060:> print(_find_admin_socket("client.admin")) 2024-03-04T12:23:25.437 DEBUG:teuthology.orchestra.run.smithi060:> ' 2024-03-04T12:23:25.564 INFO:teuthology.orchestra.run.smithi060.stdout:/var/run/ceph/ceph-client.admin.77485.asok 2024-03-04T12:23:25.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-04T12:23:25.575 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.77485.asok 2024-03-04T12:23:25.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:25.575 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.77485.asok status 2024-03-04T12:23:25.715 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: "metadata": { 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_sha1": "717ce3cce8d0166e4e577faa8004e6f9cb4128c0", 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_version": "ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev)", 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: "entity_id": "admin", 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: "hostname": "smithi060", 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: "pid": "77485", 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: "root": "/" 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: "dentry_count": 0, 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: "dentry_pinned_count": 0, 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 14787, 2024-03-04T12:23:25.716 INFO:teuthology.orchestra.run.smithi060.stdout: "inst": { 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: "name": { 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "client", 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: "num": 14787 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": { 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "192.168.0.1:0", 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 2158919632 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": { 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-04T12:23:25.717 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "192.168.0.1:0", 2024-03-04T12:23:25.718 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 2158919632 2024-03-04T12:23:25.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:23:25.718 INFO:teuthology.orchestra.run.smithi060.stdout: "inst_str": "client.14787 192.168.0.1:0/2158919632", 2024-03-04T12:23:25.718 INFO:teuthology.orchestra.run.smithi060.stdout: "addr_str": "192.168.0.1:0/2158919632", 2024-03-04T12:23:25.718 INFO:teuthology.orchestra.run.smithi060.stdout: "inode_count": 1, 2024-03-04T12:23:25.718 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_epoch": 17, 2024-03-04T12:23:25.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_epoch": 89, 2024-03-04T12:23:25.718 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_epoch_barrier": 0, 2024-03-04T12:23:25.718 INFO:teuthology.orchestra.run.smithi060.stdout: "blocklisted": false, 2024-03-04T12:23:25.718 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs" 2024-03-04T12:23:25.718 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:23:25.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:25.724 DEBUG:teuthology.orchestra.run.smithi060:> stat --file-system '--printf=%T 2024-03-04T12:23:25.724 DEBUG:teuthology.orchestra.run.smithi060:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-04T12:23:25.739 INFO:teuthology.orchestra.run.smithi060.stdout:fuseblk 2024-03-04T12:23:25.739 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-04T12:23:25.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:25.739 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-04T12:23:25.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:25.808 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-04T12:23:25.884 INFO:teuthology.orchestra.run.smithi060.stdout:# file: . 2024-03-04T12:23:25.884 INFO:teuthology.orchestra.run.smithi060.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-04T12:23:25.884 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:25.885 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:23:25.885 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T12:23:25.940 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:23:26.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.077+0000 7f389e1e6640 1 -- 172.21.15.60:0/3928125601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3898101de0 msgr2=0x7f38981041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.077+0000 7f389e1e6640 1 --2- 172.21.15.60:0/3928125601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3898101de0 0x7f38981041d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3888009900 tx=0x7f388802bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:26.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.077+0000 7f389e1e6640 1 -- 172.21.15.60:0/3928125601 shutdown_connections 2024-03-04T12:23:26.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.077+0000 7f389e1e6640 1 --2- 172.21.15.60:0/3928125601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3898107040 0x7f3898109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.077+0000 7f389e1e6640 1 --2- 172.21.15.60:0/3928125601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3898104710 0x7f3898106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.077+0000 7f389e1e6640 1 --2- 172.21.15.60:0/3928125601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3898101de0 0x7f38981041d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.077+0000 7f389e1e6640 1 -- 172.21.15.60:0/3928125601 >> 172.21.15.60:0/3928125601 conn(0x7f38980fb7d0 msgr2=0x7f38980fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:26.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.078+0000 7f389e1e6640 1 -- 172.21.15.60:0/3928125601 shutdown_connections 2024-03-04T12:23:26.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.078+0000 7f389e1e6640 1 -- 172.21.15.60:0/3928125601 wait complete. 2024-03-04T12:23:26.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.078+0000 7f389e1e6640 1 Processor -- start 2024-03-04T12:23:26.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.078+0000 7f389e1e6640 1 -- start start 2024-03-04T12:23:26.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.078+0000 7f389e1e6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3898101de0 0x7f3898197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:26.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.078+0000 7f389e1e6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3898104710 0x7f3898197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:26.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.078+0000 7f389e1e6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3898107040 0x7f389819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:26.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.078+0000 7f389e1e6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389810cff0 con 0x7f3898104710 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.078+0000 7f389e1e6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389810ce70 con 0x7f3898101de0 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.078+0000 7f389e1e6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389810d170 con 0x7f3898107040 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.078+0000 7f3896ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3898104710 0x7f3898197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3896ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3898104710 0x7f3898197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52272/0 (socket says 172.21.15.60:52272) 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3896ffd640 1 -- 172.21.15.60:0/2360830939 learned_addr learned my addr 172.21.15.60:0/2360830939 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3897fff640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3898107040 0x7f389819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f38977fe640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3898101de0 0x7f3898197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3896ffd640 1 -- 172.21.15.60:0/2360830939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3898107040 msgr2=0x7f389819ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3896ffd640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3898107040 0x7f389819ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3896ffd640 1 -- 172.21.15.60:0/2360830939 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3898101de0 msgr2=0x7f3898197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3896ffd640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3898101de0 0x7f3898197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3896ffd640 1 -- 172.21.15.60:0/2360830939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389819f2b0 con 0x7f3898104710 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3897fff640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3898107040 0x7f389819ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:26.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3896ffd640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3898104710 0x7f3898197bd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f388c00c970 tx=0x7f388c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:26.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3894ff9640 1 -- 172.21.15.60:0/2360830939 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c007bb0 con 0x7f3898104710 2024-03-04T12:23:26.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3894ff9640 1 -- 172.21.15.60:0/2360830939 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f388c007d50 con 0x7f3898104710 2024-03-04T12:23:26.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f389e1e6640 1 -- 172.21.15.60:0/2360830939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389819f5a0 con 0x7f3898104710 2024-03-04T12:23:26.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f3894ff9640 1 -- 172.21.15.60:0/2360830939 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c0187a0 con 0x7f3898104710 2024-03-04T12:23:26.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.079+0000 7f389e1e6640 1 -- 172.21.15.60:0/2360830939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389819faa0 con 0x7f3898104710 2024-03-04T12:23:26.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.080+0000 7f389e1e6640 1 -- 172.21.15.60:0/2360830939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38980fd2c0 con 0x7f3898104710 2024-03-04T12:23:26.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.081+0000 7f3894ff9640 1 -- 172.21.15.60:0/2360830939 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f388c00f450 con 0x7f3898104710 2024-03-04T12:23:26.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.082+0000 7f3894ff9640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f38800a1d80 0x7f38800a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:26.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.082+0000 7f38977fe640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f38800a1d80 0x7f38800a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:26.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.082+0000 7f3894ff9640 1 -- 172.21.15.60:0/2360830939 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f388c0ce4d0 con 0x7f3898104710 2024-03-04T12:23:26.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.083+0000 7f38977fe640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f38800a1d80 0x7f38800a4240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3888006010 tx=0x7f3888000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:26.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.086+0000 7f3894ff9640 1 -- 172.21.15.60:0/2360830939 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f388c014030 con 0x7f3898104710 2024-03-04T12:23:26.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.271+0000 7f389e1e6640 1 -- 172.21.15.60:0/2360830939 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f389802d100 con 0x7f3898104710 2024-03-04T12:23:26.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.273+0000 7f3894ff9640 1 -- 172.21.15.60:0/2360830939 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23997 (secure 0 0 0) 0x7f388c09b070 con 0x7f3898104710 2024-03-04T12:23:26.279 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:26.279 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":89,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:23:17.289925+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:23:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.275+0000 7f389e1e6640 1 -- 172.21.15.60:0/2360830939 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f38800a1d80 msgr2=0x7f38800a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.275+0000 7f389e1e6640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f38800a1d80 0x7f38800a4240 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3888006010 tx=0x7f3888000bc0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.275+0000 7f389e1e6640 1 -- 172.21.15.60:0/2360830939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3898104710 msgr2=0x7f3898197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.275+0000 7f389e1e6640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3898104710 0x7f3898197bd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f388c00c970 tx=0x7f388c00ce30 comp rx=0 tx=0).stop 2024-03-04T12:23:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.275+0000 7f389e1e6640 1 -- 172.21.15.60:0/2360830939 shutdown_connections 2024-03-04T12:23:26.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.275+0000 7f389e1e6640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f38800a1d80 0x7f38800a4240 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.275+0000 7f389e1e6640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3898107040 0x7f389819ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.275+0000 7f389e1e6640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3898104710 0x7f3898197bd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.275+0000 7f389e1e6640 1 --2- 172.21.15.60:0/2360830939 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3898101de0 0x7f3898197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.275+0000 7f389e1e6640 1 -- 172.21.15.60:0/2360830939 >> 172.21.15.60:0/2360830939 conn(0x7f38980fb7d0 msgr2=0x7f38981026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.275+0000 7f389e1e6640 1 -- 172.21.15.60:0/2360830939 shutdown_connections 2024-03-04T12:23:26.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.275+0000 7f389e1e6640 1 -- 172.21.15.60:0/2360830939 wait complete. 2024-03-04T12:23:26.291 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:25 smithi060 ceph-mon[32745]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-03-04T12:23:26.294 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T12:23:10.438928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T12:23:11.472998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T12:23:15.162500+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T12:23:26.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:23:26.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:25 smithi086 ceph-mon[36851]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-03-04T12:23:26.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:25 smithi063 ceph-mon[36717]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-03-04T12:23:26.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.395+0000 7fc6e79d4640 1 -- 172.21.15.60:0/560470167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6e0107030 msgr2=0x7fc6e0109480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.395+0000 7fc6e79d4640 1 --2- 172.21.15.60:0/560470167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6e0107030 0x7fc6e0109480 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009ca0 tx=0x7fc6d802c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.395+0000 7fc6e79d4640 1 -- 172.21.15.60:0/560470167 shutdown_connections 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.395+0000 7fc6e79d4640 1 --2- 172.21.15.60:0/560470167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6e0107030 0x7fc6e0109480 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.395+0000 7fc6e79d4640 1 --2- 172.21.15.60:0/560470167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6e0104700 0x7fc6e0106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.395+0000 7fc6e79d4640 1 --2- 172.21.15.60:0/560470167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6e0101dd0 0x7fc6e01041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.395+0000 7fc6e79d4640 1 -- 172.21.15.60:0/560470167 >> 172.21.15.60:0/560470167 conn(0x7fc6e00fb7e0 msgr2=0x7fc6e00fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.395+0000 7fc6e79d4640 1 -- 172.21.15.60:0/560470167 shutdown_connections 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.395+0000 7fc6e79d4640 1 -- 172.21.15.60:0/560470167 wait complete. 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e79d4640 1 Processor -- start 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e79d4640 1 -- start start 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e79d4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6e0101dd0 0x7fc6e0197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e79d4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6e0104700 0x7fc6e0197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e79d4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6e0107030 0x7fc6e019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:26.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e79d4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e010d040 con 0x7fc6e0107030 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e79d4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e010cec0 con 0x7fc6e0104700 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e79d4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e010d1c0 con 0x7fc6e0101dd0 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e5f71640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6e0107030 0x7fc6e019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e5f71640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6e0107030 0x7fc6e019ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52292/0 (socket says 172.21.15.60:52292) 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e5f71640 1 -- 172.21.15.60:0/679230137 learned_addr learned my addr 172.21.15.60:0/679230137 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e5770640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6e0101dd0 0x7fc6e0197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e4f6f640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6e0104700 0x7fc6e0197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e5f71640 1 -- 172.21.15.60:0/679230137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6e0101dd0 msgr2=0x7fc6e0197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e5f71640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6e0101dd0 0x7fc6e0197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e5f71640 1 -- 172.21.15.60:0/679230137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6e0104700 msgr2=0x7fc6e0197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e5f71640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6e0104700 0x7fc6e0197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e5f71640 1 -- 172.21.15.60:0/679230137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e019f2a0 con 0x7fc6e0107030 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e5770640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6e0101dd0 0x7fc6e0197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:26.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.396+0000 7fc6e4f6f640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6e0104700 0x7fc6e0197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:26.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.397+0000 7fc6e5f71640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6e0107030 0x7fc6e019ec30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8037040 tx=0x7fc6d8032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:26.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.397+0000 7fc6d67fc640 1 -- 172.21.15.60:0/679230137 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8039040 con 0x7fc6e0107030 2024-03-04T12:23:26.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.397+0000 7fc6d67fc640 1 -- 172.21.15.60:0/679230137 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc6d8044070 con 0x7fc6e0107030 2024-03-04T12:23:26.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.397+0000 7fc6e79d4640 1 -- 172.21.15.60:0/679230137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e019f530 con 0x7fc6e0107030 2024-03-04T12:23:26.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.397+0000 7fc6d67fc640 1 -- 172.21.15.60:0/679230137 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8004330 con 0x7fc6e0107030 2024-03-04T12:23:26.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.397+0000 7fc6e79d4640 1 -- 172.21.15.60:0/679230137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e019fa20 con 0x7fc6e0107030 2024-03-04T12:23:26.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.398+0000 7fc6e79d4640 1 -- 172.21.15.60:0/679230137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e00fd2d0 con 0x7fc6e0107030 2024-03-04T12:23:26.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.399+0000 7fc6d67fc640 1 -- 172.21.15.60:0/679230137 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fc6d8003a00 con 0x7fc6e0107030 2024-03-04T12:23:26.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.399+0000 7fc6d67fc640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fc6cc0a1d80 0x7fc6cc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:26.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.399+0000 7fc6e5770640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fc6cc0a1d80 0x7fc6cc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:26.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.399+0000 7fc6d67fc640 1 -- 172.21.15.60:0/679230137 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc6d80f08f0 con 0x7fc6e0107030 2024-03-04T12:23:26.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.400+0000 7fc6e5770640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fc6cc0a1d80 0x7fc6cc0a4240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc0058c0 tx=0x7fc6dc005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:26.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.404+0000 7fc6d67fc640 1 -- 172.21.15.60:0/679230137 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc6d80bdae0 con 0x7fc6e0107030 2024-03-04T12:23:26.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.586+0000 7fc6e79d4640 1 -- 172.21.15.60:0/679230137 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc6e0061f30 con 0x7fc6e0107030 2024-03-04T12:23:26.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.586+0000 7fc6d67fc640 1 -- 172.21.15.60:0/679230137 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc6d80bd3e0 con 0x7fc6e0107030 2024-03-04T12:23:26.592 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:23:26.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.588+0000 7fc6e79d4640 1 -- 172.21.15.60:0/679230137 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fc6cc0a1d80 msgr2=0x7fc6cc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.588+0000 7fc6e79d4640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fc6cc0a1d80 0x7fc6cc0a4240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc0058c0 tx=0x7fc6dc005ec0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.588+0000 7fc6e79d4640 1 -- 172.21.15.60:0/679230137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6e0107030 msgr2=0x7fc6e019ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.588+0000 7fc6e79d4640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6e0107030 0x7fc6e019ec30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8037040 tx=0x7fc6d8032f90 comp rx=0 tx=0).stop 2024-03-04T12:23:26.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.588+0000 7fc6e79d4640 1 -- 172.21.15.60:0/679230137 shutdown_connections 2024-03-04T12:23:26.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.588+0000 7fc6e79d4640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fc6cc0a1d80 0x7fc6cc0a4240 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.588+0000 7fc6e79d4640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc6e0107030 0x7fc6e019ec30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.588+0000 7fc6e79d4640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6e0104700 0x7fc6e0197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.588+0000 7fc6e79d4640 1 --2- 172.21.15.60:0/679230137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6e0101dd0 0x7fc6e0197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.588+0000 7fc6e79d4640 1 -- 172.21.15.60:0/679230137 >> 172.21.15.60:0/679230137 conn(0x7fc6e00fb7e0 msgr2=0x7fc6e0102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:26.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.589+0000 7fc6e79d4640 1 -- 172.21.15.60:0/679230137 shutdown_connections 2024-03-04T12:23:26.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.589+0000 7fc6e79d4640 1 -- 172.21.15.60:0/679230137 wait complete. 2024-03-04T12:23:26.605 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T12:23:26.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.705+0000 7fb840fab640 1 -- 172.21.15.60:0/4147620579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83c102530 msgr2=0x7fb83c1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.706+0000 7fb840fab640 1 --2- 172.21.15.60:0/4147620579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83c102530 0x7fb83c1029b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb830009ca0 tx=0x7fb83002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.706+0000 7fb840fab640 1 -- 172.21.15.60:0/4147620579 shutdown_connections 2024-03-04T12:23:26.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.706+0000 7fb840fab640 1 --2- 172.21.15.60:0/4147620579 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb83c102ef0 0x7fb83c109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.706+0000 7fb840fab640 1 --2- 172.21.15.60:0/4147620579 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83c102530 0x7fb83c1029b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.706+0000 7fb840fab640 1 --2- 172.21.15.60:0/4147620579 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb83c101340 0x7fb83c101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.706+0000 7fb840fab640 1 -- 172.21.15.60:0/4147620579 >> 172.21.15.60:0/4147620579 conn(0x7fb83c0fc8b0 msgr2=0x7fb83c0fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:26.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.706+0000 7fb840fab640 1 -- 172.21.15.60:0/4147620579 shutdown_connections 2024-03-04T12:23:26.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.706+0000 7fb840fab640 1 -- 172.21.15.60:0/4147620579 wait complete. 2024-03-04T12:23:26.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.706+0000 7fb840fab640 1 Processor -- start 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.706+0000 7fb840fab640 1 -- start start 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb840fab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83c101340 0x7fb83c197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb840fab640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb83c102530 0x7fb83c197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb840fab640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb83c102ef0 0x7fb83c19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb840fab640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c10cfd0 con 0x7fb83c101340 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb840fab640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c10ce50 con 0x7fb83c102530 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb840fab640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb83c10d150 con 0x7fb83c102ef0 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb83a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83c101340 0x7fb83c197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb83a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83c101340 0x7fb83c197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52318/0 (socket says 172.21.15.60:52318) 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb83a59c640 1 -- 172.21.15.60:0/3444489538 learned_addr learned my addr 172.21.15.60:0/3444489538 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb839d9b640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb83c102530 0x7fb83c197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb83ad9d640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb83c102ef0 0x7fb83c19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb83a59c640 1 -- 172.21.15.60:0/3444489538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb83c102ef0 msgr2=0x7fb83c19ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb83a59c640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb83c102ef0 0x7fb83c19ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb83a59c640 1 -- 172.21.15.60:0/3444489538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb83c102530 msgr2=0x7fb83c197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb83a59c640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb83c102530 0x7fb83c197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb83a59c640 1 -- 172.21.15.60:0/3444489538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c19f270 con 0x7fb83c101340 2024-03-04T12:23:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb83ad9d640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb83c102ef0 0x7fb83c19ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:23:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb83a59c640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83c101340 0x7fb83c197650 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00d950 tx=0x7fb82c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb82b7fe640 1 -- 172.21.15.60:0/3444489538 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c017020 con 0x7fb83c101340 2024-03-04T12:23:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb82b7fe640 1 -- 172.21.15.60:0/3444489538 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb82c009c30 con 0x7fb83c101340 2024-03-04T12:23:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb840fab640 1 -- 172.21.15.60:0/3444489538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb83c19f560 con 0x7fb83c101340 2024-03-04T12:23:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb82b7fe640 1 -- 172.21.15.60:0/3444489538 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c00f740 con 0x7fb83c101340 2024-03-04T12:23:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.707+0000 7fb840fab640 1 -- 172.21.15.60:0/3444489538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb83c19fa80 con 0x7fb83c101340 2024-03-04T12:23:26.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.708+0000 7fb840fab640 1 -- 172.21.15.60:0/3444489538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb83c108660 con 0x7fb83c101340 2024-03-04T12:23:26.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.709+0000 7fb82b7fe640 1 -- 172.21.15.60:0/3444489538 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fb82c01a020 con 0x7fb83c101340 2024-03-04T12:23:26.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.710+0000 7fb82b7fe640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb8240a1d80 0x7fb8240a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:26.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.710+0000 7fb839d9b640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb8240a1d80 0x7fb8240a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:26.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.710+0000 7fb82b7fe640 1 -- 172.21.15.60:0/3444489538 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb82c0cdaa0 con 0x7fb83c101340 2024-03-04T12:23:26.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.710+0000 7fb839d9b640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb8240a1d80 0x7fb8240a4240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb830006010 tx=0x7fb830032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:26.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.714+0000 7fb82b7fe640 1 -- 172.21.15.60:0/3444489538 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb82c021070 con 0x7fb83c101340 2024-03-04T12:23:26.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.895+0000 7fb840fab640 1 -- 172.21.15.60:0/3444489538 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb83c061f60 con 0x7fb83c101340 2024-03-04T12:23:26.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.895+0000 7fb82b7fe640 1 -- 172.21.15.60:0/3444489538 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb82c009de0 con 0x7fb83c101340 2024-03-04T12:23:26.901 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:23:26.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.897+0000 7fb840fab640 1 -- 172.21.15.60:0/3444489538 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb8240a1d80 msgr2=0x7fb8240a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.897+0000 7fb840fab640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb8240a1d80 0x7fb8240a4240 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb830006010 tx=0x7fb830032590 comp rx=0 tx=0).stop 2024-03-04T12:23:26.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.897+0000 7fb840fab640 1 -- 172.21.15.60:0/3444489538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83c101340 msgr2=0x7fb83c197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:26.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.897+0000 7fb840fab640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83c101340 0x7fb83c197650 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00d950 tx=0x7fb82c00de10 comp rx=0 tx=0).stop 2024-03-04T12:23:26.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.897+0000 7fb840fab640 1 -- 172.21.15.60:0/3444489538 shutdown_connections 2024-03-04T12:23:26.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.897+0000 7fb840fab640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb8240a1d80 0x7fb8240a4240 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.897+0000 7fb840fab640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb83c102ef0 0x7fb83c19ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.897+0000 7fb840fab640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb83c102530 0x7fb83c197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.897+0000 7fb840fab640 1 --2- 172.21.15.60:0/3444489538 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb83c101340 0x7fb83c197650 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:26.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.897+0000 7fb840fab640 1 -- 172.21.15.60:0/3444489538 >> 172.21.15.60:0/3444489538 conn(0x7fb83c0fc8b0 msgr2=0x7fb83c105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:26.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.897+0000 7fb840fab640 1 -- 172.21.15.60:0/3444489538 shutdown_connections 2024-03-04T12:23:26.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:26.897+0000 7fb840fab640 1 -- 172.21.15.60:0/3444489538 wait complete. 2024-03-04T12:23:26.914 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T12:23:27.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.014+0000 7fefa937d640 1 -- 172.21.15.60:0/225648805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4101e00 msgr2=0x7fefa4108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.014+0000 7fefa937d640 1 --2- 172.21.15.60:0/225648805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4101e00 0x7fefa4108690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fef9c009ca0 tx=0x7fef9c02c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.014+0000 7fefa937d640 1 -- 172.21.15.60:0/225648805 shutdown_connections 2024-03-04T12:23:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.014+0000 7fefa937d640 1 --2- 172.21.15.60:0/225648805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4101e00 0x7fefa4108690 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 --2- 172.21.15.60:0/225648805 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa4101440 0x7fefa41018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 --2- 172.21.15.60:0/225648805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefa4100250 0x7fefa4100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 -- 172.21.15.60:0/225648805 >> 172.21.15.60:0/225648805 conn(0x7fefa40fb7e0 msgr2=0x7fefa40fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 -- 172.21.15.60:0/225648805 shutdown_connections 2024-03-04T12:23:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 -- 172.21.15.60:0/225648805 wait complete. 2024-03-04T12:23:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 Processor -- start 2024-03-04T12:23:27.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 -- start start 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa4100250 0x7fefa41964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4101440 0x7fefa41969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefa4101e00 0x7fefa419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa410bee0 con 0x7fefa4101440 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa410bd60 con 0x7fefa4100250 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.015+0000 7fefa937d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa410c060 con 0x7fefa4101e00 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa27fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4101440 0x7fefa41969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa27fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4101440 0x7fefa41969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52328/0 (socket says 172.21.15.60:52328) 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa27fc640 1 -- 172.21.15.60:0/1012146190 learned_addr learned my addr 172.21.15.60:0/1012146190 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa37fe640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefa4101e00 0x7fefa419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa2ffd640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa4100250 0x7fefa41964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa27fc640 1 -- 172.21.15.60:0/1012146190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefa4101e00 msgr2=0x7fefa419da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa27fc640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefa4101e00 0x7fefa419da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa27fc640 1 -- 172.21.15.60:0/1012146190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa4100250 msgr2=0x7fefa41964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa27fc640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa4100250 0x7fefa41964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa27fc640 1 -- 172.21.15.60:0/1012146190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa419e0d0 con 0x7fefa4101440 2024-03-04T12:23:27.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa27fc640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4101440 0x7fefa41969f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fef9800d950 tx=0x7fef9800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:27.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fef8bfff640 1 -- 172.21.15.60:0/1012146190 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef98017070 con 0x7fefa4101440 2024-03-04T12:23:27.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fef8bfff640 1 -- 172.21.15.60:0/1012146190 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef98009c30 con 0x7fefa4101440 2024-03-04T12:23:27.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa937d640 1 -- 172.21.15.60:0/1012146190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa419e3c0 con 0x7fefa4101440 2024-03-04T12:23:27.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fef8bfff640 1 -- 172.21.15.60:0/1012146190 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9800f6b0 con 0x7fefa4101440 2024-03-04T12:23:27.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.016+0000 7fefa937d640 1 -- 172.21.15.60:0/1012146190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa419e910 con 0x7fefa4101440 2024-03-04T12:23:27.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.017+0000 7fefa937d640 1 -- 172.21.15.60:0/1012146190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa4107510 con 0x7fefa4101440 2024-03-04T12:23:27.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.018+0000 7fef8bfff640 1 -- 172.21.15.60:0/1012146190 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fef980109d0 con 0x7fefa4101440 2024-03-04T12:23:27.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.019+0000 7fef8bfff640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fef8c0a1d80 0x7fef8c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:27.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.019+0000 7fef8bfff640 1 -- 172.21.15.60:0/1012146190 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fef980cdc40 con 0x7fefa4101440 2024-03-04T12:23:27.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.019+0000 7fefa2ffd640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fef8c0a1d80 0x7fef8c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:27.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.019+0000 7fefa2ffd640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fef8c0a1d80 0x7fef8c0a4240 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fefa41012a0 tx=0x7fef9400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:27.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.023+0000 7fef8bfff640 1 -- 172.21.15.60:0/1012146190 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fef98009de0 con 0x7fefa4101440 2024-03-04T12:23:27.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:26 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2360830939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:27.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:26 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/679230137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:23:27.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:26 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3444489538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:23:27.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.205+0000 7fefa937d640 1 -- 172.21.15.60:0/1012146190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fefa4061f60 con 0x7fefa4101440 2024-03-04T12:23:27.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.205+0000 7fef8bfff640 1 -- 172.21.15.60:0/1012146190 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fef9809a780 con 0x7fefa4101440 2024-03-04T12:23:27.211 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-03-04T12:23:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.207+0000 7fefa937d640 1 -- 172.21.15.60:0/1012146190 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fef8c0a1d80 msgr2=0x7fef8c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.207+0000 7fefa937d640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fef8c0a1d80 0x7fef8c0a4240 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fefa41012a0 tx=0x7fef9400b390 comp rx=0 tx=0).stop 2024-03-04T12:23:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.207+0000 7fefa937d640 1 -- 172.21.15.60:0/1012146190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4101440 msgr2=0x7fefa41969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.207+0000 7fefa937d640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4101440 0x7fefa41969f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fef9800d950 tx=0x7fef9800de10 comp rx=0 tx=0).stop 2024-03-04T12:23:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.207+0000 7fefa937d640 1 -- 172.21.15.60:0/1012146190 shutdown_connections 2024-03-04T12:23:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.207+0000 7fefa937d640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fef8c0a1d80 0x7fef8c0a4240 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.207+0000 7fefa937d640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefa4101e00 0x7fefa419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.207+0000 7fefa937d640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fefa4101440 0x7fefa41969f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.207+0000 7fefa937d640 1 --2- 172.21.15.60:0/1012146190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa4100250 0x7fefa41964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.207+0000 7fefa937d640 1 -- 172.21.15.60:0/1012146190 >> 172.21.15.60:0/1012146190 conn(0x7fefa40fb7e0 msgr2=0x7fefa41046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:27.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.207+0000 7fefa937d640 1 -- 172.21.15.60:0/1012146190 shutdown_connections 2024-03-04T12:23:27.214 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.207+0000 7fefa937d640 1 -- 172.21.15.60:0/1012146190 wait complete. 2024-03-04T12:23:27.223 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T12:23:27.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.323+0000 7f506be0d640 1 -- 172.21.15.60:0/4246088131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5064100ed0 msgr2=0x7f5064108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.323+0000 7f506be0d640 1 --2- 172.21.15.60:0/4246088131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5064100ed0 0x7f5064108770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f505c009ca0 tx=0x7f505c02c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.323+0000 7f506be0d640 1 -- 172.21.15.60:0/4246088131 shutdown_connections 2024-03-04T12:23:27.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.323+0000 7f506be0d640 1 --2- 172.21.15.60:0/4246088131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5064100ed0 0x7f5064108770 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.323+0000 7f506be0d640 1 --2- 172.21.15.60:0/4246088131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5064100510 0x7f5064100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.323+0000 7f506be0d640 1 --2- 172.21.15.60:0/4246088131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50641017e0 0x7f5064101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.323+0000 7f506be0d640 1 -- 172.21.15.60:0/4246088131 >> 172.21.15.60:0/4246088131 conn(0x7f50640faf40 msgr2=0x7f50640fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:27.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.323+0000 7f506be0d640 1 -- 172.21.15.60:0/4246088131 shutdown_connections 2024-03-04T12:23:27.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.323+0000 7f506be0d640 1 -- 172.21.15.60:0/4246088131 wait complete. 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f506be0d640 1 Processor -- start 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f506be0d640 1 -- start start 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f506be0d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5064100510 0x7f50640fff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f506be0d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5064100ed0 0x7f506419c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f506be0d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50641017e0 0x7f506419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f506be0d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506410bef0 con 0x7f5064100ed0 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f506be0d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506410bd70 con 0x7f50641017e0 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f506be0d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506410c070 con 0x7f5064100510 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f50693a8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5064100ed0 0x7f506419c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f50693a8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5064100ed0 0x7f506419c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52344/0 (socket says 172.21.15.60:52344) 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f50693a8640 1 -- 172.21.15.60:0/2631736038 learned_addr learned my addr 172.21.15.60:0/2631736038 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f506a3aa640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50641017e0 0x7f506419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f5069ba9640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5064100510 0x7f50640fff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f50693a8640 1 -- 172.21.15.60:0/2631736038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5064100510 msgr2=0x7f50640fff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f50693a8640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5064100510 0x7f50640fff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f50693a8640 1 -- 172.21.15.60:0/2631736038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50641017e0 msgr2=0x7f506419e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f50693a8640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50641017e0 0x7f506419e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f50693a8640 1 -- 172.21.15.60:0/2631736038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506419f2c0 con 0x7f5064100ed0 2024-03-04T12:23:27.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f506a3aa640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50641017e0 0x7f506419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:27.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.324+0000 7f50693a8640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5064100ed0 0x7f506419c390 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f505400d950 tx=0x7f505400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:27.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.325+0000 7f505affd640 1 -- 172.21.15.60:0/2631736038 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5054017030 con 0x7f5064100ed0 2024-03-04T12:23:27.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.325+0000 7f505affd640 1 -- 172.21.15.60:0/2631736038 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5054004020 con 0x7f5064100ed0 2024-03-04T12:23:27.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.325+0000 7f506be0d640 1 -- 172.21.15.60:0/2631736038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506419f550 con 0x7f5064100ed0 2024-03-04T12:23:27.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.325+0000 7f505affd640 1 -- 172.21.15.60:0/2631736038 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50540047a0 con 0x7f5064100ed0 2024-03-04T12:23:27.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.325+0000 7f506be0d640 1 -- 172.21.15.60:0/2631736038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506419faa0 con 0x7f5064100ed0 2024-03-04T12:23:27.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.326+0000 7f506be0d640 1 -- 172.21.15.60:0/2631736038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5064107650 con 0x7f5064100ed0 2024-03-04T12:23:27.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.327+0000 7f505affd640 1 -- 172.21.15.60:0/2631736038 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f50540041d0 con 0x7f5064100ed0 2024-03-04T12:23:27.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.327+0000 7f505affd640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f50500a1d80 0x7f50500a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:27.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.327+0000 7f505affd640 1 -- 172.21.15.60:0/2631736038 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f50540cdb30 con 0x7f5064100ed0 2024-03-04T12:23:27.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.328+0000 7f5069ba9640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f50500a1d80 0x7f50500a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:27.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.328+0000 7f5069ba9640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f50500a1d80 0x7f50500a4240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5060005fd0 tx=0x7f5060005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:27.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.332+0000 7f505affd640 1 -- 172.21.15.60:0/2631736038 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f505409ad20 con 0x7f5064100ed0 2024-03-04T12:23:27.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:26 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2360830939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:27.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:26 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/679230137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:23:27.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:26 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3444489538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:23:27.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:26 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2360830939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:27.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:26 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/679230137' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:23:27.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:26 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3444489538' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:23:27.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.513+0000 7f506be0d640 1 -- 172.21.15.60:0/2631736038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5064061f60 con 0x7f5064100ed0 2024-03-04T12:23:27.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.513+0000 7f505affd640 1 -- 172.21.15.60:0/2631736038 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f505409a620 con 0x7f5064100ed0 2024-03-04T12:23:27.519 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:23:27.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.515+0000 7f506be0d640 1 -- 172.21.15.60:0/2631736038 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f50500a1d80 msgr2=0x7f50500a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.515+0000 7f506be0d640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f50500a1d80 0x7f50500a4240 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5060005fd0 tx=0x7f5060005850 comp rx=0 tx=0).stop 2024-03-04T12:23:27.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.515+0000 7f506be0d640 1 -- 172.21.15.60:0/2631736038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5064100ed0 msgr2=0x7f506419c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.515+0000 7f506be0d640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5064100ed0 0x7f506419c390 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f505400d950 tx=0x7f505400de10 comp rx=0 tx=0).stop 2024-03-04T12:23:27.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.515+0000 7f506be0d640 1 -- 172.21.15.60:0/2631736038 shutdown_connections 2024-03-04T12:23:27.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.515+0000 7f506be0d640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f50500a1d80 0x7f50500a4240 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.515+0000 7f506be0d640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50641017e0 0x7f506419e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.515+0000 7f506be0d640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5064100ed0 0x7f506419c390 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.515+0000 7f506be0d640 1 --2- 172.21.15.60:0/2631736038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5064100510 0x7f50640fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.515+0000 7f506be0d640 1 -- 172.21.15.60:0/2631736038 >> 172.21.15.60:0/2631736038 conn(0x7f50640faf40 msgr2=0x7f5064104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:27.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.515+0000 7f506be0d640 1 -- 172.21.15.60:0/2631736038 shutdown_connections 2024-03-04T12:23:27.522 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.515+0000 7f506be0d640 1 -- 172.21.15.60:0/2631736038 wait complete. 2024-03-04T12:23:27.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T12:23:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.629+0000 7f6882859640 1 -- 172.21.15.60:0/3738954781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687c101340 msgr2=0x7f687c101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.629+0000 7f6882859640 1 --2- 172.21.15.60:0/3738954781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687c101340 0x7f687c101760 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6870009900 tx=0x7f687002bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.629+0000 7f6882859640 1 -- 172.21.15.60:0/3738954781 shutdown_connections 2024-03-04T12:23:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.629+0000 7f6882859640 1 --2- 172.21.15.60:0/3738954781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f687c102ef0 0x7f687c109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.629+0000 7f6882859640 1 --2- 172.21.15.60:0/3738954781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c102530 0x7f687c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.629+0000 7f6882859640 1 --2- 172.21.15.60:0/3738954781 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687c101340 0x7f687c101760 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.629+0000 7f6882859640 1 -- 172.21.15.60:0/3738954781 >> 172.21.15.60:0/3738954781 conn(0x7f687c0fc8b0 msgr2=0x7f687c0fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.629+0000 7f6882859640 1 -- 172.21.15.60:0/3738954781 shutdown_connections 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.629+0000 7f6882859640 1 -- 172.21.15.60:0/3738954781 wait complete. 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.630+0000 7f6882859640 1 Processor -- start 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.630+0000 7f6882859640 1 -- start start 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.630+0000 7f6882859640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c101340 0x7f687c197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.630+0000 7f6882859640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f687c102530 0x7f687c197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.630+0000 7f6882859640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687c102ef0 0x7f687c19ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.630+0000 7f6882859640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c10d020 con 0x7f687c102ef0 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.630+0000 7f6882859640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c10cea0 con 0x7f687c102530 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.630+0000 7f6882859640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c10d1a0 con 0x7f687c101340 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.630+0000 7f6880df6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687c102ef0 0x7f687c19ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.630+0000 7f6880df6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687c102ef0 0x7f687c19ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52362/0 (socket says 172.21.15.60:52362) 2024-03-04T12:23:27.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.630+0000 7f6880df6640 1 -- 172.21.15.60:0/1462280991 learned_addr learned my addr 172.21.15.60:0/1462280991 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.630+0000 7f687bfff640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c101340 0x7f687c197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.631+0000 7f687b7fe640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f687c102530 0x7f687c197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.631+0000 7f6880df6640 1 -- 172.21.15.60:0/1462280991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c101340 msgr2=0x7f687c197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.631+0000 7f6880df6640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c101340 0x7f687c197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.631+0000 7f6880df6640 1 -- 172.21.15.60:0/1462280991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f687c102530 msgr2=0x7f687c197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.631+0000 7f6880df6640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f687c102530 0x7f687c197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.631+0000 7f6880df6640 1 -- 172.21.15.60:0/1462280991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c19f290 con 0x7f687c102ef0 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.631+0000 7f6880df6640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687c102ef0 0x7f687c19ec20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f687400eaa0 tx=0x7f687400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.631+0000 7f68797fa640 1 -- 172.21.15.60:0/1462280991 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687400cdb0 con 0x7f687c102ef0 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.631+0000 7f68797fa640 1 -- 172.21.15.60:0/1462280991 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6874005020 con 0x7f687c102ef0 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.631+0000 7f6882859640 1 -- 172.21.15.60:0/1462280991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687c19f580 con 0x7f687c102ef0 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.631+0000 7f68797fa640 1 -- 172.21.15.60:0/1462280991 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68740116b0 con 0x7f687c102ef0 2024-03-04T12:23:27.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.631+0000 7f6882859640 1 -- 172.21.15.60:0/1462280991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f687c19fa80 con 0x7f687c102ef0 2024-03-04T12:23:27.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.632+0000 7f6882859640 1 -- 172.21.15.60:0/1462280991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687c108660 con 0x7f687c102ef0 2024-03-04T12:23:27.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.633+0000 7f68797fa640 1 -- 172.21.15.60:0/1462280991 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f6874005560 con 0x7f687c102ef0 2024-03-04T12:23:27.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.634+0000 7f68797fa640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f68640a1d80 0x7f68640a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:27.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.634+0000 7f687bfff640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f68640a1d80 0x7f68640a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:27.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.634+0000 7f68797fa640 1 -- 172.21.15.60:0/1462280991 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f68740ce1c0 con 0x7f687c102ef0 2024-03-04T12:23:27.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.635+0000 7f687bfff640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f68640a1d80 0x7f68640a4240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f687002c170 tx=0x7f687002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:27.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.639+0000 7f68797fa640 1 -- 172.21.15.60:0/1462280991 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6874016030 con 0x7f687c102ef0 2024-03-04T12:23:27.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.833+0000 7f6882859640 1 -- 172.21.15.60:0/1462280991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f687c061f30 con 0x7f687c102ef0 2024-03-04T12:23:27.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.833+0000 7f68797fa640 1 -- 172.21.15.60:0/1462280991 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+591 (secure 0 0 0) 0x7f687409ad60 con 0x7f687c102ef0 2024-03-04T12:23:27.839 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/2852203003 2024-03-05T12:16:04.073665+0000 2024-03-04T12:23:27.839 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/3868523955 2024-03-05T12:16:04.073665+0000 2024-03-04T12:23:27.839 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6801/2246601804 2024-03-05T12:16:04.073665+0000 2024-03-04T12:23:27.839 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6800/2246601804 2024-03-05T12:16:04.073665+0000 2024-03-04T12:23:27.839 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/758894979 2024-03-05T12:15:37.282929+0000 2024-03-04T12:23:27.839 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6801/2061680026 2024-03-05T12:15:37.282929+0000 2024-03-04T12:23:27.839 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/3401854556 2024-03-05T12:15:37.282929+0000 2024-03-04T12:23:27.839 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6800/2061680026 2024-03-05T12:15:37.282929+0000 2024-03-04T12:23:27.840 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/3539227088 2024-03-05T12:15:37.282929+0000 2024-03-04T12:23:27.840 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/2028967685 2024-03-05T12:16:04.073665+0000 2024-03-04T12:23:27.840 INFO:teuthology.orchestra.run.smithi060.stderr:listed 10 entries 2024-03-04T12:23:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.835+0000 7f6882859640 1 -- 172.21.15.60:0/1462280991 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f68640a1d80 msgr2=0x7f68640a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.835+0000 7f6882859640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f68640a1d80 0x7f68640a4240 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f687002c170 tx=0x7f687002df90 comp rx=0 tx=0).stop 2024-03-04T12:23:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.835+0000 7f6882859640 1 -- 172.21.15.60:0/1462280991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687c102ef0 msgr2=0x7f687c19ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.835+0000 7f6882859640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687c102ef0 0x7f687c19ec20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f687400eaa0 tx=0x7f687400ef60 comp rx=0 tx=0).stop 2024-03-04T12:23:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.835+0000 7f6882859640 1 -- 172.21.15.60:0/1462280991 shutdown_connections 2024-03-04T12:23:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.835+0000 7f6882859640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f68640a1d80 0x7f68640a4240 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.835+0000 7f6882859640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f687c102ef0 0x7f687c19ec20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.835+0000 7f6882859640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f687c102530 0x7f687c197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.835+0000 7f6882859640 1 --2- 172.21.15.60:0/1462280991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f687c101340 0x7f687c197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.835+0000 7f6882859640 1 -- 172.21.15.60:0/1462280991 >> 172.21.15.60:0/1462280991 conn(0x7f687c0fc8b0 msgr2=0x7f687c105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.835+0000 7f6882859640 1 -- 172.21.15.60:0/1462280991 shutdown_connections 2024-03-04T12:23:27.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:27.835+0000 7f6882859640 1 -- 172.21.15.60:0/1462280991 wait complete. 2024-03-04T12:23:27.852 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:23:27.852 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T12:23:27.868 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:23:28.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.006+0000 7f99b6259640 1 -- 172.21.15.60:0/3948779001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01025b0 msgr2=0x7f99b0102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.006+0000 7f99b6259640 1 --2- 172.21.15.60:0/3948779001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01025b0 0x7f99b0102a30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009900 tx=0x7f99a402bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:28.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.006+0000 7f99b6259640 1 -- 172.21.15.60:0/3948779001 shutdown_connections 2024-03-04T12:23:28.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.006+0000 7f99b6259640 1 --2- 172.21.15.60:0/3948779001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99b0102f70 0x7f99b0109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.006+0000 7f99b6259640 1 --2- 172.21.15.60:0/3948779001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01025b0 0x7f99b0102a30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.006+0000 7f99b6259640 1 --2- 172.21.15.60:0/3948779001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99b01013c0 0x7f99b01017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.006+0000 7f99b6259640 1 -- 172.21.15.60:0/3948779001 >> 172.21.15.60:0/3948779001 conn(0x7f99b00fc950 msgr2=0x7f99b00fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.006+0000 7f99b6259640 1 -- 172.21.15.60:0/3948779001 shutdown_connections 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.006+0000 7f99b6259640 1 -- 172.21.15.60:0/3948779001 wait complete. 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.007+0000 7f99b6259640 1 Processor -- start 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.007+0000 7f99b6259640 1 -- start start 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.007+0000 7f99b6259640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01013c0 0x7f99b00750e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.007+0000 7f99b6259640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99b01025b0 0x7f99b0073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.007+0000 7f99b6259640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99b0102f70 0x7f99b0073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.007+0000 7f99b6259640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b010cfe0 con 0x7f99b01013c0 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.007+0000 7f99b6259640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b010ce60 con 0x7f99b01025b0 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.007+0000 7f99b6259640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b010d160 con 0x7f99b0102f70 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.007+0000 7f99af7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01013c0 0x7f99b00750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.007+0000 7f99af7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01013c0 0x7f99b00750e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52368/0 (socket says 172.21.15.60:52368) 2024-03-04T12:23:28.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.007+0000 7f99af7fe640 1 -- 172.21.15.60:0/1023776490 learned_addr learned my addr 172.21.15.60:0/1023776490 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.007+0000 7f99aeffd640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99b01025b0 0x7f99b0073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.008+0000 7f99affff640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99b0102f70 0x7f99b0073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.008+0000 7f99af7fe640 1 -- 172.21.15.60:0/1023776490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99b0102f70 msgr2=0x7f99b0073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.008+0000 7f99af7fe640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99b0102f70 0x7f99b0073c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.008+0000 7f99af7fe640 1 -- 172.21.15.60:0/1023776490 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99b01025b0 msgr2=0x7f99b0073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.008+0000 7f99af7fe640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99b01025b0 0x7f99b0073720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.008+0000 7f99af7fe640 1 -- 172.21.15.60:0/1023776490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b00744f0 con 0x7f99b01013c0 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.008+0000 7f99af7fe640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01013c0 0x7f99b00750e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f99a000c970 tx=0x7f99a000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.008+0000 7f99acff9640 1 -- 172.21.15.60:0/1023776490 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0004220 con 0x7f99b01013c0 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.008+0000 7f99acff9640 1 -- 172.21.15.60:0/1023776490 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99a00043c0 con 0x7f99b01013c0 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.008+0000 7f99b6259640 1 -- 172.21.15.60:0/1023776490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b01a1790 con 0x7f99b01013c0 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.008+0000 7f99acff9640 1 -- 172.21.15.60:0/1023776490 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a000f450 con 0x7f99b01013c0 2024-03-04T12:23:28.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.008+0000 7f99b6259640 1 -- 172.21.15.60:0/1023776490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b01a1cb0 con 0x7f99b01013c0 2024-03-04T12:23:28.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.009+0000 7f99b6259640 1 -- 172.21.15.60:0/1023776490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b0068600 con 0x7f99b01013c0 2024-03-04T12:23:28.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.010+0000 7f99acff9640 1 -- 172.21.15.60:0/1023776490 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f99a00079b0 con 0x7f99b01013c0 2024-03-04T12:23:28.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.011+0000 7f99acff9640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99980a1d80 0x7f99980a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.011+0000 7f99aeffd640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99980a1d80 0x7f99980a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.011+0000 7f99acff9640 1 -- 172.21.15.60:0/1023776490 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f99a00cdda0 con 0x7f99b01013c0 2024-03-04T12:23:28.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.011+0000 7f99aeffd640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99980a1d80 0x7f99980a4240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009230 tx=0x7f99a4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:28.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.015+0000 7f99acff9640 1 -- 172.21.15.60:0/1023776490 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f99a0014030 con 0x7f99b01013c0 2024-03-04T12:23:28.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:27 smithi060 ceph-mon[32745]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-04T12:23:28.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:27 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1012146190' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:23:28.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:27 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2631736038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:23:28.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:27 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1462280991' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T12:23:28.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.198+0000 7f99b6259640 1 -- 172.21.15.60:0/1023776490 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99b0061f30 con 0x7f99b01013c0 2024-03-04T12:23:28.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.200+0000 7f99acff9640 1 -- 172.21.15.60:0/1023776490 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23997 (secure 0 0 0) 0x7f99a009af90 con 0x7f99b01013c0 2024-03-04T12:23:28.206 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:28.206 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":89,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:23:17.289925+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:23:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.202+0000 7f99b6259640 1 -- 172.21.15.60:0/1023776490 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99980a1d80 msgr2=0x7f99980a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.202+0000 7f99b6259640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99980a1d80 0x7f99980a4240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f99a4009230 tx=0x7f99a4005fd0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.202+0000 7f99b6259640 1 -- 172.21.15.60:0/1023776490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01013c0 msgr2=0x7f99b00750e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.202+0000 7f99b6259640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01013c0 0x7f99b00750e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f99a000c970 tx=0x7f99a000ce30 comp rx=0 tx=0).stop 2024-03-04T12:23:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.202+0000 7f99b6259640 1 -- 172.21.15.60:0/1023776490 shutdown_connections 2024-03-04T12:23:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.202+0000 7f99b6259640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99980a1d80 0x7f99980a4240 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.202+0000 7f99b6259640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99b0102f70 0x7f99b0073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.202+0000 7f99b6259640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99b01025b0 0x7f99b0073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.202+0000 7f99b6259640 1 --2- 172.21.15.60:0/1023776490 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99b01013c0 0x7f99b00750e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.202+0000 7f99b6259640 1 -- 172.21.15.60:0/1023776490 >> 172.21.15.60:0/1023776490 conn(0x7f99b00fc950 msgr2=0x7f99b0105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.202+0000 7f99b6259640 1 -- 172.21.15.60:0/1023776490 shutdown_connections 2024-03-04T12:23:28.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.202+0000 7f99b6259640 1 -- 172.21.15.60:0/1023776490 wait complete. 2024-03-04T12:23:28.219 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T12:23:10.438928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T12:23:11.472998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T12:23:15.162500+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T12:23:28.219 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:23:28.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.319+0000 7f12b188c640 1 -- 172.21.15.60:0/1377061669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12ac102550 msgr2=0x7f12ac1029d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.319+0000 7f12b188c640 1 --2- 172.21.15.60:0/1377061669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12ac102550 0x7f12ac1029d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1294009900 tx=0x7f129402bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:28.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.319+0000 7f12b188c640 1 -- 172.21.15.60:0/1377061669 shutdown_connections 2024-03-04T12:23:28.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.319+0000 7f12b188c640 1 --2- 172.21.15.60:0/1377061669 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12ac102f10 0x7f12ac1097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.319+0000 7f12b188c640 1 --2- 172.21.15.60:0/1377061669 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12ac102550 0x7f12ac1029d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.319+0000 7f12b188c640 1 --2- 172.21.15.60:0/1377061669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac101360 0x7f12ac101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.319+0000 7f12b188c640 1 -- 172.21.15.60:0/1377061669 >> 172.21.15.60:0/1377061669 conn(0x7f12ac0fc8f0 msgr2=0x7f12ac0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:28.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.319+0000 7f12b188c640 1 -- 172.21.15.60:0/1377061669 shutdown_connections 2024-03-04T12:23:28.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.319+0000 7f12b188c640 1 -- 172.21.15.60:0/1377061669 wait complete. 2024-03-04T12:23:28.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.319+0000 7f12b188c640 1 Processor -- start 2024-03-04T12:23:28.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12b188c640 1 -- start start 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12b188c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac101360 0x7f12ac197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12b188c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12ac102550 0x7f12ac197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12b188c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12ac102f10 0x7f12ac19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12b188c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac10cff0 con 0x7f12ac102f10 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12b188c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac10ce70 con 0x7f12ac102550 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12b188c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac10d170 con 0x7f12ac101360 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12ab7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12ac102f10 0x7f12ac19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12ab7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12ac102f10 0x7f12ac19ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52376/0 (socket says 172.21.15.60:52376) 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12ab7fe640 1 -- 172.21.15.60:0/3239910060 learned_addr learned my addr 172.21.15.60:0/3239910060 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12aa7fc640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12ac102550 0x7f12ac197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12aaffd640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac101360 0x7f12ac197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12ab7fe640 1 -- 172.21.15.60:0/3239910060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac101360 msgr2=0x7f12ac197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12ab7fe640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac101360 0x7f12ac197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12ab7fe640 1 -- 172.21.15.60:0/3239910060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12ac102550 msgr2=0x7f12ac197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12ab7fe640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12ac102550 0x7f12ac197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12ab7fe640 1 -- 172.21.15.60:0/3239910060 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac19f260 con 0x7f12ac102f10 2024-03-04T12:23:28.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.320+0000 7f12ab7fe640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12ac102f10 0x7f12ac19ebf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f12a400d750 tx=0x7f12a400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:28.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.321+0000 7f12b088a640 1 -- 172.21.15.60:0/3239910060 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4017070 con 0x7f12ac102f10 2024-03-04T12:23:28.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.321+0000 7f12b088a640 1 -- 172.21.15.60:0/3239910060 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12a4005500 con 0x7f12ac102f10 2024-03-04T12:23:28.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.321+0000 7f12b188c640 1 -- 172.21.15.60:0/3239910060 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ac19f550 con 0x7f12ac102f10 2024-03-04T12:23:28.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.321+0000 7f12b088a640 1 -- 172.21.15.60:0/3239910060 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4004d90 con 0x7f12ac102f10 2024-03-04T12:23:28.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.321+0000 7f12b188c640 1 -- 172.21.15.60:0/3239910060 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ac19fa70 con 0x7f12ac102f10 2024-03-04T12:23:28.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.322+0000 7f12b188c640 1 -- 172.21.15.60:0/3239910060 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ac108680 con 0x7f12ac102f10 2024-03-04T12:23:28.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.323+0000 7f12b088a640 1 -- 172.21.15.60:0/3239910060 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f12a4004f30 con 0x7f12ac102f10 2024-03-04T12:23:28.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.323+0000 7f12b088a640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f12980a1e00 0x7f12980a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.323+0000 7f12b088a640 1 -- 172.21.15.60:0/3239910060 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f12a40cd8f0 con 0x7f12ac102f10 2024-03-04T12:23:28.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.323+0000 7f12aaffd640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f12980a1e00 0x7f12980a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.324+0000 7f12aaffd640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f12980a1e00 0x7f12980a42c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f129c005c40 tx=0x7f129c00a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:28.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.328+0000 7f12b088a640 1 -- 172.21.15.60:0/3239910060 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f12a409a490 con 0x7f12ac102f10 2024-03-04T12:23:28.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:27 smithi086 ceph-mon[36851]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-04T12:23:28.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:27 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1012146190' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:23:28.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:27 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2631736038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:23:28.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:27 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1462280991' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T12:23:28.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:27 smithi063 ceph-mon[36717]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-04T12:23:28.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:27 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1012146190' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:23:28.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:27 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2631736038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:23:28.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:27 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1462280991' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T12:23:28.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.512+0000 7f12b188c640 1 -- 172.21.15.60:0/3239910060 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f12ac061f30 con 0x7f12ac102f10 2024-03-04T12:23:28.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.512+0000 7f12b088a640 1 -- 172.21.15.60:0/3239910060 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f12a409e530 con 0x7f12ac102f10 2024-03-04T12:23:28.518 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:23:28.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.514+0000 7f12b188c640 1 -- 172.21.15.60:0/3239910060 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f12980a1e00 msgr2=0x7f12980a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.514+0000 7f12b188c640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f12980a1e00 0x7f12980a42c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f129c005c40 tx=0x7f129c00a590 comp rx=0 tx=0).stop 2024-03-04T12:23:28.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.514+0000 7f12b188c640 1 -- 172.21.15.60:0/3239910060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12ac102f10 msgr2=0x7f12ac19ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.514+0000 7f12b188c640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12ac102f10 0x7f12ac19ebf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f12a400d750 tx=0x7f12a400dc10 comp rx=0 tx=0).stop 2024-03-04T12:23:28.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.514+0000 7f12b188c640 1 -- 172.21.15.60:0/3239910060 shutdown_connections 2024-03-04T12:23:28.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.514+0000 7f12b188c640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f12980a1e00 0x7f12980a42c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.514+0000 7f12b188c640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f12ac102f10 0x7f12ac19ebf0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.514+0000 7f12b188c640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12ac102550 0x7f12ac197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.514+0000 7f12b188c640 1 --2- 172.21.15.60:0/3239910060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f12ac101360 0x7f12ac197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.514+0000 7f12b188c640 1 -- 172.21.15.60:0/3239910060 >> 172.21.15.60:0/3239910060 conn(0x7f12ac0fc8f0 msgr2=0x7f12ac105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:28.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.514+0000 7f12b188c640 1 -- 172.21.15.60:0/3239910060 shutdown_connections 2024-03-04T12:23:28.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.514+0000 7f12b188c640 1 -- 172.21.15.60:0/3239910060 wait complete. 2024-03-04T12:23:28.531 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T12:23:28.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.631+0000 7f98f12d1640 1 -- 172.21.15.60:0/3895993480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98ec101e00 msgr2=0x7f98ec108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.631+0000 7f98f12d1640 1 --2- 172.21.15.60:0/3895993480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98ec101e00 0x7f98ec108690 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f98e4009ca0 tx=0x7f98e402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.631+0000 7f98f12d1640 1 -- 172.21.15.60:0/3895993480 shutdown_connections 2024-03-04T12:23:28.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.631+0000 7f98f12d1640 1 --2- 172.21.15.60:0/3895993480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98ec101e00 0x7f98ec108690 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.631+0000 7f98f12d1640 1 --2- 172.21.15.60:0/3895993480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec101440 0x7f98ec1018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.631+0000 7f98f12d1640 1 --2- 172.21.15.60:0/3895993480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98ec100250 0x7f98ec100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.631+0000 7f98f12d1640 1 -- 172.21.15.60:0/3895993480 >> 172.21.15.60:0/3895993480 conn(0x7f98ec0fb7c0 msgr2=0x7f98ec0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.632+0000 7f98f12d1640 1 -- 172.21.15.60:0/3895993480 shutdown_connections 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.632+0000 7f98f12d1640 1 -- 172.21.15.60:0/3895993480 wait complete. 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.632+0000 7f98f12d1640 1 Processor -- start 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.632+0000 7f98f12d1640 1 -- start start 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.632+0000 7f98f12d1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98ec100250 0x7f98ec197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.632+0000 7f98f12d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98ec101440 0x7f98ec197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.632+0000 7f98f12d1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec101e00 0x7f98ec19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.632+0000 7f98f12d1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec10beb0 con 0x7f98ec101440 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.632+0000 7f98f12d1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec10bd30 con 0x7f98ec101e00 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.632+0000 7f98f12d1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec10c030 con 0x7f98ec100250 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.632+0000 7f98ea7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98ec101440 0x7f98ec197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98ea7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98ec101440 0x7f98ec197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52398/0 (socket says 172.21.15.60:52398) 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98ea7fc640 1 -- 172.21.15.60:0/695601730 learned_addr learned my addr 172.21.15.60:0/695601730 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98eb7fe640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec101e00 0x7f98ec19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98eaffd640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98ec100250 0x7f98ec197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98ea7fc640 1 -- 172.21.15.60:0/695601730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98ec100250 msgr2=0x7f98ec197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98ea7fc640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98ec100250 0x7f98ec197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98ea7fc640 1 -- 172.21.15.60:0/695601730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec101e00 msgr2=0x7f98ec19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98ea7fc640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec101e00 0x7f98ec19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98ea7fc640 1 -- 172.21.15.60:0/695601730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ec19f240 con 0x7f98ec101440 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98eaffd640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98ec100250 0x7f98ec197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98eb7fe640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec101e00 0x7f98ec19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98ea7fc640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98ec101440 0x7f98ec197b60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f98e000d950 tx=0x7f98e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98d3fff640 1 -- 172.21.15.60:0/695601730 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e0017070 con 0x7f98ec101440 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98d3fff640 1 -- 172.21.15.60:0/695601730 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98e0009c30 con 0x7f98ec101440 2024-03-04T12:23:28.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98f12d1640 1 -- 172.21.15.60:0/695601730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ec19f530 con 0x7f98ec101440 2024-03-04T12:23:28.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98d3fff640 1 -- 172.21.15.60:0/695601730 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e000f6b0 con 0x7f98ec101440 2024-03-04T12:23:28.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.633+0000 7f98f12d1640 1 -- 172.21.15.60:0/695601730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ec19fa80 con 0x7f98ec101440 2024-03-04T12:23:28.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.634+0000 7f98f12d1640 1 -- 172.21.15.60:0/695601730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ec19fce0 con 0x7f98ec101440 2024-03-04T12:23:28.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.635+0000 7f98d3fff640 1 -- 172.21.15.60:0/695601730 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f98e00109d0 con 0x7f98ec101440 2024-03-04T12:23:28.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.636+0000 7f98d3fff640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f98d40a1d80 0x7f98d40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.636+0000 7f98eaffd640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f98d40a1d80 0x7f98d40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.636+0000 7f98d3fff640 1 -- 172.21.15.60:0/695601730 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f98e00cdc60 con 0x7f98ec101440 2024-03-04T12:23:28.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.636+0000 7f98eaffd640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f98d40a1d80 0x7f98d40a4240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f98dc005d00 tx=0x7f98dc005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:28.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.639+0000 7f98d3fff640 1 -- 172.21.15.60:0/695601730 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f98e0009de0 con 0x7f98ec101440 2024-03-04T12:23:28.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.821+0000 7f98f12d1640 1 -- 172.21.15.60:0/695601730 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f98ec1a03b0 con 0x7f98ec101440 2024-03-04T12:23:28.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.821+0000 7f98d3fff640 1 -- 172.21.15.60:0/695601730 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f98e009ae50 con 0x7f98ec101440 2024-03-04T12:23:28.827 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:23:28.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.823+0000 7f98f12d1640 1 -- 172.21.15.60:0/695601730 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f98d40a1d80 msgr2=0x7f98d40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.823+0000 7f98f12d1640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f98d40a1d80 0x7f98d40a4240 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f98dc005d00 tx=0x7f98dc005c90 comp rx=0 tx=0).stop 2024-03-04T12:23:28.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.823+0000 7f98f12d1640 1 -- 172.21.15.60:0/695601730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98ec101440 msgr2=0x7f98ec197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.823+0000 7f98f12d1640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98ec101440 0x7f98ec197b60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f98e000d950 tx=0x7f98e000de10 comp rx=0 tx=0).stop 2024-03-04T12:23:28.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.823+0000 7f98f12d1640 1 -- 172.21.15.60:0/695601730 shutdown_connections 2024-03-04T12:23:28.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.823+0000 7f98f12d1640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f98d40a1d80 0x7f98d40a4240 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.823+0000 7f98f12d1640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f98ec101e00 0x7f98ec19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.823+0000 7f98f12d1640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f98ec101440 0x7f98ec197b60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.823+0000 7f98f12d1640 1 --2- 172.21.15.60:0/695601730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98ec100250 0x7f98ec197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.823+0000 7f98f12d1640 1 -- 172.21.15.60:0/695601730 >> 172.21.15.60:0/695601730 conn(0x7f98ec0fb7c0 msgr2=0x7f98ec0699d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:28.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.823+0000 7f98f12d1640 1 -- 172.21.15.60:0/695601730 shutdown_connections 2024-03-04T12:23:28.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.823+0000 7f98f12d1640 1 -- 172.21.15.60:0/695601730 wait complete. 2024-03-04T12:23:28.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T12:23:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.942+0000 7fb5b340b640 1 -- 172.21.15.60:0/1457191663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5ac102ee0 msgr2=0x7fb5ac109770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.942+0000 7fb5b340b640 1 --2- 172.21.15.60:0/1457191663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5ac102ee0 0x7fb5ac109770 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009ca0 tx=0x7fb5a402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.942+0000 7fb5b340b640 1 -- 172.21.15.60:0/1457191663 shutdown_connections 2024-03-04T12:23:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.942+0000 7fb5b340b640 1 --2- 172.21.15.60:0/1457191663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5ac102ee0 0x7fb5ac109770 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.942+0000 7fb5b340b640 1 --2- 172.21.15.60:0/1457191663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5ac102520 0x7fb5ac1029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.942+0000 7fb5b340b640 1 --2- 172.21.15.60:0/1457191663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac101330 0x7fb5ac101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.942+0000 7fb5b340b640 1 -- 172.21.15.60:0/1457191663 >> 172.21.15.60:0/1457191663 conn(0x7fb5ac0fc880 msgr2=0x7fb5ac0fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.942+0000 7fb5b340b640 1 -- 172.21.15.60:0/1457191663 shutdown_connections 2024-03-04T12:23:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.942+0000 7fb5b340b640 1 -- 172.21.15.60:0/1457191663 wait complete. 2024-03-04T12:23:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b340b640 1 Processor -- start 2024-03-04T12:23:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b340b640 1 -- start start 2024-03-04T12:23:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b340b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac101330 0x7fb5ac075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b340b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5ac102520 0x7fb5ac073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b340b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5ac102ee0 0x7fb5ac073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b340b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac10d020 con 0x7fb5ac102520 2024-03-04T12:23:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b340b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac10cea0 con 0x7fb5ac102ee0 2024-03-04T12:23:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b340b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac10d1a0 con 0x7fb5ac101330 2024-03-04T12:23:28.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b09a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5ac102520 0x7fb5ac073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b09a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5ac102520 0x7fb5ac073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52418/0 (socket says 172.21.15.60:52418) 2024-03-04T12:23:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b09a6640 1 -- 172.21.15.60:0/3097867085 learned_addr learned my addr 172.21.15.60:0/3097867085 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b19a8640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5ac102ee0 0x7fb5ac073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b11a7640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac101330 0x7fb5ac075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b09a6640 1 -- 172.21.15.60:0/3097867085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac101330 msgr2=0x7fb5ac075090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b09a6640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac101330 0x7fb5ac075090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b09a6640 1 -- 172.21.15.60:0/3097867085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5ac102ee0 msgr2=0x7fb5ac073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b09a6640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5ac102ee0 0x7fb5ac073c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:28.952 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b09a6640 1 -- 172.21.15.60:0/3097867085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac074520 con 0x7fb5ac102520 2024-03-04T12:23:28.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5b09a6640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5ac102520 0x7fb5ac073720 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00c890 tx=0x7fb59c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:28.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.943+0000 7fb5a27fc640 1 -- 172.21.15.60:0/3097867085 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c004220 con 0x7fb5ac102520 2024-03-04T12:23:28.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.944+0000 7fb5a27fc640 1 -- 172.21.15.60:0/3097867085 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb59c0043c0 con 0x7fb5ac102520 2024-03-04T12:23:28.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.944+0000 7fb5b340b640 1 -- 172.21.15.60:0/3097867085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ac1a17f0 con 0x7fb5ac102520 2024-03-04T12:23:28.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.944+0000 7fb5a27fc640 1 -- 172.21.15.60:0/3097867085 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c0049e0 con 0x7fb5ac102520 2024-03-04T12:23:28.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.944+0000 7fb5b340b640 1 -- 172.21.15.60:0/3097867085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac1a1d10 con 0x7fb5ac102520 2024-03-04T12:23:28.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.944+0000 7fb5b340b640 1 -- 172.21.15.60:0/3097867085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ac068600 con 0x7fb5ac102520 2024-03-04T12:23:28.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.946+0000 7fb5a27fc640 1 -- 172.21.15.60:0/3097867085 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fb59c0079b0 con 0x7fb5ac102520 2024-03-04T12:23:28.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.946+0000 7fb5a27fc640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb5980a1d80 0x7fb5980a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:28.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.946+0000 7fb5b11a7640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb5980a1d80 0x7fb5980a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:28.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.946+0000 7fb5a27fc640 1 -- 172.21.15.60:0/3097867085 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb59c0ce000 con 0x7fb5ac102520 2024-03-04T12:23:28.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.947+0000 7fb5b11a7640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb5980a1d80 0x7fb5980a4240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac102380 tx=0x7fb5a8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:28.956 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:28.951+0000 7fb5a27fc640 1 -- 172.21.15.60:0/3097867085 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb59c014030 con 0x7fb5ac102520 2024-03-04T12:23:29.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:28 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1023776490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:29.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:28 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3239910060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:23:29.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:28 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/695601730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:23:29.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.133+0000 7fb5b340b640 1 -- 172.21.15.60:0/3097867085 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb5ac061f30 con 0x7fb5ac102520 2024-03-04T12:23:29.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.134+0000 7fb5a27fc640 1 -- 172.21.15.60:0/3097867085 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb59c09b1f0 con 0x7fb5ac102520 2024-03-04T12:23:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-03-04T12:23:29.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.136+0000 7fb5b340b640 1 -- 172.21.15.60:0/3097867085 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb5980a1d80 msgr2=0x7fb5980a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:29.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.136+0000 7fb5b340b640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb5980a1d80 0x7fb5980a4240 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac102380 tx=0x7fb5a8008040 comp rx=0 tx=0).stop 2024-03-04T12:23:29.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.136+0000 7fb5b340b640 1 -- 172.21.15.60:0/3097867085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5ac102520 msgr2=0x7fb5ac073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:29.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.136+0000 7fb5b340b640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5ac102520 0x7fb5ac073720 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00c890 tx=0x7fb59c00cd50 comp rx=0 tx=0).stop 2024-03-04T12:23:29.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.136+0000 7fb5b340b640 1 -- 172.21.15.60:0/3097867085 shutdown_connections 2024-03-04T12:23:29.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.136+0000 7fb5b340b640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb5980a1d80 0x7fb5980a4240 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.136+0000 7fb5b340b640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb5ac102ee0 0x7fb5ac073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.136+0000 7fb5b340b640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb5ac102520 0x7fb5ac073720 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.136+0000 7fb5b340b640 1 --2- 172.21.15.60:0/3097867085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5ac101330 0x7fb5ac075090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.136+0000 7fb5b340b640 1 -- 172.21.15.60:0/3097867085 >> 172.21.15.60:0/3097867085 conn(0x7fb5ac0fc880 msgr2=0x7fb5ac1058a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:29.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.136+0000 7fb5b340b640 1 -- 172.21.15.60:0/3097867085 shutdown_connections 2024-03-04T12:23:29.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.136+0000 7fb5b340b640 1 -- 172.21.15.60:0/3097867085 wait complete. 2024-03-04T12:23:29.151 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:28 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1023776490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:29.151 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:28 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3239910060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:23:29.151 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:28 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/695601730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:23:29.152 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T12:23:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.252+0000 7fb0534d7640 1 -- 172.21.15.60:0/1842163908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb04c1070d0 msgr2=0x7fb04c1094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.252+0000 7fb0534d7640 1 --2- 172.21.15.60:0/1842163908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb04c1070d0 0x7fb04c1094c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb048009ca0 tx=0x7fb04802c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.252+0000 7fb0534d7640 1 -- 172.21.15.60:0/1842163908 shutdown_connections 2024-03-04T12:23:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.252+0000 7fb0534d7640 1 --2- 172.21.15.60:0/1842163908 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb04c1070d0 0x7fb04c1094c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.252+0000 7fb0534d7640 1 --2- 172.21.15.60:0/1842163908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c1047a0 0x7fb04c106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.252+0000 7fb0534d7640 1 --2- 172.21.15.60:0/1842163908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb04c101e70 0x7fb04c104260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.252+0000 7fb0534d7640 1 -- 172.21.15.60:0/1842163908 >> 172.21.15.60:0/1842163908 conn(0x7fb04c0fb7c0 msgr2=0x7fb04c0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb0534d7640 1 -- 172.21.15.60:0/1842163908 shutdown_connections 2024-03-04T12:23:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb0534d7640 1 -- 172.21.15.60:0/1842163908 wait complete. 2024-03-04T12:23:29.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb0534d7640 1 Processor -- start 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb0534d7640 1 -- start start 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb0534d7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb04c101e70 0x7fb04c1017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb0534d7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c1047a0 0x7fb04c0ffe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb0534d7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb04c1070d0 0x7fb04c100360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb0534d7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c10d000 con 0x7fb04c1070d0 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb0534d7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c10ce80 con 0x7fb04c1047a0 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb0534d7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c10d180 con 0x7fb04c101e70 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb051a74640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb04c1070d0 0x7fb04c100360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb051a74640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb04c1070d0 0x7fb04c100360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52430/0 (socket says 172.21.15.60:52430) 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.253+0000 7fb051a74640 1 -- 172.21.15.60:0/1431993935 learned_addr learned my addr 172.21.15.60:0/1431993935 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb051273640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb04c101e70 0x7fb04c1017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb050a72640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c1047a0 0x7fb04c0ffe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb051a74640 1 -- 172.21.15.60:0/1431993935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb04c101e70 msgr2=0x7fb04c1017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:29.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb051a74640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb04c101e70 0x7fb04c1017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb051a74640 1 -- 172.21.15.60:0/1431993935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c1047a0 msgr2=0x7fb04c0ffe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb051a74640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c1047a0 0x7fb04c0ffe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb051a74640 1 -- 172.21.15.60:0/1431993935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c100bf0 con 0x7fb04c1070d0 2024-03-04T12:23:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb050a72640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c1047a0 0x7fb04c0ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:23:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb051a74640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb04c1070d0 0x7fb04c100360 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb048037040 tx=0x7fb048032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb0427fc640 1 -- 172.21.15.60:0/1431993935 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb048039040 con 0x7fb04c1070d0 2024-03-04T12:23:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb0427fc640 1 -- 172.21.15.60:0/1431993935 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb048044070 con 0x7fb04c1070d0 2024-03-04T12:23:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb0534d7640 1 -- 172.21.15.60:0/1431993935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04c1a1630 con 0x7fb04c1070d0 2024-03-04T12:23:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb0427fc640 1 -- 172.21.15.60:0/1431993935 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb048004330 con 0x7fb04c1070d0 2024-03-04T12:23:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.254+0000 7fb0534d7640 1 -- 172.21.15.60:0/1431993935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04c1a1b80 con 0x7fb04c1070d0 2024-03-04T12:23:29.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.255+0000 7fb0534d7640 1 -- 172.21.15.60:0/1431993935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04c0fd2b0 con 0x7fb04c1070d0 2024-03-04T12:23:29.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.256+0000 7fb0427fc640 1 -- 172.21.15.60:0/1431993935 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fb048003a00 con 0x7fb04c1070d0 2024-03-04T12:23:29.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.257+0000 7fb0427fc640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb0380a1d80 0x7fb0380a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:29.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.257+0000 7fb051273640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb0380a1d80 0x7fb0380a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:29.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.257+0000 7fb0427fc640 1 -- 172.21.15.60:0/1431993935 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb0480f0ee0 con 0x7fb04c1070d0 2024-03-04T12:23:29.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.257+0000 7fb051273640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb0380a1d80 0x7fb0380a4240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb0440058c0 tx=0x7fb04400a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:29.266 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.260+0000 7fb0427fc640 1 -- 172.21.15.60:0/1431993935 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb0480be1f0 con 0x7fb04c1070d0 2024-03-04T12:23:29.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:28 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1023776490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:29.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:28 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3239910060' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:23:29.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:28 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/695601730' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:23:29.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.441+0000 7fb0534d7640 1 -- 172.21.15.60:0/1431993935 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb04c02ceb0 con 0x7fb04c1070d0 2024-03-04T12:23:29.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.442+0000 7fb0427fc640 1 -- 172.21.15.60:0/1431993935 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb0480bdaf0 con 0x7fb04c1070d0 2024-03-04T12:23:29.448 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:23:29.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.444+0000 7fb0534d7640 1 -- 172.21.15.60:0/1431993935 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb0380a1d80 msgr2=0x7fb0380a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:29.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.444+0000 7fb0534d7640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb0380a1d80 0x7fb0380a4240 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb0440058c0 tx=0x7fb04400a4c0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.444+0000 7fb0534d7640 1 -- 172.21.15.60:0/1431993935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb04c1070d0 msgr2=0x7fb04c100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:29.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.444+0000 7fb0534d7640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb04c1070d0 0x7fb04c100360 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb048037040 tx=0x7fb048032f90 comp rx=0 tx=0).stop 2024-03-04T12:23:29.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.444+0000 7fb0534d7640 1 -- 172.21.15.60:0/1431993935 shutdown_connections 2024-03-04T12:23:29.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.444+0000 7fb0534d7640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb0380a1d80 0x7fb0380a4240 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.444+0000 7fb0534d7640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb04c1070d0 0x7fb04c100360 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.444+0000 7fb0534d7640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb04c1047a0 0x7fb04c0ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.444+0000 7fb0534d7640 1 --2- 172.21.15.60:0/1431993935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb04c101e70 0x7fb04c1017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.444+0000 7fb0534d7640 1 -- 172.21.15.60:0/1431993935 >> 172.21.15.60:0/1431993935 conn(0x7fb04c0fb7c0 msgr2=0x7fb04c1027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:29.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.444+0000 7fb0534d7640 1 -- 172.21.15.60:0/1431993935 shutdown_connections 2024-03-04T12:23:29.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.444+0000 7fb0534d7640 1 -- 172.21.15.60:0/1431993935 wait complete. 2024-03-04T12:23:29.461 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T12:23:29.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 -- 172.21.15.60:0/325461042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d90101df0 msgr2=0x7f7d90108680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:29.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 --2- 172.21.15.60:0/325461042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d90101df0 0x7f7d90108680 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7d88009ca0 tx=0x7f7d8802c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 -- 172.21.15.60:0/325461042 shutdown_connections 2024-03-04T12:23:29.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 --2- 172.21.15.60:0/325461042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d90101df0 0x7f7d90108680 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 --2- 172.21.15.60:0/325461042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d90101430 0x7f7d901018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 --2- 172.21.15.60:0/325461042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90100240 0x7f7d90100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 -- 172.21.15.60:0/325461042 >> 172.21.15.60:0/325461042 conn(0x7f7d900fb790 msgr2=0x7f7d900fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 -- 172.21.15.60:0/325461042 shutdown_connections 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 -- 172.21.15.60:0/325461042 wait complete. 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 Processor -- start 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 -- start start 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90100240 0x7f7d90197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d90101430 0x7f7d90197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d90101df0 0x7f7d9019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9010bed0 con 0x7f7d90101df0 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.559+0000 7f7d96bb3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9010bd50 con 0x7f7d90100240 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d96bb3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9010c050 con 0x7f7d90101430 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d95150640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d90101df0 0x7f7d9019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:29.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d95150640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d90101df0 0x7f7d9019ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52448/0 (socket says 172.21.15.60:52448) 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d95150640 1 -- 172.21.15.60:0/2128436756 learned_addr learned my addr 172.21.15.60:0/2128436756 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d9494f640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90100240 0x7f7d90197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d8ffff640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d90101430 0x7f7d90197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d95150640 1 -- 172.21.15.60:0/2128436756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d90101430 msgr2=0x7f7d90197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d95150640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d90101430 0x7f7d90197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d95150640 1 -- 172.21.15.60:0/2128436756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90100240 msgr2=0x7f7d90197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d95150640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90100240 0x7f7d90197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d95150640 1 -- 172.21.15.60:0/2128436756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9019f260 con 0x7f7d90101df0 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d95150640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d90101df0 0x7f7d9019ebf0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7d8802c700 tx=0x7f7d88032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d8dffb640 1 -- 172.21.15.60:0/2128436756 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d88039040 con 0x7f7d90101df0 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d8dffb640 1 -- 172.21.15.60:0/2128436756 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d88044070 con 0x7f7d90101df0 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d96bb3640 1 -- 172.21.15.60:0/2128436756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9019f4f0 con 0x7f7d90101df0 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d8dffb640 1 -- 172.21.15.60:0/2128436756 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d880042a0 con 0x7f7d90101df0 2024-03-04T12:23:29.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.560+0000 7f7d96bb3640 1 -- 172.21.15.60:0/2128436756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d9019fa30 con 0x7f7d90101df0 2024-03-04T12:23:29.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.561+0000 7f7d96bb3640 1 -- 172.21.15.60:0/2128436756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d9019fc90 con 0x7f7d90101df0 2024-03-04T12:23:29.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.563+0000 7f7d8dffb640 1 -- 172.21.15.60:0/2128436756 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f7d88003970 con 0x7f7d90101df0 2024-03-04T12:23:29.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.563+0000 7f7d8dffb640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7d800a1e00 0x7f7d800a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:29.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.563+0000 7f7d9494f640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7d800a1e00 0x7f7d800a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:29.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.563+0000 7f7d8dffb640 1 -- 172.21.15.60:0/2128436756 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7d880f0b30 con 0x7f7d90101df0 2024-03-04T12:23:29.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.564+0000 7f7d9494f640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7d800a1e00 0x7f7d800a42c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7d84005fd0 tx=0x7f7d84005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:29.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.567+0000 7f7d8dffb640 1 -- 172.21.15.60:0/2128436756 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7d880bddb0 con 0x7f7d90101df0 2024-03-04T12:23:29.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.761+0000 7f7d96bb3640 1 -- 172.21.15.60:0/2128436756 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7d9019fe60 con 0x7f7d90101df0 2024-03-04T12:23:29.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.762+0000 7f7d8dffb640 1 -- 172.21.15.60:0/2128436756 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+591 (secure 0 0 0) 0x7f7d880bd6b0 con 0x7f7d90101df0 2024-03-04T12:23:29.768 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/2852203003 2024-03-05T12:16:04.073665+0000 2024-03-04T12:23:29.768 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/3868523955 2024-03-05T12:16:04.073665+0000 2024-03-04T12:23:29.768 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6801/2246601804 2024-03-05T12:16:04.073665+0000 2024-03-04T12:23:29.768 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6800/2246601804 2024-03-05T12:16:04.073665+0000 2024-03-04T12:23:29.768 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/758894979 2024-03-05T12:15:37.282929+0000 2024-03-04T12:23:29.768 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6801/2061680026 2024-03-05T12:15:37.282929+0000 2024-03-04T12:23:29.768 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/3401854556 2024-03-05T12:15:37.282929+0000 2024-03-04T12:23:29.768 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6800/2061680026 2024-03-05T12:15:37.282929+0000 2024-03-04T12:23:29.768 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/3539227088 2024-03-05T12:15:37.282929+0000 2024-03-04T12:23:29.768 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/2028967685 2024-03-05T12:16:04.073665+0000 2024-03-04T12:23:29.769 INFO:teuthology.orchestra.run.smithi060.stderr:listed 10 entries 2024-03-04T12:23:29.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.764+0000 7f7d96bb3640 1 -- 172.21.15.60:0/2128436756 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7d800a1e00 msgr2=0x7f7d800a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:29.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.764+0000 7f7d96bb3640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7d800a1e00 0x7f7d800a42c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7d84005fd0 tx=0x7f7d84005850 comp rx=0 tx=0).stop 2024-03-04T12:23:29.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.764+0000 7f7d96bb3640 1 -- 172.21.15.60:0/2128436756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d90101df0 msgr2=0x7f7d9019ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:29.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.764+0000 7f7d96bb3640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d90101df0 0x7f7d9019ebf0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7d8802c700 tx=0x7f7d88032f20 comp rx=0 tx=0).stop 2024-03-04T12:23:29.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.764+0000 7f7d96bb3640 1 -- 172.21.15.60:0/2128436756 shutdown_connections 2024-03-04T12:23:29.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.764+0000 7f7d96bb3640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7d800a1e00 0x7f7d800a42c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.764+0000 7f7d96bb3640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7d90101df0 0x7f7d9019ebf0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.764+0000 7f7d96bb3640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d90101430 0x7f7d90197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.764+0000 7f7d96bb3640 1 --2- 172.21.15.60:0/2128436756 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d90100240 0x7f7d90197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:29.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.764+0000 7f7d96bb3640 1 -- 172.21.15.60:0/2128436756 >> 172.21.15.60:0/2128436756 conn(0x7f7d900fb790 msgr2=0x7f7d900fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:29.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.764+0000 7f7d96bb3640 1 -- 172.21.15.60:0/2128436756 shutdown_connections 2024-03-04T12:23:29.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:29.764+0000 7f7d96bb3640 1 -- 172.21.15.60:0/2128436756 wait complete. 2024-03-04T12:23:29.780 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi060.front.sepia.ceph.com... 2024-03-04T12:23:29.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:29.781 DEBUG:teuthology.orchestra.run.smithi060:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-04T12:23:29.806 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi060.stderr:ceph-fuse[77485]: fuse finished with error 0 and tester_r 0 2024-03-04T12:23:29.808 INFO:teuthology.orchestra.run:waiting for 300 2024-03-04T12:23:30.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:29 smithi060 ceph-mon[32745]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 300 B/s wr, 0 op/s 2024-03-04T12:23:30.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:29 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3097867085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:23:30.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:29 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1431993935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:23:30.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:29 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2128436756' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T12:23:30.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:29 smithi086 ceph-mon[36851]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 300 B/s wr, 0 op/s 2024-03-04T12:23:30.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:29 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3097867085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:23:30.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:29 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1431993935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:23:30.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:29 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2128436756' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T12:23:30.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:29 smithi063 ceph-mon[36717]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 300 B/s wr, 0 op/s 2024-03-04T12:23:30.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:29 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3097867085' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:23:30.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:29 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1431993935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:23:30.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:29 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2128436756' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T12:23:32.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:31 smithi086 ceph-mon[36851]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-04T12:23:32.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:31 smithi063 ceph-mon[36717]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-04T12:23:32.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:31 smithi060 ceph-mon[32745]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-04T12:23:34.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:33 smithi086 ceph-mon[36851]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-04T12:23:34.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:33 smithi063 ceph-mon[36717]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-04T12:23:34.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:33 smithi060 ceph-mon[32745]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-04T12:23:35.910 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2024-03-04T12:23:35.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:23:35.910 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-04T12:23:35.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:23:36.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.088+0000 7f583bc8e640 1 -- 172.21.15.60:0/2713804685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5834101450 msgr2=0x7f58341018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.088+0000 7f583bc8e640 1 --2- 172.21.15.60:0/2713804685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5834101450 0x7f58341018d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5828009900 tx=0x7f582802bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.088+0000 7f583bc8e640 1 -- 172.21.15.60:0/2713804685 shutdown_connections 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.088+0000 7f583bc8e640 1 --2- 172.21.15.60:0/2713804685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5834101e10 0x7f58341086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.088+0000 7f583bc8e640 1 --2- 172.21.15.60:0/2713804685 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5834101450 0x7f58341018d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.088+0000 7f583bc8e640 1 --2- 172.21.15.60:0/2713804685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834100260 0x7f5834100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.088+0000 7f583bc8e640 1 -- 172.21.15.60:0/2713804685 >> 172.21.15.60:0/2713804685 conn(0x7f58340fb7f0 msgr2=0x7f58340fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.088+0000 7f583bc8e640 1 -- 172.21.15.60:0/2713804685 shutdown_connections 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.088+0000 7f583bc8e640 1 -- 172.21.15.60:0/2713804685 wait complete. 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f583bc8e640 1 Processor -- start 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f583bc8e640 1 -- start start 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f583bc8e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834100260 0x7f583410a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f583bc8e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5834101450 0x7f5834108f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f583bc8e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5834101e10 0x7f5834109460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:36.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f583bc8e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583410cff0 con 0x7f5834101450 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f583bc8e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583410ce70 con 0x7f5834101e10 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f583bc8e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583410d170 con 0x7f5834100260 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f5839229640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5834101450 0x7f5834108f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f5839229640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5834101450 0x7f5834108f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37768/0 (socket says 172.21.15.60:37768) 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f5839229640 1 -- 172.21.15.60:0/1729456953 learned_addr learned my addr 172.21.15.60:0/1729456953 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f583a22b640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5834101e10 0x7f5834109460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.089+0000 7f5839a2a640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834100260 0x7f583410a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.090+0000 7f5839229640 1 -- 172.21.15.60:0/1729456953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834100260 msgr2=0x7f583410a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.090+0000 7f5839229640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834100260 0x7f583410a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.090+0000 7f5839229640 1 -- 172.21.15.60:0/1729456953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5834101e10 msgr2=0x7f5834109460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.090+0000 7f5839229640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5834101e10 0x7f5834109460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.090+0000 7f5839229640 1 -- 172.21.15.60:0/1729456953 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5834109cf0 con 0x7f5834101450 2024-03-04T12:23:36.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.090+0000 7f5839a2a640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834100260 0x7f583410a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:36.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.090+0000 7f5839229640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5834101450 0x7f5834108f20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5828009230 tx=0x7f5828005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:36.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.090+0000 7f5832ffd640 1 -- 172.21.15.60:0/1729456953 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5828041070 con 0x7f5834101450 2024-03-04T12:23:36.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.090+0000 7f583bc8e640 1 -- 172.21.15.60:0/1729456953 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58341a16e0 con 0x7f5834101450 2024-03-04T12:23:36.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.090+0000 7f5832ffd640 1 -- 172.21.15.60:0/1729456953 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58280054e0 con 0x7f5834101450 2024-03-04T12:23:36.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.090+0000 7f5832ffd640 1 -- 172.21.15.60:0/1729456953 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5828034440 con 0x7f5834101450 2024-03-04T12:23:36.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.090+0000 7f583bc8e640 1 -- 172.21.15.60:0/1729456953 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58341a1c30 con 0x7f5834101450 2024-03-04T12:23:36.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.091+0000 7f583bc8e640 1 -- 172.21.15.60:0/1729456953 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5834107580 con 0x7f5834101450 2024-03-04T12:23:36.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.092+0000 7f5832ffd640 1 -- 172.21.15.60:0/1729456953 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f5828044070 con 0x7f5834101450 2024-03-04T12:23:36.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.093+0000 7f5832ffd640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f581c0a1d80 0x7f581c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:36.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.093+0000 7f5832ffd640 1 -- 172.21.15.60:0/1729456953 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f58280f00e0 con 0x7f5834101450 2024-03-04T12:23:36.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.093+0000 7f5839a2a640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f581c0a1d80 0x7f581c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:36.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.093+0000 7f5839a2a640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f581c0a1d80 0x7f581c0a4240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f582c0058c0 tx=0x7f582c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:36.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.096+0000 7f5832ffd640 1 -- 172.21.15.60:0/1729456953 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f58280bd2d0 con 0x7f5834101450 2024-03-04T12:23:36.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.279+0000 7f583bc8e640 1 -- 172.21.15.60:0/1729456953 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f583402d080 con 0x7f5834101450 2024-03-04T12:23:36.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.280+0000 7f5832ffd640 1 -- 172.21.15.60:0/1729456953 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23997 (secure 0 0 0) 0x7f58280bcbd0 con 0x7f5834101450 2024-03-04T12:23:36.286 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:36.287 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":89,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:23:17.289925+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:23:36.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.282+0000 7f583bc8e640 1 -- 172.21.15.60:0/1729456953 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f581c0a1d80 msgr2=0x7f581c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.282+0000 7f583bc8e640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f581c0a1d80 0x7f581c0a4240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f582c0058c0 tx=0x7f582c005850 comp rx=0 tx=0).stop 2024-03-04T12:23:36.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.282+0000 7f583bc8e640 1 -- 172.21.15.60:0/1729456953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5834101450 msgr2=0x7f5834108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.282+0000 7f583bc8e640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5834101450 0x7f5834108f20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5828009230 tx=0x7f5828005dd0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.282+0000 7f583bc8e640 1 -- 172.21.15.60:0/1729456953 shutdown_connections 2024-03-04T12:23:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.282+0000 7f583bc8e640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f581c0a1d80 0x7f581c0a4240 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.282+0000 7f583bc8e640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5834101e10 0x7f5834109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.282+0000 7f583bc8e640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5834101450 0x7f5834108f20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.282+0000 7f583bc8e640 1 --2- 172.21.15.60:0/1729456953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5834100260 0x7f583410a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.282+0000 7f583bc8e640 1 -- 172.21.15.60:0/1729456953 >> 172.21.15.60:0/1729456953 conn(0x7f58340fb7f0 msgr2=0x7f5834104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.282+0000 7f583bc8e640 1 -- 172.21.15.60:0/1729456953 shutdown_connections 2024-03-04T12:23:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.283+0000 7f583bc8e640 1 -- 172.21.15.60:0/1729456953 wait complete. 2024-03-04T12:23:36.298 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:35 smithi060 ceph-mon[32745]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-04T12:23:36.299 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:23:36.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:35 smithi086 ceph-mon[36851]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-04T12:23:36.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:35 smithi063 ceph-mon[36717]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-04T12:23:36.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.396+0000 7fd86668c640 1 -- 172.21.15.60:0/2156402853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860102600 msgr2=0x7fd860102a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.396+0000 7fd86668c640 1 --2- 172.21.15.60:0/2156402853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860102600 0x7fd860102a80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd850009930 tx=0x7fd85002bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:36.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.396+0000 7fd86668c640 1 -- 172.21.15.60:0/2156402853 shutdown_connections 2024-03-04T12:23:36.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.396+0000 7fd86668c640 1 --2- 172.21.15.60:0/2156402853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd860102fc0 0x7fd860109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.396+0000 7fd86668c640 1 --2- 172.21.15.60:0/2156402853 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860102600 0x7fd860102a80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.396+0000 7fd86668c640 1 --2- 172.21.15.60:0/2156402853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8600ff630 0x7fd8600ffa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.396+0000 7fd86668c640 1 -- 172.21.15.60:0/2156402853 >> 172.21.15.60:0/2156402853 conn(0x7fd8600faf60 msgr2=0x7fd8600fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:36.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.396+0000 7fd86668c640 1 -- 172.21.15.60:0/2156402853 shutdown_connections 2024-03-04T12:23:36.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.396+0000 7fd86668c640 1 -- 172.21.15.60:0/2156402853 wait complete. 2024-03-04T12:23:36.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd86668c640 1 Processor -- start 2024-03-04T12:23:36.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd86668c640 1 -- start start 2024-03-04T12:23:36.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd86668c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8600ff630 0x7fd8601976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd86668c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860102600 0x7fd860197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd86668c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd860102fc0 0x7fd86019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd86668c640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86010d020 con 0x7fd860102600 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd86668c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86010cea0 con 0x7fd860102fc0 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd86668c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86010d1a0 con 0x7fd8600ff630 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd85f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860102600 0x7fd860197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd85f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860102600 0x7fd860197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37788/0 (socket says 172.21.15.60:37788) 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd85f7fe640 1 -- 172.21.15.60:0/23964601 learned_addr learned my addr 172.21.15.60:0/23964601 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd864c29640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd860102fc0 0x7fd86019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.397+0000 7fd85ffff640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8600ff630 0x7fd8601976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.398+0000 7fd85f7fe640 1 -- 172.21.15.60:0/23964601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8600ff630 msgr2=0x7fd8601976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.398+0000 7fd85f7fe640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8600ff630 0x7fd8601976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.398+0000 7fd85f7fe640 1 -- 172.21.15.60:0/23964601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd860102fc0 msgr2=0x7fd86019ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.398+0000 7fd85f7fe640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd860102fc0 0x7fd86019ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.398+0000 7fd85f7fe640 1 -- 172.21.15.60:0/23964601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86019f2f0 con 0x7fd860102600 2024-03-04T12:23:36.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.398+0000 7fd85ffff640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8600ff630 0x7fd8601976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:23:36.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.398+0000 7fd85f7fe640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860102600 0x7fd860197c10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd850000c00 tx=0x7fd850005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:36.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.398+0000 7fd85d7fa640 1 -- 172.21.15.60:0/23964601 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850041070 con 0x7fd860102600 2024-03-04T12:23:36.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.398+0000 7fd85d7fa640 1 -- 172.21.15.60:0/23964601 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8500054e0 con 0x7fd860102600 2024-03-04T12:23:36.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.398+0000 7fd86668c640 1 -- 172.21.15.60:0/23964601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86019f580 con 0x7fd860102600 2024-03-04T12:23:36.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.398+0000 7fd85d7fa640 1 -- 172.21.15.60:0/23964601 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8500346a0 con 0x7fd860102600 2024-03-04T12:23:36.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.398+0000 7fd86668c640 1 -- 172.21.15.60:0/23964601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86019fa70 con 0x7fd860102600 2024-03-04T12:23:36.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.399+0000 7fd86668c640 1 -- 172.21.15.60:0/23964601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd860108850 con 0x7fd860102600 2024-03-04T12:23:36.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.400+0000 7fd85d7fa640 1 -- 172.21.15.60:0/23964601 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fd850044070 con 0x7fd860102600 2024-03-04T12:23:36.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.401+0000 7fd85d7fa640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8480a1d80 0x7fd8480a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:36.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.401+0000 7fd85ffff640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8480a1d80 0x7fd8480a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:36.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.401+0000 7fd85d7fa640 1 -- 172.21.15.60:0/23964601 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd8500f0020 con 0x7fd860102600 2024-03-04T12:23:36.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.401+0000 7fd85ffff640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8480a1d80 0x7fd8480a4240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd860069830 tx=0x7fd854005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:36.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.404+0000 7fd85d7fa640 1 -- 172.21.15.60:0/23964601 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd8500bd210 con 0x7fd860102600 2024-03-04T12:23:36.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.586+0000 7fd86668c640 1 -- 172.21.15.60:0/23964601 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd86002d100 con 0x7fd860102600 2024-03-04T12:23:36.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.587+0000 7fd85d7fa640 1 -- 172.21.15.60:0/23964601 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23997 (secure 0 0 0) 0x7fd850003e20 con 0x7fd860102600 2024-03-04T12:23:36.593 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:36.593 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":89,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:23:17.289925+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:23:36.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.589+0000 7fd86668c640 1 -- 172.21.15.60:0/23964601 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8480a1d80 msgr2=0x7fd8480a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.589+0000 7fd86668c640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8480a1d80 0x7fd8480a4240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd860069830 tx=0x7fd854005e70 comp rx=0 tx=0).stop 2024-03-04T12:23:36.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.589+0000 7fd86668c640 1 -- 172.21.15.60:0/23964601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860102600 msgr2=0x7fd860197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.589+0000 7fd86668c640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860102600 0x7fd860197c10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd850000c00 tx=0x7fd850005dd0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.589+0000 7fd86668c640 1 -- 172.21.15.60:0/23964601 shutdown_connections 2024-03-04T12:23:36.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.589+0000 7fd86668c640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8480a1d80 0x7fd8480a4240 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.589+0000 7fd86668c640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd860102fc0 0x7fd86019ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.589+0000 7fd86668c640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd860102600 0x7fd860197c10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.589+0000 7fd86668c640 1 --2- 172.21.15.60:0/23964601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8600ff630 0x7fd8601976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.589+0000 7fd86668c640 1 -- 172.21.15.60:0/23964601 >> 172.21.15.60:0/23964601 conn(0x7fd8600faf60 msgr2=0x7fd860105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:36.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.590+0000 7fd86668c640 1 -- 172.21.15.60:0/23964601 shutdown_connections 2024-03-04T12:23:36.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.590+0000 7fd86668c640 1 -- 172.21.15.60:0/23964601 wait complete. 2024-03-04T12:23:36.607 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-04T12:23:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.707+0000 7f3e8daab640 1 -- 172.21.15.60:0/1377685680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e88101440 msgr2=0x7f3e881018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.707+0000 7f3e8daab640 1 --2- 172.21.15.60:0/1377685680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e88101440 0x7f3e881018c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c009900 tx=0x7f3e7c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.707+0000 7f3e8daab640 1 -- 172.21.15.60:0/1377685680 shutdown_connections 2024-03-04T12:23:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.707+0000 7f3e8daab640 1 --2- 172.21.15.60:0/1377685680 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e88101e00 0x7f3e88108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.707+0000 7f3e8daab640 1 --2- 172.21.15.60:0/1377685680 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e88101440 0x7f3e881018c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.707+0000 7f3e8daab640 1 --2- 172.21.15.60:0/1377685680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e88100250 0x7f3e88100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.707+0000 7f3e8daab640 1 -- 172.21.15.60:0/1377685680 >> 172.21.15.60:0/1377685680 conn(0x7f3e880fb7e0 msgr2=0x7f3e880fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.707+0000 7f3e8daab640 1 -- 172.21.15.60:0/1377685680 shutdown_connections 2024-03-04T12:23:36.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.707+0000 7f3e8daab640 1 -- 172.21.15.60:0/1377685680 wait complete. 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.707+0000 7f3e8daab640 1 Processor -- start 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.707+0000 7f3e8daab640 1 -- start start 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e8daab640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e88100250 0x7f3e8806a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e8daab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e88101440 0x7f3e8819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e8daab640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e88101e00 0x7f3e881a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e8daab640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8810bf30 con 0x7f3e88101440 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e8daab640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8810bdb0 con 0x7f3e88101e00 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e8daab640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8810c0b0 con 0x7f3e88100250 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e86ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e88101440 0x7f3e8819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e86ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e88101440 0x7f3e8819e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37806/0 (socket says 172.21.15.60:37806) 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e86ffd640 1 -- 172.21.15.60:0/4211216845 learned_addr learned my addr 172.21.15.60:0/4211216845 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e87fff640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e88101e00 0x7f3e881a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e86ffd640 1 -- 172.21.15.60:0/4211216845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e88100250 msgr2=0x7f3e8806a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e877fe640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e88100250 0x7f3e8806a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e86ffd640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e88100250 0x7f3e8806a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e86ffd640 1 -- 172.21.15.60:0/4211216845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e88101e00 msgr2=0x7f3e881a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e86ffd640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e88101e00 0x7f3e881a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e86ffd640 1 -- 172.21.15.60:0/4211216845 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e881a1430 con 0x7f3e88101440 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e87fff640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e88101e00 0x7f3e881a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e877fe640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e88100250 0x7f3e8806a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e86ffd640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e88101440 0x7f3e8819e6f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c000c00 tx=0x7f3e7c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e84ff9640 1 -- 172.21.15.60:0/4211216845 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c043020 con 0x7f3e88101440 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e8daab640 1 -- 172.21.15.60:0/4211216845 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e881a1660 con 0x7f3e88101440 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e84ff9640 1 -- 172.21.15.60:0/4211216845 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e7c004440 con 0x7f3e88101440 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e84ff9640 1 -- 172.21.15.60:0/4211216845 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7c03e450 con 0x7f3e88101440 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.708+0000 7f3e8daab640 1 -- 172.21.15.60:0/4211216845 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e881a1c00 con 0x7f3e88101440 2024-03-04T12:23:36.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.709+0000 7f3e8daab640 1 -- 172.21.15.60:0/4211216845 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e881074a0 con 0x7f3e88101440 2024-03-04T12:23:36.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.711+0000 7f3e84ff9640 1 -- 172.21.15.60:0/4211216845 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f3e7c046020 con 0x7f3e88101440 2024-03-04T12:23:36.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.712+0000 7f3e84ff9640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f3e700a1db0 0x7f3e700a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:36.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.712+0000 7f3e877fe640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f3e700a1db0 0x7f3e700a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:36.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.712+0000 7f3e84ff9640 1 -- 172.21.15.60:0/4211216845 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3e7c0f07a0 con 0x7f3e88101440 2024-03-04T12:23:36.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.712+0000 7f3e877fe640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f3e700a1db0 0x7f3e700a4270 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3e881012a0 tx=0x7f3e78005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:36.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.715+0000 7f3e84ff9640 1 -- 172.21.15.60:0/4211216845 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3e7c04d020 con 0x7f3e88101440 2024-03-04T12:23:36.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.940+0000 7f3e8daab640 1 -- 172.21.15.60:0/4211216845 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3e8802ceb0 con 0x7f3e88101440 2024-03-04T12:23:36.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.983+0000 7f3e84ff9640 1 -- 172.21.15.60:0/4211216845 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3e7c0bd990 con 0x7f3e88101440 2024-03-04T12:23:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.985+0000 7f3e8daab640 1 -- 172.21.15.60:0/4211216845 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f3e700a1db0 msgr2=0x7f3e700a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.985+0000 7f3e8daab640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f3e700a1db0 0x7f3e700a4270 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3e881012a0 tx=0x7f3e78005d30 comp rx=0 tx=0).stop 2024-03-04T12:23:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.985+0000 7f3e8daab640 1 -- 172.21.15.60:0/4211216845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e88101440 msgr2=0x7f3e8819e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.985+0000 7f3e8daab640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e88101440 0x7f3e8819e6f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c000c00 tx=0x7f3e7c004040 comp rx=0 tx=0).stop 2024-03-04T12:23:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.985+0000 7f3e8daab640 1 -- 172.21.15.60:0/4211216845 shutdown_connections 2024-03-04T12:23:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.985+0000 7f3e8daab640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f3e700a1db0 0x7f3e700a4270 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.985+0000 7f3e8daab640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e88101e00 0x7f3e881a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.985+0000 7f3e8daab640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3e88101440 0x7f3e8819e6f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.985+0000 7f3e8daab640 1 --2- 172.21.15.60:0/4211216845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e88100250 0x7f3e8806a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:36.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.985+0000 7f3e8daab640 1 -- 172.21.15.60:0/4211216845 >> 172.21.15.60:0/4211216845 conn(0x7f3e880fb7e0 msgr2=0x7f3e88104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:36.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.985+0000 7f3e8daab640 1 -- 172.21.15.60:0/4211216845 shutdown_connections 2024-03-04T12:23:36.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:36.985+0000 7f3e8daab640 1 -- 172.21.15.60:0/4211216845 wait complete. 2024-03-04T12:23:37.002 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-04T12:23:37.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-04T12:23:37.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.102+0000 7f925d8dc640 1 -- 172.21.15.60:0/1520391530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92581070b0 msgr2=0x7f92581094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:37.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.102+0000 7f925d8dc640 1 --2- 172.21.15.60:0/1520391530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92581070b0 0x7f92581094a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9248009ca0 tx=0x7f924802c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:37.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.102+0000 7f925d8dc640 1 -- 172.21.15.60:0/1520391530 shutdown_connections 2024-03-04T12:23:37.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.102+0000 7f925d8dc640 1 --2- 172.21.15.60:0/1520391530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92581070b0 0x7f92581094a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:37.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.102+0000 7f925d8dc640 1 --2- 172.21.15.60:0/1520391530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9258104780 0x7f9258106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:37.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.102+0000 7f925d8dc640 1 --2- 172.21.15.60:0/1520391530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9258101e50 0x7f9258104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.102+0000 7f925d8dc640 1 -- 172.21.15.60:0/1520391530 >> 172.21.15.60:0/1520391530 conn(0x7f92580fb7c0 msgr2=0x7f92580fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.102+0000 7f925d8dc640 1 -- 172.21.15.60:0/1520391530 shutdown_connections 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.102+0000 7f925d8dc640 1 -- 172.21.15.60:0/1520391530 wait complete. 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f925d8dc640 1 Processor -- start 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f925d8dc640 1 -- start start 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f925d8dc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9258101e50 0x7f92581976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f925d8dc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9258104780 0x7f9258197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f925d8dc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92581070b0 0x7f925819ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f925d8dc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925810d040 con 0x7f92581070b0 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f925d8dc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925810cec0 con 0x7f9258104780 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f925d8dc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925810d1c0 con 0x7f9258101e50 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f92577fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92581070b0 0x7f925819ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f92577fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92581070b0 0x7f925819ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37824/0 (socket says 172.21.15.60:37824) 2024-03-04T12:23:37.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f92577fe640 1 -- 172.21.15.60:0/2933421312 learned_addr learned my addr 172.21.15.60:0/2933421312 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f9256ffd640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9258101e50 0x7f92581976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f92567fc640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9258104780 0x7f9258197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f92577fe640 1 -- 172.21.15.60:0/2933421312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9258101e50 msgr2=0x7f92581976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f92577fe640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9258101e50 0x7f92581976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f92577fe640 1 -- 172.21.15.60:0/2933421312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9258104780 msgr2=0x7f9258197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f92577fe640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9258104780 0x7f9258197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.103+0000 7f92577fe640 1 -- 172.21.15.60:0/2933421312 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925819f2d0 con 0x7f92581070b0 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.104+0000 7f92577fe640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92581070b0 0x7f925819ec60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f924802c700 tx=0x7f9248032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.104+0000 7f925c8da640 1 -- 172.21.15.60:0/2933421312 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9248039040 con 0x7f92581070b0 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.104+0000 7f925c8da640 1 -- 172.21.15.60:0/2933421312 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9248044070 con 0x7f92581070b0 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.104+0000 7f925d8dc640 1 -- 172.21.15.60:0/2933421312 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925819f560 con 0x7f92581070b0 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.104+0000 7f925c8da640 1 -- 172.21.15.60:0/2933421312 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92480042a0 con 0x7f92581070b0 2024-03-04T12:23:37.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.104+0000 7f925d8dc640 1 -- 172.21.15.60:0/2933421312 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925819fa50 con 0x7f92581070b0 2024-03-04T12:23:37.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.105+0000 7f925d8dc640 1 -- 172.21.15.60:0/2933421312 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92580fd2b0 con 0x7f92581070b0 2024-03-04T12:23:37.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.106+0000 7f925c8da640 1 -- 172.21.15.60:0/2933421312 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f9248003970 con 0x7f92581070b0 2024-03-04T12:23:37.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.107+0000 7f925c8da640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f92400a1d80 0x7f92400a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:37.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.107+0000 7f9256ffd640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f92400a1d80 0x7f92400a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:37.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.107+0000 7f925c8da640 1 -- 172.21.15.60:0/2933421312 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f92480f0c90 con 0x7f92581070b0 2024-03-04T12:23:37.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.107+0000 7f9256ffd640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f92400a1d80 0x7f92400a4240 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9250009890 tx=0x7f9250008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:37.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.111+0000 7f925c8da640 1 -- 172.21.15.60:0/2933421312 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f92480bdf10 con 0x7f92581070b0 2024-03-04T12:23:37.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.332+0000 7f925d8dc640 1 -- 172.21.15.60:0/2933421312 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f925802ceb0 con 0x7f92581070b0 2024-03-04T12:23:37.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:36 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1729456953' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:37.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:36 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/23964601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:37.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:36 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/4211216845' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-04T12:23:37.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:36 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1729456953' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:37.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:36 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/23964601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:37.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:36 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/4211216845' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-04T12:23:37.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:36 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1729456953' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:37.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:36 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/23964601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:37.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:36 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/4211216845' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-04T12:23:37.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.989+0000 7f925c8da640 1 -- 172.21.15.60:0/2933421312 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f9248047e40 con 0x7f92581070b0 2024-03-04T12:23:37.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.992+0000 7f925d8dc640 1 -- 172.21.15.60:0/2933421312 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f92400a1d80 msgr2=0x7f92400a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:37.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.992+0000 7f925d8dc640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f92400a1d80 0x7f92400a4240 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9250009890 tx=0x7f9250008040 comp rx=0 tx=0).stop 2024-03-04T12:23:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.992+0000 7f925d8dc640 1 -- 172.21.15.60:0/2933421312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92581070b0 msgr2=0x7f925819ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.992+0000 7f925d8dc640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92581070b0 0x7f925819ec60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f924802c700 tx=0x7f9248032f20 comp rx=0 tx=0).stop 2024-03-04T12:23:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.992+0000 7f925d8dc640 1 -- 172.21.15.60:0/2933421312 shutdown_connections 2024-03-04T12:23:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.992+0000 7f925d8dc640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f92400a1d80 0x7f92400a4240 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.992+0000 7f925d8dc640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f92581070b0 0x7f925819ec60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.992+0000 7f925d8dc640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9258104780 0x7f9258197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.992+0000 7f925d8dc640 1 --2- 172.21.15.60:0/2933421312 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9258101e50 0x7f92581976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.993+0000 7f925d8dc640 1 -- 172.21.15.60:0/2933421312 >> 172.21.15.60:0/2933421312 conn(0x7f92580fb7c0 msgr2=0x7f9258102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.993+0000 7f925d8dc640 1 -- 172.21.15.60:0/2933421312 shutdown_connections 2024-03-04T12:23:37.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:37.993+0000 7f925d8dc640 1 -- 172.21.15.60:0/2933421312 wait complete. 2024-03-04T12:23:38.014 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-04T12:23:38.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.120+0000 7fdde7661640 1 -- 172.21.15.60:0/2362540117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdde0107030 msgr2=0x7fdde0109480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:38.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.120+0000 7fdde7661640 1 --2- 172.21.15.60:0/2362540117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdde0107030 0x7fdde0109480 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009ca0 tx=0x7fddd002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:38.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 -- 172.21.15.60:0/2362540117 shutdown_connections 2024-03-04T12:23:38.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 --2- 172.21.15.60:0/2362540117 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdde0107030 0x7fdde0109480 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:38.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 --2- 172.21.15.60:0/2362540117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde0104700 0x7fdde0106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:38.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 --2- 172.21.15.60:0/2362540117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde0101dd0 0x7fdde01041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 -- 172.21.15.60:0/2362540117 >> 172.21.15.60:0/2362540117 conn(0x7fdde00fb7c0 msgr2=0x7fdde00fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 -- 172.21.15.60:0/2362540117 shutdown_connections 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 -- 172.21.15.60:0/2362540117 wait complete. 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 Processor -- start 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 -- start start 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde0101dd0 0x7fdde0101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde0104700 0x7fdde00ffd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdde0107030 0x7fdde01002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde010cfe0 con 0x7fdde0107030 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde010ce60 con 0x7fdde0104700 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde7661640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde010d160 con 0x7fdde0101dd0 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.121+0000 7fdde5bfe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdde0107030 0x7fdde01002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fdde5bfe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdde0107030 0x7fdde01002c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37830/0 (socket says 172.21.15.60:37830) 2024-03-04T12:23:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fdde5bfe640 1 -- 172.21.15.60:0/1655376102 learned_addr learned my addr 172.21.15.60:0/1655376102 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fdde53fd640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde0101dd0 0x7fdde0101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fdde5bfe640 1 -- 172.21.15.60:0/1655376102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde0101dd0 msgr2=0x7fdde0101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fdde5bfe640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde0101dd0 0x7fdde0101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fdde4bfc640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde0104700 0x7fdde00ffd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fdde5bfe640 1 -- 172.21.15.60:0/1655376102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde0104700 msgr2=0x7fdde00ffd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fdde5bfe640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde0104700 0x7fdde00ffd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fdde5bfe640 1 -- 172.21.15.60:0/1655376102 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde0100b50 con 0x7fdde0107030 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fdde5bfe640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdde0107030 0x7fdde01002c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009bf0 tx=0x7fddd0035ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fddde7fc640 1 -- 172.21.15.60:0/1655376102 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0039040 con 0x7fdde0107030 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fddde7fc640 1 -- 172.21.15.60:0/1655376102 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddd0044070 con 0x7fdde0107030 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fdde7661640 1 -- 172.21.15.60:0/1655376102 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde01a16a0 con 0x7fdde0107030 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fddde7fc640 1 -- 172.21.15.60:0/1655376102 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd00043a0 con 0x7fdde0107030 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.122+0000 7fdde7661640 1 -- 172.21.15.60:0/1655376102 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde01a1bf0 con 0x7fdde0107030 2024-03-04T12:23:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.123+0000 7fdde7661640 1 -- 172.21.15.60:0/1655376102 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde00fd2b0 con 0x7fdde0107030 2024-03-04T12:23:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.124+0000 7fddde7fc640 1 -- 172.21.15.60:0/1655376102 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fddd0003ae0 con 0x7fdde0107030 2024-03-04T12:23:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.125+0000 7fddde7fc640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fddc80a1d80 0x7fddc80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.125+0000 7fdde53fd640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fddc80a1d80 0x7fddc80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.125+0000 7fddde7fc640 1 -- 172.21.15.60:0/1655376102 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fddd00f0740 con 0x7fdde0107030 2024-03-04T12:23:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.126+0000 7fdde53fd640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fddc80a1d80 0x7fddc80a4240 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fddd80058c0 tx=0x7fddd800a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:38.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.130+0000 7fddde7fc640 1 -- 172.21.15.60:0/1655376102 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fddd00bd8d0 con 0x7fdde0107030 2024-03-04T12:23:38.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:37 smithi086 ceph-mon[36851]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-04T12:23:38.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:37 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/4211216845' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-04T12:23:38.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:37 smithi086 ceph-mon[36851]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:38.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:37 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2933421312' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-04T12:23:38.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:37 smithi063 ceph-mon[36717]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-04T12:23:38.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:37 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/4211216845' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-04T12:23:38.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:37 smithi063 ceph-mon[36717]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:38.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:37 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2933421312' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-04T12:23:38.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:38.350+0000 7fdde7661640 1 -- 172.21.15.60:0/1655376102 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fdde0061f30 con 0x7fdde0107030 2024-03-04T12:23:38.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:37 smithi060 ceph-mon[32745]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-04T12:23:38.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:37 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/4211216845' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-04T12:23:38.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:37 smithi060 ceph-mon[32745]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:38.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:37 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2933421312' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-04T12:23:39.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.000+0000 7fddde7fc640 1 -- 172.21.15.60:0/1655376102 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7fddd00bd1d0 con 0x7fdde0107030 2024-03-04T12:23:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.003+0000 7fdde7661640 1 -- 172.21.15.60:0/1655376102 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fddc80a1d80 msgr2=0x7fddc80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.003+0000 7fdde7661640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fddc80a1d80 0x7fddc80a4240 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fddd80058c0 tx=0x7fddd800a4c0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.004+0000 7fdde7661640 1 -- 172.21.15.60:0/1655376102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdde0107030 msgr2=0x7fdde01002c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.004+0000 7fdde7661640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdde0107030 0x7fdde01002c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009bf0 tx=0x7fddd0035ae0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.004+0000 7fdde7661640 1 -- 172.21.15.60:0/1655376102 shutdown_connections 2024-03-04T12:23:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.004+0000 7fdde7661640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fddc80a1d80 0x7fddc80a4240 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.004+0000 7fdde7661640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdde0107030 0x7fdde01002c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.004+0000 7fdde7661640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdde0104700 0x7fdde00ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.004+0000 7fdde7661640 1 --2- 172.21.15.60:0/1655376102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde0101dd0 0x7fdde0101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.004+0000 7fdde7661640 1 -- 172.21.15.60:0/1655376102 >> 172.21.15.60:0/1655376102 conn(0x7fdde00fb7c0 msgr2=0x7fdde0102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.004+0000 7fdde7661640 1 -- 172.21.15.60:0/1655376102 shutdown_connections 2024-03-04T12:23:39.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.004+0000 7fdde7661640 1 -- 172.21.15.60:0/1655376102 wait complete. 2024-03-04T12:23:39.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-04T12:23:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.134+0000 7fcf26081640 1 -- 172.21.15.60:0/3419050380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf20101440 msgr2=0x7fcf201018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.134+0000 7fcf26081640 1 --2- 172.21.15.60:0/3419050380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf20101440 0x7fcf201018c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009900 tx=0x7fcf1402bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.134+0000 7fcf26081640 1 -- 172.21.15.60:0/3419050380 shutdown_connections 2024-03-04T12:23:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.134+0000 7fcf26081640 1 --2- 172.21.15.60:0/3419050380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf20101e00 0x7fcf20108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.134+0000 7fcf26081640 1 --2- 172.21.15.60:0/3419050380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf20101440 0x7fcf201018c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.134+0000 7fcf26081640 1 --2- 172.21.15.60:0/3419050380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf20100250 0x7fcf20100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.134+0000 7fcf26081640 1 -- 172.21.15.60:0/3419050380 >> 172.21.15.60:0/3419050380 conn(0x7fcf200fb7c0 msgr2=0x7fcf200fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.134+0000 7fcf26081640 1 -- 172.21.15.60:0/3419050380 shutdown_connections 2024-03-04T12:23:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.134+0000 7fcf26081640 1 -- 172.21.15.60:0/3419050380 wait complete. 2024-03-04T12:23:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.134+0000 7fcf26081640 1 Processor -- start 2024-03-04T12:23:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.134+0000 7fcf26081640 1 -- start start 2024-03-04T12:23:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf26081640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf20100250 0x7fcf201976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf26081640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf20101440 0x7fcf20197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf26081640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf20101e00 0x7fcf2019eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf26081640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2010bee0 con 0x7fcf20101440 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf26081640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2010bd60 con 0x7fcf20100250 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf26081640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2010c060 con 0x7fcf20101e00 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf20101440 0x7fcf20197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf20101440 0x7fcf20197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37854/0 (socket says 172.21.15.60:37854) 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1effd640 1 -- 172.21.15.60:0/2010277080 learned_addr learned my addr 172.21.15.60:0/2010277080 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1f7fe640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf20100250 0x7fcf201976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1ffff640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf20101e00 0x7fcf2019eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1effd640 1 -- 172.21.15.60:0/2010277080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf20101e00 msgr2=0x7fcf2019eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1effd640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf20101e00 0x7fcf2019eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1effd640 1 -- 172.21.15.60:0/2010277080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf20100250 msgr2=0x7fcf201976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:39.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1effd640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf20100250 0x7fcf201976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1effd640 1 -- 172.21.15.60:0/2010277080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2019f310 con 0x7fcf20101440 2024-03-04T12:23:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1effd640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf20101440 0x7fcf20197c30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009230 tx=0x7fcf14005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1cff9640 1 -- 172.21.15.60:0/2010277080 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14041070 con 0x7fcf20101440 2024-03-04T12:23:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1cff9640 1 -- 172.21.15.60:0/2010277080 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf140054e0 con 0x7fcf20101440 2024-03-04T12:23:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf26081640 1 -- 172.21.15.60:0/2010277080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2019f5a0 con 0x7fcf20101440 2024-03-04T12:23:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf1cff9640 1 -- 172.21.15.60:0/2010277080 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14034440 con 0x7fcf20101440 2024-03-04T12:23:39.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.135+0000 7fcf26081640 1 -- 172.21.15.60:0/2010277080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2019fae0 con 0x7fcf20101440 2024-03-04T12:23:39.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.137+0000 7fcf26081640 1 -- 172.21.15.60:0/2010277080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2019fd40 con 0x7fcf20101440 2024-03-04T12:23:39.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.138+0000 7fcf1cff9640 1 -- 172.21.15.60:0/2010277080 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fcf14044070 con 0x7fcf20101440 2024-03-04T12:23:39.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.138+0000 7fcf1cff9640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcf080a1e00 0x7fcf080a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:39.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.138+0000 7fcf1cff9640 1 -- 172.21.15.60:0/2010277080 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcf140efd30 con 0x7fcf20101440 2024-03-04T12:23:39.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.138+0000 7fcf1f7fe640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcf080a1e00 0x7fcf080a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:39.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.139+0000 7fcf1f7fe640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcf080a1e00 0x7fcf080a42c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcf10005fd0 tx=0x7fcf10005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:39.148 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.143+0000 7fcf1cff9640 1 -- 172.21.15.60:0/2010277080 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcf140bcf20 con 0x7fcf20101440 2024-03-04T12:23:39.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:38 smithi086 ceph-mon[36851]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-04T12:23:39.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:38 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2933421312' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-04T12:23:39.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:38 smithi086 ceph-mon[36851]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-04T12:23:39.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:38 smithi086 ceph-mon[36851]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:39.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:38 smithi086 ceph-mon[36851]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-04T12:23:39.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:38 smithi086 ceph-mon[36851]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-04T12:23:39.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:38 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1655376102' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-04T12:23:39.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:38 smithi063 ceph-mon[36717]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-04T12:23:39.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:38 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2933421312' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-04T12:23:39.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:38 smithi063 ceph-mon[36717]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-04T12:23:39.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:38 smithi063 ceph-mon[36717]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:39.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:38 smithi063 ceph-mon[36717]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-04T12:23:39.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:38 smithi063 ceph-mon[36717]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-04T12:23:39.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:38 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1655376102' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-04T12:23:39.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.362+0000 7fcf26081640 1 -- 172.21.15.60:0/2010277080 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fcf2019ff10 con 0x7fcf20101440 2024-03-04T12:23:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:38 smithi060 ceph-mon[32745]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-04T12:23:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:38 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2933421312' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-04T12:23:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:38 smithi060 ceph-mon[32745]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-04T12:23:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:38 smithi060 ceph-mon[32745]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-04T12:23:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:38 smithi060 ceph-mon[32745]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-04T12:23:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:38 smithi060 ceph-mon[32745]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-04T12:23:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:38 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1655376102' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-04T12:23:39.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.880+0000 7fcf1cff9640 1 -- 172.21.15.60:0/2010277080 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7fcf140bc820 con 0x7fcf20101440 2024-03-04T12:23:39.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.883+0000 7fcf26081640 1 -- 172.21.15.60:0/2010277080 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcf080a1e00 msgr2=0x7fcf080a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:39.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.883+0000 7fcf26081640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcf080a1e00 0x7fcf080a42c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcf10005fd0 tx=0x7fcf10005850 comp rx=0 tx=0).stop 2024-03-04T12:23:39.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.883+0000 7fcf26081640 1 -- 172.21.15.60:0/2010277080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf20101440 msgr2=0x7fcf20197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:39.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.883+0000 7fcf26081640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf20101440 0x7fcf20197c30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009230 tx=0x7fcf14005dd0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.883+0000 7fcf26081640 1 -- 172.21.15.60:0/2010277080 shutdown_connections 2024-03-04T12:23:39.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.883+0000 7fcf26081640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcf080a1e00 0x7fcf080a42c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.883+0000 7fcf26081640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf20101e00 0x7fcf2019eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.883+0000 7fcf26081640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcf20101440 0x7fcf20197c30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.883+0000 7fcf26081640 1 --2- 172.21.15.60:0/2010277080 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf20100250 0x7fcf201976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:39.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.883+0000 7fcf26081640 1 -- 172.21.15.60:0/2010277080 >> 172.21.15.60:0/2010277080 conn(0x7fcf200fb7c0 msgr2=0x7fcf200fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:39.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.883+0000 7fcf26081640 1 -- 172.21.15.60:0/2010277080 shutdown_connections 2024-03-04T12:23:39.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:39.883+0000 7fcf26081640 1 -- 172.21.15.60:0/2010277080 wait complete. 2024-03-04T12:23:39.905 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-04T12:23:39.906 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-04T12:23:40.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 -- 172.21.15.60:0/2290400230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20f0100250 msgr2=0x7f20f0100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 --2- 172.21.15.60:0/2290400230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20f0100250 0x7f20f0100670 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f20ec009900 tx=0x7f20ec02bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:40.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 -- 172.21.15.60:0/2290400230 shutdown_connections 2024-03-04T12:23:40.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 --2- 172.21.15.60:0/2290400230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20f0101e00 0x7f20f0108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 --2- 172.21.15.60:0/2290400230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f0101440 0x7f20f01018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 --2- 172.21.15.60:0/2290400230 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20f0100250 0x7f20f0100670 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 -- 172.21.15.60:0/2290400230 >> 172.21.15.60:0/2290400230 conn(0x7f20f00fb7c0 msgr2=0x7f20f00fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:40.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 -- 172.21.15.60:0/2290400230 shutdown_connections 2024-03-04T12:23:40.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 -- 172.21.15.60:0/2290400230 wait complete. 2024-03-04T12:23:40.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 Processor -- start 2024-03-04T12:23:40.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 -- start start 2024-03-04T12:23:40.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20f0100250 0x7f20f0104190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:40.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f0101440 0x7f20f0102820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20f0101e00 0x7f20f0102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f010bee0 con 0x7f20f0100250 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f010bd60 con 0x7f20f0101e00 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.012+0000 7f20f86f1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f010c060 con 0x7f20f0101440 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f648d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20f0100250 0x7f20f0104190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f5c8c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f0101440 0x7f20f0102820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f648d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20f0100250 0x7f20f0104190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37874/0 (socket says 172.21.15.60:37874) 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f648d640 1 -- 172.21.15.60:0/405176462 learned_addr learned my addr 172.21.15.60:0/405176462 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f6c8e640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20f0101e00 0x7f20f0102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f5c8c640 1 -- 172.21.15.60:0/405176462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20f0101e00 msgr2=0x7f20f0102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f5c8c640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20f0101e00 0x7f20f0102d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f5c8c640 1 -- 172.21.15.60:0/405176462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20f0100250 msgr2=0x7f20f0104190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f5c8c640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20f0100250 0x7f20f0104190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f5c8c640 1 -- 172.21.15.60:0/405176462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f0103620 con 0x7f20f0101440 2024-03-04T12:23:40.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f648d640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20f0100250 0x7f20f0104190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:23:40.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f5c8c640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f0101440 0x7f20f0102820 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20e0005fd0 tx=0x7f20e000dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:40.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20e77fe640 1 -- 172.21.15.60:0/405176462 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e0009c30 con 0x7f20f0101440 2024-03-04T12:23:40.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20e77fe640 1 -- 172.21.15.60:0/405176462 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20e0009dd0 con 0x7f20f0101440 2024-03-04T12:23:40.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f86f1640 1 -- 172.21.15.60:0/405176462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f01a17e0 con 0x7f20f0101440 2024-03-04T12:23:40.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20e77fe640 1 -- 172.21.15.60:0/405176462 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e0010460 con 0x7f20f0101440 2024-03-04T12:23:40.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.013+0000 7f20f86f1640 1 -- 172.21.15.60:0/405176462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f01a1d00 con 0x7f20f0101440 2024-03-04T12:23:40.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.014+0000 7f20f86f1640 1 -- 172.21.15.60:0/405176462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f01a1f60 con 0x7f20f0101440 2024-03-04T12:23:40.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.015+0000 7f20e77fe640 1 -- 172.21.15.60:0/405176462 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f20e0003ef0 con 0x7f20f0101440 2024-03-04T12:23:40.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.016+0000 7f20e77fe640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f20dc0a1d80 0x7f20dc0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:40.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.016+0000 7f20f648d640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f20dc0a1d80 0x7f20dc0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:40.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.016+0000 7f20e77fe640 1 -- 172.21.15.60:0/405176462 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f20e00cdbf0 con 0x7f20f0101440 2024-03-04T12:23:40.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.016+0000 7f20f648d640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f20dc0a1d80 0x7f20dc0a4240 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f20ec02c8e0 tx=0x7f20ec005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:40.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.021+0000 7f20e77fe640 1 -- 172.21.15.60:0/405176462 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f20e0017020 con 0x7f20f0101440 2024-03-04T12:23:40.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-04T12:23:40.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1655376102' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-04T12:23:40.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] up:active 2024-03-04T12:23:40.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-04T12:23:40.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-04T12:23:40.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-04T12:23:40.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-04T12:23:40.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2010277080' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-04T12:23:40.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2010277080' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-04T12:23:40.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] up:active 2024-03-04T12:23:40.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: daemon mds.k assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-04T12:23:40.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-04T12:23:40.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=k=up:creating} 3 up:standby-replay 5 up:standby 2024-03-04T12:23:40.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-mon[32745]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-03-04T12:23:40.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.229+0000 7f20f86f1640 1 -- 172.21.15.60:0/405176462 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f20f00fd9d0 con 0x7f20dc0a1d80 2024-03-04T12:23:40.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.257+0000 7f20e77fe640 1 -- 172.21.15.60:0/405176462 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f20f00fd9d0 con 0x7f20dc0a1d80 2024-03-04T12:23:40.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.259+0000 7f20f86f1640 1 -- 172.21.15.60:0/405176462 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f20dc0a1d80 msgr2=0x7f20dc0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.259+0000 7f20f86f1640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f20dc0a1d80 0x7f20dc0a4240 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f20ec02c8e0 tx=0x7f20ec005b50 comp rx=0 tx=0).stop 2024-03-04T12:23:40.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.259+0000 7f20f86f1640 1 -- 172.21.15.60:0/405176462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f0101440 msgr2=0x7f20f0102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.259+0000 7f20f86f1640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f0101440 0x7f20f0102820 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20e0005fd0 tx=0x7f20e000dde0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.259+0000 7f20f86f1640 1 -- 172.21.15.60:0/405176462 shutdown_connections 2024-03-04T12:23:40.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.259+0000 7f20f86f1640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f20dc0a1d80 0x7f20dc0a4240 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.259+0000 7f20f86f1640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20f0101e00 0x7f20f0102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.259+0000 7f20f86f1640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f0101440 0x7f20f0102820 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.259+0000 7f20f86f1640 1 --2- 172.21.15.60:0/405176462 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f20f0100250 0x7f20f0104190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.259+0000 7f20f86f1640 1 -- 172.21.15.60:0/405176462 >> 172.21.15.60:0/405176462 conn(0x7f20f00fb7c0 msgr2=0x7f20f00fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:40.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.259+0000 7f20f86f1640 1 -- 172.21.15.60:0/405176462 shutdown_connections 2024-03-04T12:23:40.265 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.259+0000 7f20f86f1640 1 -- 172.21.15.60:0/405176462 wait complete. 2024-03-04T12:23:40.276 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-04T12:23:40.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-04T12:23:40.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1655376102' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-04T12:23:40.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] up:active 2024-03-04T12:23:40.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-04T12:23:40.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-04T12:23:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-04T12:23:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-04T12:23:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2010277080' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-04T12:23:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2010277080' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-04T12:23:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] up:active 2024-03-04T12:23:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: daemon mds.k assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-04T12:23:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-04T12:23:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=k=up:creating} 3 up:standby-replay 5 up:standby 2024-03-04T12:23:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:40 smithi086 ceph-mon[36851]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-03-04T12:23:40.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-04T12:23:40.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1655376102' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-04T12:23:40.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] up:active 2024-03-04T12:23:40.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-04T12:23:40.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-04T12:23:40.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-04T12:23:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-04T12:23:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2010277080' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-04T12:23:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2010277080' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-04T12:23:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] up:active 2024-03-04T12:23:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: daemon mds.k assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-04T12:23:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-04T12:23:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=k=up:creating} 3 up:standby-replay 5 up:standby 2024-03-04T12:23:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:40 smithi063 ceph-mon[36717]: daemon mds.k is now active in filesystem cephfs as rank 3 2024-03-04T12:23:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.379+0000 7f167fbdf640 1 -- 172.21.15.60:0/3591080843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1678104780 msgr2=0x7f1678106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.379+0000 7f167fbdf640 1 --2- 172.21.15.60:0/3591080843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1678104780 0x7f1678106b70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1674009900 tx=0x7f167402bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.379+0000 7f167fbdf640 1 -- 172.21.15.60:0/3591080843 shutdown_connections 2024-03-04T12:23:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.379+0000 7f167fbdf640 1 --2- 172.21.15.60:0/3591080843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16781070b0 0x7f16781094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.379+0000 7f167fbdf640 1 --2- 172.21.15.60:0/3591080843 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1678104780 0x7f1678106b70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.379+0000 7f167fbdf640 1 --2- 172.21.15.60:0/3591080843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1678101e50 0x7f1678104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.379+0000 7f167fbdf640 1 -- 172.21.15.60:0/3591080843 >> 172.21.15.60:0/3591080843 conn(0x7f16780fb7c0 msgr2=0x7f16780fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167fbdf640 1 -- 172.21.15.60:0/3591080843 shutdown_connections 2024-03-04T12:23:40.388 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167fbdf640 1 -- 172.21.15.60:0/3591080843 wait complete. 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167fbdf640 1 Processor -- start 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167fbdf640 1 -- start start 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167fbdf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1678101e50 0x7f16781017c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167fbdf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1678104780 0x7f16780ffe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167fbdf640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16781070b0 0x7f1678100340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167fbdf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167810cfe0 con 0x7f1678101e50 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167fbdf640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167810ce60 con 0x7f16781070b0 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167fbdf640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167810d160 con 0x7f1678104780 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167d97b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1678101e50 0x7f16781017c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167d97b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1678101e50 0x7f16781017c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37910/0 (socket says 172.21.15.60:37910) 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.380+0000 7f167d97b640 1 -- 172.21.15.60:0/3696251891 learned_addr learned my addr 172.21.15.60:0/3696251891 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f167d17a640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1678104780 0x7f16780ffe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f167e17c640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16781070b0 0x7f1678100340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f167d97b640 1 -- 172.21.15.60:0/3696251891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1678104780 msgr2=0x7f16780ffe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f167d97b640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1678104780 0x7f16780ffe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f167d97b640 1 -- 172.21.15.60:0/3696251891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16781070b0 msgr2=0x7f1678100340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f167d97b640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16781070b0 0x7f1678100340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f167d97b640 1 -- 172.21.15.60:0/3696251891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1678100bd0 con 0x7f1678101e50 2024-03-04T12:23:40.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f167d97b640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1678101e50 0x7f16781017c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f166c00e9f0 tx=0x7f166c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:40.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f166affd640 1 -- 172.21.15.60:0/3696251891 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c00cd80 con 0x7f1678101e50 2024-03-04T12:23:40.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f167fbdf640 1 -- 172.21.15.60:0/3696251891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16781a1790 con 0x7f1678101e50 2024-03-04T12:23:40.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f166affd640 1 -- 172.21.15.60:0/3696251891 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f166c005500 con 0x7f1678101e50 2024-03-04T12:23:40.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f166affd640 1 -- 172.21.15.60:0/3696251891 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f166c010690 con 0x7f1678101e50 2024-03-04T12:23:40.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.381+0000 7f167fbdf640 1 -- 172.21.15.60:0/3696251891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16781a1cb0 con 0x7f1678101e50 2024-03-04T12:23:40.390 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.382+0000 7f167fbdf640 1 -- 172.21.15.60:0/3696251891 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16780681b0 con 0x7f1678101e50 2024-03-04T12:23:40.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.384+0000 7f166affd640 1 -- 172.21.15.60:0/3696251891 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f166c0046c0 con 0x7f1678101e50 2024-03-04T12:23:40.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.385+0000 7f166affd640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f16640a1d80 0x7f16640a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:40.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.385+0000 7f166affd640 1 -- 172.21.15.60:0/3696251891 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f166c0ce790 con 0x7f1678101e50 2024-03-04T12:23:40.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.385+0000 7f167d17a640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f16640a1d80 0x7f16640a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:40.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.385+0000 7f167d17a640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f16640a1d80 0x7f16640a4240 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f167402c7b0 tx=0x7f1674005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:40.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.388+0000 7f166affd640 1 -- 172.21.15.60:0/3696251891 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f166c052d20 con 0x7f1678101e50 2024-03-04T12:23:40.559 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:23:40.234+0000 7f59f83c7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:23:40.559 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:23:40.234+0000 7f59f83c7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:23:40.559 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:23:40.234+0000 7f59f83c7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:23:40.559 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:23:40.234+0000 7f59f83c7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:23:40.559 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:23:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:23:40.234+0000 7f59f83c7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:23:40.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.600+0000 7f167fbdf640 1 -- 172.21.15.60:0/3696251891 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f1678061770 con 0x7f16640a1d80 2024-03-04T12:23:40.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.622+0000 7f166affd640 1 -- 172.21.15.60:0/3696251891 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1678061770 con 0x7f16640a1d80 2024-03-04T12:23:40.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.624+0000 7f167fbdf640 1 -- 172.21.15.60:0/3696251891 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f16640a1d80 msgr2=0x7f16640a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.624+0000 7f167fbdf640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f16640a1d80 0x7f16640a4240 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f167402c7b0 tx=0x7f1674005fd0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.624+0000 7f167fbdf640 1 -- 172.21.15.60:0/3696251891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1678101e50 msgr2=0x7f16781017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.624+0000 7f167fbdf640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1678101e50 0x7f16781017c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f166c00e9f0 tx=0x7f166c00edf0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.624+0000 7f167fbdf640 1 -- 172.21.15.60:0/3696251891 shutdown_connections 2024-03-04T12:23:40.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.624+0000 7f167fbdf640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f16640a1d80 0x7f16640a4240 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.624+0000 7f167fbdf640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16781070b0 0x7f1678100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.624+0000 7f167fbdf640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1678104780 0x7f16780ffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.624+0000 7f167fbdf640 1 --2- 172.21.15.60:0/3696251891 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1678101e50 0x7f16781017c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.624+0000 7f167fbdf640 1 -- 172.21.15.60:0/3696251891 >> 172.21.15.60:0/3696251891 conn(0x7f16780fb7c0 msgr2=0x7f16781028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:40.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.624+0000 7f167fbdf640 1 -- 172.21.15.60:0/3696251891 shutdown_connections 2024-03-04T12:23:40.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.624+0000 7f167fbdf640 1 -- 172.21.15.60:0/3696251891 wait complete. 2024-03-04T12:23:40.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-04T12:23:40.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.746+0000 7f9b65ed1640 1 -- 172.21.15.60:0/3357606866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60104720 msgr2=0x7f9b60106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.746+0000 7f9b65ed1640 1 --2- 172.21.15.60:0/3357606866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60104720 0x7f9b60106b10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009900 tx=0x7f9b4802bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:40.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.746+0000 7f9b65ed1640 1 -- 172.21.15.60:0/3357606866 shutdown_connections 2024-03-04T12:23:40.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.746+0000 7f9b65ed1640 1 --2- 172.21.15.60:0/3357606866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b60107050 0x7f9b60109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.746+0000 7f9b65ed1640 1 --2- 172.21.15.60:0/3357606866 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60104720 0x7f9b60106b10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.746+0000 7f9b65ed1640 1 --2- 172.21.15.60:0/3357606866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b60101df0 0x7f9b601041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.746+0000 7f9b65ed1640 1 -- 172.21.15.60:0/3357606866 >> 172.21.15.60:0/3357606866 conn(0x7f9b600fb760 msgr2=0x7f9b600fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:40.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.746+0000 7f9b65ed1640 1 -- 172.21.15.60:0/3357606866 shutdown_connections 2024-03-04T12:23:40.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.746+0000 7f9b65ed1640 1 -- 172.21.15.60:0/3357606866 wait complete. 2024-03-04T12:23:40.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.746+0000 7f9b65ed1640 1 Processor -- start 2024-03-04T12:23:40.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b65ed1640 1 -- start start 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b65ed1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60101df0 0x7f9b60101760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b65ed1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b60104720 0x7f9b600ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b65ed1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b60107050 0x7f9b601002e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b65ed1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6010d010 con 0x7f9b60101df0 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b65ed1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6010ce90 con 0x7f9b60107050 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b65ed1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6010d190 con 0x7f9b60104720 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b5f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60101df0 0x7f9b60101760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b5f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60101df0 0x7f9b60101760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37930/0 (socket says 172.21.15.60:37930) 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b5f7fe640 1 -- 172.21.15.60:0/2597835125 learned_addr learned my addr 172.21.15.60:0/2597835125 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b5effd640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b60104720 0x7f9b600ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b5ffff640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b60107050 0x7f9b601002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b5f7fe640 1 -- 172.21.15.60:0/2597835125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b60104720 msgr2=0x7f9b600ffda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b5f7fe640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b60104720 0x7f9b600ffda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b5f7fe640 1 -- 172.21.15.60:0/2597835125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b60107050 msgr2=0x7f9b601002e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b5f7fe640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b60107050 0x7f9b601002e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b5f7fe640 1 -- 172.21.15.60:0/2597835125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b60100b70 con 0x7f9b60101df0 2024-03-04T12:23:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.747+0000 7f9b5f7fe640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60101df0 0x7f9b60101760 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000ea80 tx=0x7f9b5000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.748+0000 7f9b5cff9640 1 -- 172.21.15.60:0/2597835125 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5000ce40 con 0x7f9b60101df0 2024-03-04T12:23:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.748+0000 7f9b5cff9640 1 -- 172.21.15.60:0/2597835125 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b50005020 con 0x7f9b60101df0 2024-03-04T12:23:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.748+0000 7f9b5cff9640 1 -- 172.21.15.60:0/2597835125 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b50011760 con 0x7f9b60101df0 2024-03-04T12:23:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.748+0000 7f9b65ed1640 1 -- 172.21.15.60:0/2597835125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b601a1750 con 0x7f9b60101df0 2024-03-04T12:23:40.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.748+0000 7f9b65ed1640 1 -- 172.21.15.60:0/2597835125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b601a1c20 con 0x7f9b60101df0 2024-03-04T12:23:40.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.749+0000 7f9b65ed1640 1 -- 172.21.15.60:0/2597835125 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b600681b0 con 0x7f9b60101df0 2024-03-04T12:23:40.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.750+0000 7f9b5cff9640 1 -- 172.21.15.60:0/2597835125 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 16) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f9b500051d0 con 0x7f9b60101df0 2024-03-04T12:23:40.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.750+0000 7f9b5cff9640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9b4c0a1d80 0x7f9b4c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:40.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.750+0000 7f9b5effd640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9b4c0a1d80 0x7f9b4c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:40.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.750+0000 7f9b5cff9640 1 -- 172.21.15.60:0/2597835125 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9b500ce6c0 con 0x7f9b60101df0 2024-03-04T12:23:40.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.751+0000 7f9b5effd640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9b4c0a1d80 0x7f9b4c0a4240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9b4802c170 tx=0x7f9b4802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:40.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.755+0000 7f9b5cff9640 1 -- 172.21.15.60:0/2597835125 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9b50016030 con 0x7f9b60101df0 2024-03-04T12:23:40.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.976+0000 7f9b65ed1640 1 -- 172.21.15.60:0/2597835125 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b60061770 con 0x7f9b4c0a1d80 2024-03-04T12:23:40.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.981+0000 7f9b5cff9640 1 -- 172.21.15.60:0/2597835125 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f9b60061770 con 0x7f9b4c0a1d80 2024-03-04T12:23:40.987 INFO:teuthology.orchestra.run.smithi060.stdout:/volumes/qa/sv_0/acfbd4ea-e7b0-4921-aacf-84e020bfae5b 2024-03-04T12:23:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.983+0000 7f9b65ed1640 1 -- 172.21.15.60:0/2597835125 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9b4c0a1d80 msgr2=0x7f9b4c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.983+0000 7f9b65ed1640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9b4c0a1d80 0x7f9b4c0a4240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9b4802c170 tx=0x7f9b4802df90 comp rx=0 tx=0).stop 2024-03-04T12:23:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.983+0000 7f9b65ed1640 1 -- 172.21.15.60:0/2597835125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60101df0 msgr2=0x7f9b60101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.983+0000 7f9b65ed1640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60101df0 0x7f9b60101760 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9b5000ea80 tx=0x7f9b5000ef40 comp rx=0 tx=0).stop 2024-03-04T12:23:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.983+0000 7f9b65ed1640 1 -- 172.21.15.60:0/2597835125 shutdown_connections 2024-03-04T12:23:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.983+0000 7f9b65ed1640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9b4c0a1d80 0x7f9b4c0a4240 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.983+0000 7f9b65ed1640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b60107050 0x7f9b601002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.983+0000 7f9b65ed1640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b60104720 0x7f9b600ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.983+0000 7f9b65ed1640 1 --2- 172.21.15.60:0/2597835125 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b60101df0 0x7f9b60101760 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.983+0000 7f9b65ed1640 1 -- 172.21.15.60:0/2597835125 >> 172.21.15.60:0/2597835125 conn(0x7f9b600fb760 msgr2=0x7f9b601026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:40.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.983+0000 7f9b65ed1640 1 -- 172.21.15.60:0/2597835125 shutdown_connections 2024-03-04T12:23:40.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:40.983+0000 7f9b65ed1640 1 -- 172.21.15.60:0/2597835125 wait complete. 2024-03-04T12:23:41.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-04T12:23:41.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.106+0000 7efcd94a7640 1 -- 172.21.15.60:0/2632149127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd4101440 msgr2=0x7efcd41018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.106+0000 7efcd94a7640 1 --2- 172.21.15.60:0/2632149127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd4101440 0x7efcd41018c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009900 tx=0x7efcc802bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.106+0000 7efcd94a7640 1 -- 172.21.15.60:0/2632149127 shutdown_connections 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.106+0000 7efcd94a7640 1 --2- 172.21.15.60:0/2632149127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcd4101e00 0x7efcd4108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.106+0000 7efcd94a7640 1 --2- 172.21.15.60:0/2632149127 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd4101440 0x7efcd41018c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.106+0000 7efcd94a7640 1 --2- 172.21.15.60:0/2632149127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efcd4100250 0x7efcd4100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.106+0000 7efcd94a7640 1 -- 172.21.15.60:0/2632149127 >> 172.21.15.60:0/2632149127 conn(0x7efcd40fb7c0 msgr2=0x7efcd40fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.106+0000 7efcd94a7640 1 -- 172.21.15.60:0/2632149127 shutdown_connections 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.106+0000 7efcd94a7640 1 -- 172.21.15.60:0/2632149127 wait complete. 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.107+0000 7efcd94a7640 1 Processor -- start 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.107+0000 7efcd94a7640 1 -- start start 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.107+0000 7efcd94a7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcd4100250 0x7efcd4069170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.107+0000 7efcd94a7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efcd4101440 0x7efcd419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.107+0000 7efcd94a7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd4101e00 0x7efcd41a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.107+0000 7efcd94a7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd410bee0 con 0x7efcd4101e00 2024-03-04T12:23:41.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.107+0000 7efcd94a7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd410bd60 con 0x7efcd4101440 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.107+0000 7efcd94a7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd410c060 con 0x7efcd4100250 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.107+0000 7efcd37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd4101e00 0x7efcd41a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.107+0000 7efcd37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd4101e00 0x7efcd41a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37958/0 (socket says 172.21.15.60:37958) 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.107+0000 7efcd37fe640 1 -- 172.21.15.60:0/2122150519 learned_addr learned my addr 172.21.15.60:0/2122150519 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcd27fc640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efcd4101440 0x7efcd419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcd2ffd640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcd4100250 0x7efcd4069170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcd37fe640 1 -- 172.21.15.60:0/2122150519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcd4100250 msgr2=0x7efcd4069170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcd37fe640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcd4100250 0x7efcd4069170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcd37fe640 1 -- 172.21.15.60:0/2122150519 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efcd4101440 msgr2=0x7efcd419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcd37fe640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efcd4101440 0x7efcd419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcd37fe640 1 -- 172.21.15.60:0/2122150519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd41a1440 con 0x7efcd4101e00 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcd2ffd640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcd4100250 0x7efcd4069170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcd37fe640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd4101e00 0x7efcd41a0c50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7efccc00cce0 tx=0x7efccc00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:41.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcbbfff640 1 -- 172.21.15.60:0/2122150519 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc00f450 con 0x7efcd4101e00 2024-03-04T12:23:41.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcbbfff640 1 -- 172.21.15.60:0/2122150519 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efccc00fad0 con 0x7efcd4101e00 2024-03-04T12:23:41.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcd94a7640 1 -- 172.21.15.60:0/2122150519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd41a16d0 con 0x7efcd4101e00 2024-03-04T12:23:41.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcbbfff640 1 -- 172.21.15.60:0/2122150519 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc018c10 con 0x7efcd4101e00 2024-03-04T12:23:41.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.108+0000 7efcd94a7640 1 -- 172.21.15.60:0/2122150519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd41a1ba0 con 0x7efcd4101e00 2024-03-04T12:23:41.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.109+0000 7efcd94a7640 1 -- 172.21.15.60:0/2122150519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd4107510 con 0x7efcd4101e00 2024-03-04T12:23:41.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.110+0000 7efcbbfff640 1 -- 172.21.15.60:0/2122150519 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7efccc00f5f0 con 0x7efcd4101e00 2024-03-04T12:23:41.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.111+0000 7efcbbfff640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efcbc0a1e40 0x7efcbc0a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:41.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.111+0000 7efcd2ffd640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efcbc0a1e40 0x7efcbc0a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:41.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.111+0000 7efcbbfff640 1 -- 172.21.15.60:0/2122150519 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7efccc0ce850 con 0x7efcd4101e00 2024-03-04T12:23:41.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.112+0000 7efcd2ffd640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efcbc0a1e40 0x7efcbc0a4300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efcd41012a0 tx=0x7efcc400a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:41.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.116+0000 7efcbbfff640 1 -- 172.21.15.60:0/2122150519 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efccc014030 con 0x7efcd4101e00 2024-03-04T12:23:41.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.324+0000 7efcd94a7640 1 -- 172.21.15.60:0/2122150519 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7efcd4061800 con 0x7efcbc0a1e40 2024-03-04T12:23:41.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.346+0000 7efcbbfff640 1 -- 172.21.15.60:0/2122150519 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efcd4061800 con 0x7efcbc0a1e40 2024-03-04T12:23:41.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.348+0000 7efcd94a7640 1 -- 172.21.15.60:0/2122150519 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efcbc0a1e40 msgr2=0x7efcbc0a4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.348+0000 7efcd94a7640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efcbc0a1e40 0x7efcbc0a4300 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efcd41012a0 tx=0x7efcc400a5e0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.348+0000 7efcd94a7640 1 -- 172.21.15.60:0/2122150519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd4101e00 msgr2=0x7efcd41a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.348+0000 7efcd94a7640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd4101e00 0x7efcd41a0c50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7efccc00cce0 tx=0x7efccc00a620 comp rx=0 tx=0).stop 2024-03-04T12:23:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.348+0000 7efcd94a7640 1 -- 172.21.15.60:0/2122150519 shutdown_connections 2024-03-04T12:23:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.348+0000 7efcd94a7640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efcbc0a1e40 0x7efcbc0a4300 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.348+0000 7efcd94a7640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcd4101e00 0x7efcd41a0c50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.348+0000 7efcd94a7640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efcd4101440 0x7efcd419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.348+0000 7efcd94a7640 1 --2- 172.21.15.60:0/2122150519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcd4100250 0x7efcd4069170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.348+0000 7efcd94a7640 1 -- 172.21.15.60:0/2122150519 >> 172.21.15.60:0/2122150519 conn(0x7efcd40fb7c0 msgr2=0x7efcd41046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.349+0000 7efcd94a7640 1 -- 172.21.15.60:0/2122150519 shutdown_connections 2024-03-04T12:23:41.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.349+0000 7efcd94a7640 1 -- 172.21.15.60:0/2122150519 wait complete. 2024-03-04T12:23:41.365 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-04T12:23:41.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.467+0000 7f991e02b640 1 -- 172.21.15.60:0/3517547547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9918101db0 msgr2=0x7f99181041a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.467+0000 7f991e02b640 1 --2- 172.21.15.60:0/3517547547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9918101db0 0x7f99181041a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9908009900 tx=0x7f990802bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:41.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.467+0000 7f991e02b640 1 -- 172.21.15.60:0/3517547547 shutdown_connections 2024-03-04T12:23:41.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.467+0000 7f991e02b640 1 --2- 172.21.15.60:0/3517547547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9918107010 0x7f9918109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.467+0000 7f991e02b640 1 --2- 172.21.15.60:0/3517547547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f99181046e0 0x7f9918106ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.467+0000 7f991e02b640 1 --2- 172.21.15.60:0/3517547547 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9918101db0 0x7f99181041a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.467+0000 7f991e02b640 1 -- 172.21.15.60:0/3517547547 >> 172.21.15.60:0/3517547547 conn(0x7f99180fb760 msgr2=0x7f99180fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.467+0000 7f991e02b640 1 -- 172.21.15.60:0/3517547547 shutdown_connections 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.467+0000 7f991e02b640 1 -- 172.21.15.60:0/3517547547 wait complete. 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f991e02b640 1 Processor -- start 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f991e02b640 1 -- start start 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f991e02b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9918101db0 0x7f9918197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f991e02b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99181046e0 0x7f9918197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f991e02b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9918107010 0x7f991819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f991e02b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991810cfc0 con 0x7f99181046e0 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f991e02b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991810ce40 con 0x7f9918101db0 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f991e02b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991810d140 con 0x7f9918107010 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f9916ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99181046e0 0x7f9918197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f9916ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99181046e0 0x7f9918197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37978/0 (socket says 172.21.15.60:37978) 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f9916ffd640 1 -- 172.21.15.60:0/2691234993 learned_addr learned my addr 172.21.15.60:0/2691234993 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f99177fe640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9918101db0 0x7f9918197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:41.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f9917fff640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9918107010 0x7f991819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:41.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f9916ffd640 1 -- 172.21.15.60:0/2691234993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9918107010 msgr2=0x7f991819eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f9916ffd640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9918107010 0x7f991819eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f9916ffd640 1 -- 172.21.15.60:0/2691234993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9918101db0 msgr2=0x7f9918197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f9916ffd640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9918101db0 0x7f9918197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.468+0000 7f9916ffd640 1 -- 172.21.15.60:0/2691234993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991819f210 con 0x7f99181046e0 2024-03-04T12:23:41.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.469+0000 7f9916ffd640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99181046e0 0x7f9918197b80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f990c005fd0 tx=0x7f990c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:41.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.469+0000 7f9914ff9640 1 -- 172.21.15.60:0/2691234993 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c017070 con 0x7f99181046e0 2024-03-04T12:23:41.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.469+0000 7f9914ff9640 1 -- 172.21.15.60:0/2691234993 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f990c009c30 con 0x7f99181046e0 2024-03-04T12:23:41.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.469+0000 7f991e02b640 1 -- 172.21.15.60:0/2691234993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991819f500 con 0x7f99181046e0 2024-03-04T12:23:41.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.469+0000 7f9914ff9640 1 -- 172.21.15.60:0/2691234993 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c010460 con 0x7f99181046e0 2024-03-04T12:23:41.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.469+0000 7f991e02b640 1 -- 172.21.15.60:0/2691234993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991819faa0 con 0x7f99181046e0 2024-03-04T12:23:41.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.470+0000 7f991e02b640 1 -- 172.21.15.60:0/2691234993 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99180fd250 con 0x7f99181046e0 2024-03-04T12:23:41.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.471+0000 7f9914ff9640 1 -- 172.21.15.60:0/2691234993 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f990c003e70 con 0x7f99181046e0 2024-03-04T12:23:41.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.472+0000 7f9914ff9640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99000a1e90 0x7f99000a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:41.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.472+0000 7f99177fe640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99000a1e90 0x7f99000a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:41.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.472+0000 7f9914ff9640 1 -- 172.21.15.60:0/2691234993 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f990c0cde50 con 0x7f99181046e0 2024-03-04T12:23:41.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.472+0000 7f99177fe640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99000a1e90 0x7f99000a4350 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f990802c7b0 tx=0x7f9908005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:41.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.476+0000 7f9914ff9640 1 -- 172.21.15.60:0/2691234993 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f990c020030 con 0x7f99181046e0 2024-03-04T12:23:41.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.690+0000 7f991e02b640 1 -- 172.21.15.60:0/2691234993 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9918061800 con 0x7f99000a1e90 2024-03-04T12:23:41.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.694+0000 7f9914ff9640 1 -- 172.21.15.60:0/2691234993 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f9918061800 con 0x7f99000a1e90 2024-03-04T12:23:41.700 INFO:teuthology.orchestra.run.smithi060.stdout:/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1 2024-03-04T12:23:41.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.696+0000 7f991e02b640 1 -- 172.21.15.60:0/2691234993 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99000a1e90 msgr2=0x7f99000a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.696+0000 7f991e02b640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99000a1e90 0x7f99000a4350 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f990802c7b0 tx=0x7f9908005fd0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.696+0000 7f991e02b640 1 -- 172.21.15.60:0/2691234993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99181046e0 msgr2=0x7f9918197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.696+0000 7f991e02b640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99181046e0 0x7f9918197b80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f990c005fd0 tx=0x7f990c00dde0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.696+0000 7f991e02b640 1 -- 172.21.15.60:0/2691234993 shutdown_connections 2024-03-04T12:23:41.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.696+0000 7f991e02b640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99000a1e90 0x7f99000a4350 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.696+0000 7f991e02b640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9918107010 0x7f991819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.696+0000 7f991e02b640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f99181046e0 0x7f9918197b80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.696+0000 7f991e02b640 1 --2- 172.21.15.60:0/2691234993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9918101db0 0x7f9918197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.696+0000 7f991e02b640 1 -- 172.21.15.60:0/2691234993 >> 172.21.15.60:0/2691234993 conn(0x7f99180fb760 msgr2=0x7f9918102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:41.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.696+0000 7f991e02b640 1 -- 172.21.15.60:0/2691234993 shutdown_connections 2024-03-04T12:23:41.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.696+0000 7f991e02b640 1 -- 172.21.15.60:0/2691234993 wait complete. 2024-03-04T12:23:41.713 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:23:41.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.809+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/3872224805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdf4100260 msgr2=0x7fcdf4100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.809+0000 7fcdf9bd4640 1 --2- 172.21.15.60:0/3872224805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdf4100260 0x7fcdf4100680 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009900 tx=0x7fcde402bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:41.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/3872224805 shutdown_connections 2024-03-04T12:23:41.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 --2- 172.21.15.60:0/3872224805 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdf4101e10 0x7fcdf41086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 --2- 172.21.15.60:0/3872224805 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf4101450 0x7fcdf41018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 --2- 172.21.15.60:0/3872224805 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdf4100260 0x7fcdf4100680 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/3872224805 >> 172.21.15.60:0/3872224805 conn(0x7fcdf40fb7d0 msgr2=0x7fcdf40fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:41.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/3872224805 shutdown_connections 2024-03-04T12:23:41.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/3872224805 wait complete. 2024-03-04T12:23:41.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 Processor -- start 2024-03-04T12:23:41.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 -- start start 2024-03-04T12:23:41.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdf4100260 0x7fcdf4196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:41.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf4101450 0x7fcdf4196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdf4101e10 0x7fcdf419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf410bef0 con 0x7fcdf4101e10 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf410bd70 con 0x7fcdf4101450 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.810+0000 7fcdf9bd4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf410c070 con 0x7fcdf4100260 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdf4101e10 0x7fcdf419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdf4101e10 0x7fcdf419daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37994/0 (socket says 172.21.15.60:37994) 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf3fff640 1 -- 172.21.15.60:0/797153292 learned_addr learned my addr 172.21.15.60:0/797153292 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf2ffd640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf4101450 0x7fcdf4196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf37fe640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdf4100260 0x7fcdf4196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf3fff640 1 -- 172.21.15.60:0/797153292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdf4100260 msgr2=0x7fcdf4196540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf3fff640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdf4100260 0x7fcdf4196540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf3fff640 1 -- 172.21.15.60:0/797153292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf4101450 msgr2=0x7fcdf4196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf3fff640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf4101450 0x7fcdf4196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf3fff640 1 -- 172.21.15.60:0/797153292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf419e160 con 0x7fcdf4101e10 2024-03-04T12:23:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf3fff640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdf4101e10 0x7fcdf419daf0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00eca0 tx=0x7fcdec00b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:41.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf0ff9640 1 -- 172.21.15.60:0/797153292 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec004b20 con 0x7fcdf4101e10 2024-03-04T12:23:41.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf0ff9640 1 -- 172.21.15.60:0/797153292 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcdec004cc0 con 0x7fcdf4101e10 2024-03-04T12:23:41.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/797153292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf4068660 con 0x7fcdf4101e10 2024-03-04T12:23:41.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf0ff9640 1 -- 172.21.15.60:0/797153292 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec017650 con 0x7fcdf4101e10 2024-03-04T12:23:41.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.811+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/797153292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf4068bb0 con 0x7fcdf4101e10 2024-03-04T12:23:41.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.812+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/797153292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf40681b0 con 0x7fcdf4101e10 2024-03-04T12:23:41.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.813+0000 7fcdf0ff9640 1 -- 172.21.15.60:0/797153292 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fcdec005020 con 0x7fcdf4101e10 2024-03-04T12:23:41.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.814+0000 7fcdf0ff9640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcddc0a1e10 0x7fcddc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:41.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.814+0000 7fcdf37fe640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcddc0a1e10 0x7fcddc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:41.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.814+0000 7fcdf0ff9640 1 -- 172.21.15.60:0/797153292 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcdec0cfc40 con 0x7fcdf4101e10 2024-03-04T12:23:41.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.815+0000 7fcdf37fe640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcddc0a1e10 0x7fcddc0a42d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcde402c7b0 tx=0x7fcde4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:41.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:41.819+0000 7fcdf0ff9640 1 -- 172.21.15.60:0/797153292 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcdec0d6080 con 0x7fcdf4101e10 2024-03-04T12:23:42.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.040+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/797153292 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdf41071b0 con 0x7fcdf4101e10 2024-03-04T12:23:42.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.040+0000 7fcdf0ff9640 1 -- 172.21.15.60:0/797153292 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11373 (secure 0 0 0) 0x7fcdec09ccd0 con 0x7fcdf4101e10 2024-03-04T12:23:42.046 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:42.046 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:23:41.896895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:23:42.047 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 28 2024-03-04T12:23:42.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.042+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/797153292 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcddc0a1e10 msgr2=0x7fcddc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.042+0000 7fcdf9bd4640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcddc0a1e10 0x7fcddc0a42d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcde402c7b0 tx=0x7fcde4005fd0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.042+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/797153292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdf4101e10 msgr2=0x7fcdf419daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.042+0000 7fcdf9bd4640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdf4101e10 0x7fcdf419daf0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00eca0 tx=0x7fcdec00b690 comp rx=0 tx=0).stop 2024-03-04T12:23:42.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.042+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/797153292 shutdown_connections 2024-03-04T12:23:42.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.042+0000 7fcdf9bd4640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcddc0a1e10 0x7fcddc0a42d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.042+0000 7fcdf9bd4640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcdf4101e10 0x7fcdf419daf0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.042+0000 7fcdf9bd4640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf4101450 0x7fcdf4196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.042+0000 7fcdf9bd4640 1 --2- 172.21.15.60:0/797153292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdf4100260 0x7fcdf4196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.042+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/797153292 >> 172.21.15.60:0/797153292 conn(0x7fcdf40fb7d0 msgr2=0x7fcdf4104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:42.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.042+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/797153292 shutdown_connections 2024-03-04T12:23:42.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.043+0000 7fcdf9bd4640 1 -- 172.21.15.60:0/797153292 wait complete. 2024-03-04T12:23:42.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:41 smithi060 ceph-mon[32745]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 1 op/s 2024-03-04T12:23:42.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:41 smithi060 ceph-mon[32745]: from='client.24688 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:42.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:41 smithi060 ceph-mon[32745]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:42.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:41 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] up:active 2024-03-04T12:23:42.058 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:41 smithi060 ceph-mon[32745]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-04T12:23:42.059 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:41 smithi060 ceph-mon[32745]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-04T12:23:42.059 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:41 smithi060 ceph-mon[32745]: Cluster is now healthy 2024-03-04T12:23:42.059 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:41 smithi060 ceph-mon[32745]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=k=up:active} 3 up:standby-replay 5 up:standby 2024-03-04T12:23:42.059 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:41 smithi060 ceph-mon[32745]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-03-04T12:23:42.059 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:41 smithi060 ceph-mon[32745]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-03-04T12:23:42.059 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:41 smithi060 ceph-mon[32745]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:42.059 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:41 smithi060 ceph-mon[32745]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-04T12:23:42.060 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:23:42.101 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b[67411]: 2024-03-04T12:23:41.751+0000 7ff844c68700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-04T12:23:42.102 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b[67411]: 2024-03-04T12:23:41.751+0000 7ff844c68700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-04T12:23:42.102 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b[67411]: 2024-03-04T12:23:41.751+0000 7ff844c68700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-04T12:23:42.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-mon[36717]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 1 op/s 2024-03-04T12:23:42.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-mon[36717]: from='client.24688 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:42.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-mon[36717]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:42.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] up:active 2024-03-04T12:23:42.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-mon[36717]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-04T12:23:42.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-mon[36717]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-04T12:23:42.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-mon[36717]: Cluster is now healthy 2024-03-04T12:23:42.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-mon[36717]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=k=up:active} 3 up:standby-replay 5 up:standby 2024-03-04T12:23:42.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-mon[36717]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-03-04T12:23:42.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-mon[36717]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-03-04T12:23:42.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-mon[36717]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:42.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:41 smithi063 ceph-mon[36717]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-04T12:23:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.156+0000 7fd90c37e640 1 -- 172.21.15.60:0/1531544324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd904101e10 msgr2=0x7fd9041086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.156+0000 7fd90c37e640 1 --2- 172.21.15.60:0/1531544324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd904101e10 0x7fd9041086a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd900009ca0 tx=0x7fd90002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.156+0000 7fd90c37e640 1 -- 172.21.15.60:0/1531544324 shutdown_connections 2024-03-04T12:23:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.156+0000 7fd90c37e640 1 --2- 172.21.15.60:0/1531544324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd904101e10 0x7fd9041086a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.156+0000 7fd90c37e640 1 --2- 172.21.15.60:0/1531544324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd904101450 0x7fd9041018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.156+0000 7fd90c37e640 1 --2- 172.21.15.60:0/1531544324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd904100260 0x7fd904100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.156+0000 7fd90c37e640 1 -- 172.21.15.60:0/1531544324 >> 172.21.15.60:0/1531544324 conn(0x7fd9040fb7f0 msgr2=0x7fd9040fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:42.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.156+0000 7fd90c37e640 1 -- 172.21.15.60:0/1531544324 shutdown_connections 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.156+0000 7fd90c37e640 1 -- 172.21.15.60:0/1531544324 wait complete. 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd90c37e640 1 Processor -- start 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd90c37e640 1 -- start start 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd90c37e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd904100260 0x7fd904197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd90c37e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd904101450 0x7fd904197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd90c37e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd904101e10 0x7fd90419eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd90c37e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90410bef0 con 0x7fd904101450 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd90c37e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90410bd70 con 0x7fd904101e10 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd90c37e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90410c070 con 0x7fd904100260 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd909919640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd904101450 0x7fd904197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd909919640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd904101450 0x7fd904197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38018/0 (socket says 172.21.15.60:38018) 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd909919640 1 -- 172.21.15.60:0/1222187001 learned_addr learned my addr 172.21.15.60:0/1222187001 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd90a11a640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd904100260 0x7fd904197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd909919640 1 -- 172.21.15.60:0/1222187001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd904100260 msgr2=0x7fd904197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd909919640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd904100260 0x7fd904197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd90a91b640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd904101e10 0x7fd90419eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd909919640 1 -- 172.21.15.60:0/1222187001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd904101e10 msgr2=0x7fd90419eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd909919640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd904101e10 0x7fd90419eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.157+0000 7fd909919640 1 -- 172.21.15.60:0/1222187001 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90419f1b0 con 0x7fd904101450 2024-03-04T12:23:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.158+0000 7fd909919640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd904101450 0x7fd904197ad0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400c890 tx=0x7fd8f400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.158+0000 7fd8fb7fe640 1 -- 172.21.15.60:0/1222187001 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4004220 con 0x7fd904101450 2024-03-04T12:23:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.158+0000 7fd90c37e640 1 -- 172.21.15.60:0/1222187001 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90419f4a0 con 0x7fd904101450 2024-03-04T12:23:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.158+0000 7fd8fb7fe640 1 -- 172.21.15.60:0/1222187001 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8f40043c0 con 0x7fd904101450 2024-03-04T12:23:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.158+0000 7fd8fb7fe640 1 -- 172.21.15.60:0/1222187001 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4004a00 con 0x7fd904101450 2024-03-04T12:23:42.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.158+0000 7fd90c37e640 1 -- 172.21.15.60:0/1222187001 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90419f9f0 con 0x7fd904101450 2024-03-04T12:23:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.159+0000 7fd90c37e640 1 -- 172.21.15.60:0/1222187001 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90419fcb0 con 0x7fd904101450 2024-03-04T12:23:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.160+0000 7fd8fb7fe640 1 -- 172.21.15.60:0/1222187001 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fd8f40079b0 con 0x7fd904101450 2024-03-04T12:23:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.160+0000 7fd8fb7fe640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8f00a1e10 0x7fd8f00a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.161+0000 7fd8fb7fe640 1 -- 172.21.15.60:0/1222187001 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd8f40cdf40 con 0x7fd904101450 2024-03-04T12:23:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.161+0000 7fd90a11a640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8f00a1e10 0x7fd8f00a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:42.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.161+0000 7fd90a11a640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8f00a1e10 0x7fd8f00a42d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc005fd0 tx=0x7fd8fc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:42.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.165+0000 7fd8fb7fe640 1 -- 172.21.15.60:0/1222187001 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd8f4014030 con 0x7fd904101450 2024-03-04T12:23:42.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:41 smithi086 ceph-mon[36851]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 1 op/s 2024-03-04T12:23:42.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:41 smithi086 ceph-mon[36851]: from='client.24688 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:42.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:41 smithi086 ceph-mon[36851]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:42.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:41 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] up:active 2024-03-04T12:23:42.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:41 smithi086 ceph-mon[36851]: daemon mds.l assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-04T12:23:42.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:41 smithi086 ceph-mon[36851]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-04T12:23:42.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:41 smithi086 ceph-mon[36851]: Cluster is now healthy 2024-03-04T12:23:42.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:41 smithi086 ceph-mon[36851]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=k=up:active} 3 up:standby-replay 5 up:standby 2024-03-04T12:23:42.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:41 smithi086 ceph-mon[36851]: fsmap cephfs:5 3 up:standby-replay 1 up:creating 4 up:standby 4 up:active 2024-03-04T12:23:42.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:41 smithi086 ceph-mon[36851]: daemon mds.l is now active in filesystem cephfs as rank 4 2024-03-04T12:23:42.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:41 smithi086 ceph-mon[36851]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:42.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:41 smithi086 ceph-mon[36851]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-04T12:23:42.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.346+0000 7fd90c37e640 1 -- 172.21.15.60:0/1222187001 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd90419fe80 con 0x7fd904101450 2024-03-04T12:23:42.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.348+0000 7fd8fb7fe640 1 -- 172.21.15.60:0/1222187001 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23997 (secure 0 0 0) 0x7fd8f409afd0 con 0x7fd904101450 2024-03-04T12:23:42.354 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:42.354 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":89,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:23:17.289925+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:23:42.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.350+0000 7fd90c37e640 1 -- 172.21.15.60:0/1222187001 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8f00a1e10 msgr2=0x7fd8f00a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.350+0000 7fd90c37e640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8f00a1e10 0x7fd8f00a42d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc005fd0 tx=0x7fd8fc005850 comp rx=0 tx=0).stop 2024-03-04T12:23:42.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.350+0000 7fd90c37e640 1 -- 172.21.15.60:0/1222187001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd904101450 msgr2=0x7fd904197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.350+0000 7fd90c37e640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd904101450 0x7fd904197ad0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400c890 tx=0x7fd8f400cd50 comp rx=0 tx=0).stop 2024-03-04T12:23:42.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.350+0000 7fd90c37e640 1 -- 172.21.15.60:0/1222187001 shutdown_connections 2024-03-04T12:23:42.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.350+0000 7fd90c37e640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd8f00a1e10 0x7fd8f00a42d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.350+0000 7fd90c37e640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd904101e10 0x7fd90419eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.350+0000 7fd90c37e640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd904101450 0x7fd904197ad0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.350+0000 7fd90c37e640 1 --2- 172.21.15.60:0/1222187001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd904100260 0x7fd904197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.350+0000 7fd90c37e640 1 -- 172.21.15.60:0/1222187001 >> 172.21.15.60:0/1222187001 conn(0x7fd9040fb7f0 msgr2=0x7fd9040fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:42.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.350+0000 7fd90c37e640 1 -- 172.21.15.60:0/1222187001 shutdown_connections 2024-03-04T12:23:42.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.350+0000 7fd90c37e640 1 -- 172.21.15.60:0/1222187001 wait complete. 2024-03-04T12:23:42.367 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-04T12:23:42.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T12:23:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.464+0000 7f21b8d9e640 1 -- 172.21.15.60:0/123397302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b4101de0 msgr2=0x7f21b41041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.464+0000 7f21b8d9e640 1 --2- 172.21.15.60:0/123397302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b4101de0 0x7f21b41041d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009900 tx=0x7f21ac02bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.465+0000 7f21b8d9e640 1 -- 172.21.15.60:0/123397302 shutdown_connections 2024-03-04T12:23:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.465+0000 7f21b8d9e640 1 --2- 172.21.15.60:0/123397302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21b4107040 0x7f21b4109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.465+0000 7f21b8d9e640 1 --2- 172.21.15.60:0/123397302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21b4104710 0x7f21b4106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.465+0000 7f21b8d9e640 1 --2- 172.21.15.60:0/123397302 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b4101de0 0x7f21b41041d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.465+0000 7f21b8d9e640 1 -- 172.21.15.60:0/123397302 >> 172.21.15.60:0/123397302 conn(0x7f21b40fb7d0 msgr2=0x7f21b40fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.465+0000 7f21b8d9e640 1 -- 172.21.15.60:0/123397302 shutdown_connections 2024-03-04T12:23:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.465+0000 7f21b8d9e640 1 -- 172.21.15.60:0/123397302 wait complete. 2024-03-04T12:23:42.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.465+0000 7f21b8d9e640 1 Processor -- start 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.465+0000 7f21b8d9e640 1 -- start start 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.465+0000 7f21b8d9e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b4101de0 0x7f21b4197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b8d9e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21b4104710 0x7f21b4197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b8d9e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21b4107040 0x7f21b419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b8d9e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b410cff0 con 0x7f21b4101de0 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b8d9e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b410ce70 con 0x7f21b4104710 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b8d9e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b410d170 con 0x7f21b4107040 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b4101de0 0x7f21b4197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b4101de0 0x7f21b4197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38040/0 (socket says 172.21.15.60:38040) 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b259c640 1 -- 172.21.15.60:0/1292948124 learned_addr learned my addr 172.21.15.60:0/1292948124 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b2d9d640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21b4107040 0x7f21b419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b259c640 1 -- 172.21.15.60:0/1292948124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21b4107040 msgr2=0x7f21b419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b259c640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21b4107040 0x7f21b419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b259c640 1 -- 172.21.15.60:0/1292948124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21b4104710 msgr2=0x7f21b4197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b259c640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21b4104710 0x7f21b4197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b259c640 1 -- 172.21.15.60:0/1292948124 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b419f2b0 con 0x7f21b4101de0 2024-03-04T12:23:42.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b259c640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b4101de0 0x7f21b4197690 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009230 tx=0x7f21ac005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:42.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21ab7fe640 1 -- 172.21.15.60:0/1292948124 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac041070 con 0x7f21b4101de0 2024-03-04T12:23:42.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21ab7fe640 1 -- 172.21.15.60:0/1292948124 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21ac0054e0 con 0x7f21b4101de0 2024-03-04T12:23:42.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b8d9e640 1 -- 172.21.15.60:0/1292948124 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b419f540 con 0x7f21b4101de0 2024-03-04T12:23:42.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21ab7fe640 1 -- 172.21.15.60:0/1292948124 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac034440 con 0x7f21b4101de0 2024-03-04T12:23:42.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.466+0000 7f21b8d9e640 1 -- 172.21.15.60:0/1292948124 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b419fa30 con 0x7f21b4101de0 2024-03-04T12:23:42.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.467+0000 7f21b8d9e640 1 -- 172.21.15.60:0/1292948124 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b40fd2c0 con 0x7f21b4101de0 2024-03-04T12:23:42.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.468+0000 7f21ab7fe640 1 -- 172.21.15.60:0/1292948124 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f21ac044070 con 0x7f21b4101de0 2024-03-04T12:23:42.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.469+0000 7f21ab7fe640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21a00a1e10 0x7f21a00a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:42.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.469+0000 7f21b1d9b640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21a00a1e10 0x7f21a00a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:42.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.469+0000 7f21ab7fe640 1 -- 172.21.15.60:0/1292948124 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f21ac0efe00 con 0x7f21b4101de0 2024-03-04T12:23:42.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.469+0000 7f21b1d9b640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21a00a1e10 0x7f21a00a42d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f219c005fd0 tx=0x7f219c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:42.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.473+0000 7f21ab7fe640 1 -- 172.21.15.60:0/1292948124 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f21ac0bcfa0 con 0x7f21b4101de0 2024-03-04T12:23:42.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.652+0000 7f21b8d9e640 1 -- 172.21.15.60:0/1292948124 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21b4061770 con 0x7f21a00a1e10 2024-03-04T12:23:42.667 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.662+0000 7f21ab7fe640 1 -- 172.21.15.60:0/1292948124 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340564 (secure 0 0 0) 0x7f21b4061770 con 0x7f21a00a1e10 2024-03-04T12:23:42.672 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:42.674 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-04T12:23:42.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.665+0000 7f21b8d9e640 1 -- 172.21.15.60:0/1292948124 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21a00a1e10 msgr2=0x7f21a00a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.665+0000 7f21b8d9e640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21a00a1e10 0x7f21a00a42d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f219c005fd0 tx=0x7f219c005850 comp rx=0 tx=0).stop 2024-03-04T12:23:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.665+0000 7f21b8d9e640 1 -- 172.21.15.60:0/1292948124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b4101de0 msgr2=0x7f21b4197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.665+0000 7f21b8d9e640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b4101de0 0x7f21b4197690 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f21ac009230 tx=0x7f21ac005dd0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.665+0000 7f21b8d9e640 1 -- 172.21.15.60:0/1292948124 shutdown_connections 2024-03-04T12:23:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.665+0000 7f21b8d9e640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21a00a1e10 0x7f21a00a42d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.665+0000 7f21b8d9e640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21b4107040 0x7f21b419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.665+0000 7f21b8d9e640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21b4104710 0x7f21b4197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.665+0000 7f21b8d9e640 1 --2- 172.21.15.60:0/1292948124 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f21b4101de0 0x7f21b4197690 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.665+0000 7f21b8d9e640 1 -- 172.21.15.60:0/1292948124 >> 172.21.15.60:0/1292948124 conn(0x7f21b40fb7d0 msgr2=0x7f21b41026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.666+0000 7f21b8d9e640 1 -- 172.21.15.60:0/1292948124 shutdown_connections 2024-03-04T12:23:42.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.666+0000 7f21b8d9e640 1 -- 172.21.15.60:0/1292948124 wait complete. 2024-03-04T12:23:42.682 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-03-04T12:23:42.159947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595142,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":137,"num_read_kb":120,"num_write":268,"num_write_kb":2966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381556,"kb_used_data":11572,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560268,"statfs":{"total":1151940427776,"available":1151549714432,"internally_reserved":0,"allocated":11849728,"data_stored":7513922,"data_compressed":285408,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":275,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":40,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003371"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678736+0000","last_change":"2024-03-04T12:23:12.926673+0000","last_active":"2024-03-04T12:23:18.678736+0000","last_peered":"2024-03-04T12:23:18.678736+0000","last_clean":"2024-03-04T12:23:18.678736+0000","last_became_active":"2024-03-04T12:23:12.926299+0000","last_became_peered":"2024-03-04T12:23:12.926299+0000","last_unstale":"2024-03-04T12:23:18.678736+0000","last_undegraded":"2024-03-04T12:23:18.678736+0000","last_fullsized":"2024-03-04T12:23:18.678736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:19.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302937+0000","last_change":"2024-03-04T12:23:12.921758+0000","last_active":"2024-03-04T12:23:19.302937+0000","last_peered":"2024-03-04T12:23:19.302937+0000","last_clean":"2024-03-04T12:23:19.302937+0000","last_became_active":"2024-03-04T12:23:12.921317+0000","last_became_peered":"2024-03-04T12:23:12.921317+0000","last_unstale":"2024-03-04T12:23:19.302937+0000","last_undegraded":"2024-03-04T12:23:19.302937+0000","last_fullsized":"2024-03-04T12:23:19.302937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:20:25.179004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302592+0000","last_change":"2024-03-04T12:23:12.925771+0000","last_active":"2024-03-04T12:23:19.302592+0000","last_peered":"2024-03-04T12:23:19.302592+0000","last_clean":"2024-03-04T12:23:19.302592+0000","last_became_active":"2024-03-04T12:23:12.925447+0000","last_became_peered":"2024-03-04T12:23:12.925447+0000","last_unstale":"2024-03-04T12:23:19.302592+0000","last_undegraded":"2024-03-04T12:23:19.302592+0000","last_fullsized":"2024-03-04T12:23:19.302592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:12.380831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979569+0000","last_change":"2024-03-04T12:23:12.915973+0000","last_active":"2024-03-04T12:23:17.979569+0000","last_peered":"2024-03-04T12:23:17.979569+0000","last_clean":"2024-03-04T12:23:17.979569+0000","last_became_active":"2024-03-04T12:23:12.914914+0000","last_became_peered":"2024-03-04T12:23:12.914914+0000","last_unstale":"2024-03-04T12:23:17.979569+0000","last_undegraded":"2024-03-04T12:23:17.979569+0000","last_fullsized":"2024-03-04T12:23:17.979569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:52:09.908889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079975+0000","last_change":"2024-03-04T12:23:12.921118+0000","last_active":"2024-03-04T12:23:19.079975+0000","last_peered":"2024-03-04T12:23:19.079975+0000","last_clean":"2024-03-04T12:23:19.079975+0000","last_became_active":"2024-03-04T12:23:12.920127+0000","last_became_peered":"2024-03-04T12:23:12.920127+0000","last_unstale":"2024-03-04T12:23:19.079975+0000","last_undegraded":"2024-03-04T12:23:19.079975+0000","last_fullsized":"2024-03-04T12:23:19.079975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:22:10.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079522+0000","last_change":"2024-03-04T12:23:12.916751+0000","last_active":"2024-03-04T12:23:19.079522+0000","last_peered":"2024-03-04T12:23:19.079522+0000","last_clean":"2024-03-04T12:23:19.079522+0000","last_became_active":"2024-03-04T12:23:12.916404+0000","last_became_peered":"2024-03-04T12:23:12.916404+0000","last_unstale":"2024-03-04T12:23:19.079522+0000","last_undegraded":"2024-03-04T12:23:19.079522+0000","last_fullsized":"2024-03-04T12:23:19.079522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:39.998992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930022+0000","last_change":"2024-03-04T12:23:12.909105+0000","last_active":"2024-03-04T12:23:17.930022+0000","last_peered":"2024-03-04T12:23:17.930022+0000","last_clean":"2024-03-04T12:23:17.930022+0000","last_became_active":"2024-03-04T12:23:12.908743+0000","last_became_peered":"2024-03-04T12:23:12.908743+0000","last_unstale":"2024-03-04T12:23:17.930022+0000","last_undegraded":"2024-03-04T12:23:17.930022+0000","last_fullsized":"2024-03-04T12:23:17.930022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:02:44.159796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816254+0000","last_change":"2024-03-04T12:23:12.917627+0000","last_active":"2024-03-04T12:23:18.816254+0000","last_peered":"2024-03-04T12:23:18.816254+0000","last_clean":"2024-03-04T12:23:18.816254+0000","last_became_active":"2024-03-04T12:23:12.917279+0000","last_became_peered":"2024-03-04T12:23:12.917279+0000","last_unstale":"2024-03-04T12:23:18.816254+0000","last_undegraded":"2024-03-04T12:23:18.816254+0000","last_fullsized":"2024-03-04T12:23:18.816254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:33:03.487273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079679+0000","last_change":"2024-03-04T12:23:12.922601+0000","last_active":"2024-03-04T12:23:19.079679+0000","last_peered":"2024-03-04T12:23:19.079679+0000","last_clean":"2024-03-04T12:23:19.079679+0000","last_became_active":"2024-03-04T12:23:12.922037+0000","last_became_peered":"2024-03-04T12:23:12.922037+0000","last_unstale":"2024-03-04T12:23:19.079679+0000","last_undegraded":"2024-03-04T12:23:19.079679+0000","last_fullsized":"2024-03-04T12:23:19.079679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:32.076439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080427+0000","last_change":"2024-03-04T12:23:12.916308+0000","last_active":"2024-03-04T12:23:19.080427+0000","last_peered":"2024-03-04T12:23:19.080427+0000","last_clean":"2024-03-04T12:23:19.080427+0000","last_became_active":"2024-03-04T12:23:12.915897+0000","last_became_peered":"2024-03-04T12:23:12.915897+0000","last_unstale":"2024-03-04T12:23:19.080427+0000","last_undegraded":"2024-03-04T12:23:19.080427+0000","last_fullsized":"2024-03-04T12:23:19.080427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:15:53.133053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930704+0000","last_change":"2024-03-04T12:23:12.930700+0000","last_active":"2024-03-04T12:23:17.930704+0000","last_peered":"2024-03-04T12:23:17.930704+0000","last_clean":"2024-03-04T12:23:17.930704+0000","last_became_active":"2024-03-04T12:23:12.930328+0000","last_became_peered":"2024-03-04T12:23:12.930328+0000","last_unstale":"2024-03-04T12:23:17.930704+0000","last_undegraded":"2024-03-04T12:23:17.930704+0000","last_fullsized":"2024-03-04T12:23:17.930704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:29.990527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979581+0000","last_change":"2024-03-04T12:23:12.915471+0000","last_active":"2024-03-04T12:23:17.979581+0000","last_peered":"2024-03-04T12:23:17.979581+0000","last_clean":"2024-03-04T12:23:17.979581+0000","last_became_active":"2024-03-04T12:23:12.913627+0000","last_became_peered":"2024-03-04T12:23:12.913627+0000","last_unstale":"2024-03-04T12:23:17.979581+0000","last_undegraded":"2024-03-04T12:23:17.979581+0000","last_fullsized":"2024-03-04T12:23:17.979581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:42:56.762782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079647+0000","last_change":"2024-03-04T12:23:12.899143+0000","last_active":"2024-03-04T12:23:19.079647+0000","last_peered":"2024-03-04T12:23:19.079647+0000","last_clean":"2024-03-04T12:23:19.079647+0000","last_became_active":"2024-03-04T12:23:12.898498+0000","last_became_peered":"2024-03-04T12:23:12.898498+0000","last_unstale":"2024-03-04T12:23:19.079647+0000","last_undegraded":"2024-03-04T12:23:19.079647+0000","last_fullsized":"2024-03-04T12:23:19.079647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:33.840683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302557+0000","last_change":"2024-03-04T12:23:12.920873+0000","last_active":"2024-03-04T12:23:19.302557+0000","last_peered":"2024-03-04T12:23:19.302557+0000","last_clean":"2024-03-04T12:23:19.302557+0000","last_became_active":"2024-03-04T12:23:12.919080+0000","last_became_peered":"2024-03-04T12:23:12.919080+0000","last_unstale":"2024-03-04T12:23:19.302557+0000","last_undegraded":"2024-03-04T12:23:19.302557+0000","last_fullsized":"2024-03-04T12:23:19.302557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:47:26.710624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302576+0000","last_change":"2024-03-04T12:23:12.918345+0000","last_active":"2024-03-04T12:23:19.302576+0000","last_peered":"2024-03-04T12:23:19.302576+0000","last_clean":"2024-03-04T12:23:19.302576+0000","last_became_active":"2024-03-04T12:23:12.917967+0000","last_became_peered":"2024-03-04T12:23:12.917967+0000","last_unstale":"2024-03-04T12:23:19.302576+0000","last_undegraded":"2024-03-04T12:23:19.302576+0000","last_fullsized":"2024-03-04T12:23:19.302576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:15.888300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079588+0000","last_change":"2024-03-04T12:23:12.923860+0000","last_active":"2024-03-04T12:23:19.079588+0000","last_peered":"2024-03-04T12:23:19.079588+0000","last_clean":"2024-03-04T12:23:19.079588+0000","last_became_active":"2024-03-04T12:23:12.923530+0000","last_became_peered":"2024-03-04T12:23:12.923530+0000","last_unstale":"2024-03-04T12:23:19.079588+0000","last_undegraded":"2024-03-04T12:23:19.079588+0000","last_fullsized":"2024-03-04T12:23:19.079588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:30:01.466458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678633+0000","last_change":"2024-03-04T12:23:12.927159+0000","last_active":"2024-03-04T12:23:18.678633+0000","last_peered":"2024-03-04T12:23:18.678633+0000","last_clean":"2024-03-04T12:23:18.678633+0000","last_became_active":"2024-03-04T12:23:12.926800+0000","last_became_peered":"2024-03-04T12:23:12.926800+0000","last_unstale":"2024-03-04T12:23:18.678633+0000","last_undegraded":"2024-03-04T12:23:18.678633+0000","last_fullsized":"2024-03-04T12:23:18.678633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:52.633648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930475+0000","last_change":"2024-03-04T12:23:12.920607+0000","last_active":"2024-03-04T12:23:17.930475+0000","last_peered":"2024-03-04T12:23:17.930475+0000","last_clean":"2024-03-04T12:23:17.930475+0000","last_became_active":"2024-03-04T12:23:12.920257+0000","last_became_peered":"2024-03-04T12:23:12.920257+0000","last_unstale":"2024-03-04T12:23:17.930475+0000","last_undegraded":"2024-03-04T12:23:17.930475+0000","last_fullsized":"2024-03-04T12:23:17.930475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:39.353673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930163+0000","last_change":"2024-03-04T12:23:12.906635+0000","last_active":"2024-03-04T12:23:17.930163+0000","last_peered":"2024-03-04T12:23:17.930163+0000","last_clean":"2024-03-04T12:23:17.930163+0000","last_became_active":"2024-03-04T12:23:12.906243+0000","last_became_peered":"2024-03-04T12:23:12.906243+0000","last_unstale":"2024-03-04T12:23:17.930163+0000","last_undegraded":"2024-03-04T12:23:17.930163+0000","last_fullsized":"2024-03-04T12:23:17.930163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:44.375934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929791+0000","last_change":"2024-03-04T12:23:16.951685+0000","last_active":"2024-03-04T12:23:17.929791+0000","last_peered":"2024-03-04T12:23:17.929791+0000","last_clean":"2024-03-04T12:23:17.929791+0000","last_became_active":"2024-03-04T12:23:16.951329+0000","last_became_peered":"2024-03-04T12:23:16.951329+0000","last_unstale":"2024-03-04T12:23:17.929791+0000","last_undegraded":"2024-03-04T12:23:17.929791+0000","last_fullsized":"2024-03-04T12:23:17.929791+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:03:27.233466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'11","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:39.897732+0000","last_change":"2024-03-04T12:23:11.900681+0000","last_active":"2024-03-04T12:23:39.897732+0000","last_peered":"2024-03-04T12:23:39.897732+0000","last_clean":"2024-03-04T12:23:39.897732+0000","last_became_active":"2024-03-04T12:23:11.900365+0000","last_became_peered":"2024-03-04T12:23:11.900365+0000","last_unstale":"2024-03-04T12:23:39.897732+0000","last_undegraded":"2024-03-04T12:23:39.897732+0000","last_fullsized":"2024-03-04T12:23:39.897732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:53.052513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816892+0000","last_change":"2024-03-04T12:23:12.908525+0000","last_active":"2024-03-04T12:23:18.816892+0000","last_peered":"2024-03-04T12:23:18.816892+0000","last_clean":"2024-03-04T12:23:18.816892+0000","last_became_active":"2024-03-04T12:23:12.908074+0000","last_became_peered":"2024-03-04T12:23:12.908074+0000","last_unstale":"2024-03-04T12:23:18.816892+0000","last_undegraded":"2024-03-04T12:23:18.816892+0000","last_fullsized":"2024-03-04T12:23:18.816892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.061750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.978887+0000","last_change":"2024-03-04T12:23:16.953704+0000","last_active":"2024-03-04T12:23:17.978887+0000","last_peered":"2024-03-04T12:23:17.978887+0000","last_clean":"2024-03-04T12:23:17.978887+0000","last_became_active":"2024-03-04T12:23:16.953177+0000","last_became_peered":"2024-03-04T12:23:16.953177+0000","last_unstale":"2024-03-04T12:23:17.978887+0000","last_undegraded":"2024-03-04T12:23:17.978887+0000","last_fullsized":"2024-03-04T12:23:17.978887+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:37:00.469767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.817006+0000","last_change":"2024-03-04T12:23:11.904271+0000","last_active":"2024-03-04T12:23:18.817006+0000","last_peered":"2024-03-04T12:23:18.817006+0000","last_clean":"2024-03-04T12:23:18.817006+0000","last_became_active":"2024-03-04T12:23:11.903963+0000","last_became_peered":"2024-03-04T12:23:11.903963+0000","last_unstale":"2024-03-04T12:23:18.817006+0000","last_undegraded":"2024-03-04T12:23:18.817006+0000","last_fullsized":"2024-03-04T12:23:18.817006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:44.774621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302762+0000","last_change":"2024-03-04T12:23:12.925750+0000","last_active":"2024-03-04T12:23:19.302762+0000","last_peered":"2024-03-04T12:23:19.302762+0000","last_clean":"2024-03-04T12:23:19.302762+0000","last_became_active":"2024-03-04T12:23:12.925380+0000","last_became_peered":"2024-03-04T12:23:12.925380+0000","last_unstale":"2024-03-04T12:23:19.302762+0000","last_undegraded":"2024-03-04T12:23:19.302762+0000","last_fullsized":"2024-03-04T12:23:19.302762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:14.665127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979103+0000","last_change":"2024-03-04T12:23:16.952303+0000","last_active":"2024-03-04T12:23:17.979103+0000","last_peered":"2024-03-04T12:23:17.979103+0000","last_clean":"2024-03-04T12:23:17.979103+0000","last_became_active":"2024-03-04T12:23:16.951666+0000","last_became_peered":"2024-03-04T12:23:16.951666+0000","last_unstale":"2024-03-04T12:23:17.979103+0000","last_undegraded":"2024-03-04T12:23:17.979103+0000","last_fullsized":"2024-03-04T12:23:17.979103+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:41.326808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:39.016126+0000","last_change":"2024-03-04T12:23:11.903871+0000","last_active":"2024-03-04T12:23:39.016126+0000","last_peered":"2024-03-04T12:23:39.016126+0000","last_clean":"2024-03-04T12:23:39.016126+0000","last_became_active":"2024-03-04T12:23:11.902735+0000","last_became_peered":"2024-03-04T12:23:11.902735+0000","last_unstale":"2024-03-04T12:23:39.016126+0000","last_undegraded":"2024-03-04T12:23:39.016126+0000","last_fullsized":"2024-03-04T12:23:39.016126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:45:09.897740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080316+0000","last_change":"2024-03-04T12:23:12.924504+0000","last_active":"2024-03-04T12:23:19.080316+0000","last_peered":"2024-03-04T12:23:19.080316+0000","last_clean":"2024-03-04T12:23:19.080316+0000","last_became_active":"2024-03-04T12:23:12.922934+0000","last_became_peered":"2024-03-04T12:23:12.922934+0000","last_unstale":"2024-03-04T12:23:19.080316+0000","last_undegraded":"2024-03-04T12:23:19.080316+0000","last_fullsized":"2024-03-04T12:23:19.080316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:28.974601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.978937+0000","last_change":"2024-03-04T12:23:16.952324+0000","last_active":"2024-03-04T12:23:17.978937+0000","last_peered":"2024-03-04T12:23:17.978937+0000","last_clean":"2024-03-04T12:23:17.978937+0000","last_became_active":"2024-03-04T12:23:16.951145+0000","last_became_peered":"2024-03-04T12:23:16.951145+0000","last_unstale":"2024-03-04T12:23:17.978937+0000","last_undegraded":"2024-03-04T12:23:17.978937+0000","last_fullsized":"2024-03-04T12:23:17.978937+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:20:47.029523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080345+0000","last_change":"2024-03-04T12:23:11.904391+0000","last_active":"2024-03-04T12:23:19.080345+0000","last_peered":"2024-03-04T12:23:19.080345+0000","last_clean":"2024-03-04T12:23:19.080345+0000","last_became_active":"2024-03-04T12:23:11.903244+0000","last_became_peered":"2024-03-04T12:23:11.903244+0000","last_unstale":"2024-03-04T12:23:19.080345+0000","last_undegraded":"2024-03-04T12:23:19.080345+0000","last_fullsized":"2024-03-04T12:23:19.080345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:27.590418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080535+0000","last_change":"2024-03-04T12:23:12.925623+0000","last_active":"2024-03-04T12:23:19.080535+0000","last_peered":"2024-03-04T12:23:19.080535+0000","last_clean":"2024-03-04T12:23:19.080535+0000","last_became_active":"2024-03-04T12:23:12.925243+0000","last_became_peered":"2024-03-04T12:23:12.925243+0000","last_unstale":"2024-03-04T12:23:19.080535+0000","last_undegraded":"2024-03-04T12:23:19.080535+0000","last_fullsized":"2024-03-04T12:23:19.080535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:11:04.142655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930053+0000","last_change":"2024-03-04T12:23:16.946064+0000","last_active":"2024-03-04T12:23:17.930053+0000","last_peered":"2024-03-04T12:23:17.930053+0000","last_clean":"2024-03-04T12:23:17.930053+0000","last_became_active":"2024-03-04T12:23:16.945342+0000","last_became_peered":"2024-03-04T12:23:16.945342+0000","last_unstale":"2024-03-04T12:23:17.930053+0000","last_undegraded":"2024-03-04T12:23:17.930053+0000","last_fullsized":"2024-03-04T12:23:17.930053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:43:37.500216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080124+0000","last_change":"2024-03-04T12:23:11.899478+0000","last_active":"2024-03-04T12:23:19.080124+0000","last_peered":"2024-03-04T12:23:19.080124+0000","last_clean":"2024-03-04T12:23:19.080124+0000","last_became_active":"2024-03-04T12:23:11.899153+0000","last_became_peered":"2024-03-04T12:23:11.899153+0000","last_unstale":"2024-03-04T12:23:19.080124+0000","last_undegraded":"2024-03-04T12:23:19.080124+0000","last_fullsized":"2024-03-04T12:23:19.080124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:10.709054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080459+0000","last_change":"2024-03-04T12:23:12.908560+0000","last_active":"2024-03-04T12:23:19.080459+0000","last_peered":"2024-03-04T12:23:19.080459+0000","last_clean":"2024-03-04T12:23:19.080459+0000","last_became_active":"2024-03-04T12:23:12.908221+0000","last_became_peered":"2024-03-04T12:23:12.908221+0000","last_unstale":"2024-03-04T12:23:19.080459+0000","last_undegraded":"2024-03-04T12:23:19.080459+0000","last_fullsized":"2024-03-04T12:23:19.080459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:28:06.545945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929928+0000","last_change":"2024-03-04T12:23:16.950976+0000","last_active":"2024-03-04T12:23:17.929928+0000","last_peered":"2024-03-04T12:23:17.929928+0000","last_clean":"2024-03-04T12:23:17.929928+0000","last_became_active":"2024-03-04T12:23:16.950589+0000","last_became_peered":"2024-03-04T12:23:16.950589+0000","last_unstale":"2024-03-04T12:23:17.929928+0000","last_undegraded":"2024-03-04T12:23:17.929928+0000","last_fullsized":"2024-03-04T12:23:17.929928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:23:47.395349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079520+0000","last_change":"2024-03-04T12:23:11.898360+0000","last_active":"2024-03-04T12:23:19.079520+0000","last_peered":"2024-03-04T12:23:19.079520+0000","last_clean":"2024-03-04T12:23:19.079520+0000","last_became_active":"2024-03-04T12:23:11.897988+0000","last_became_peered":"2024-03-04T12:23:11.897988+0000","last_unstale":"2024-03-04T12:23:19.079520+0000","last_undegraded":"2024-03-04T12:23:19.079520+0000","last_fullsized":"2024-03-04T12:23:19.079520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:17.448032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080442+0000","last_change":"2024-03-04T12:23:12.918392+0000","last_active":"2024-03-04T12:23:19.080442+0000","last_peered":"2024-03-04T12:23:19.080442+0000","last_clean":"2024-03-04T12:23:19.080442+0000","last_became_active":"2024-03-04T12:23:12.917656+0000","last_became_peered":"2024-03-04T12:23:12.917656+0000","last_unstale":"2024-03-04T12:23:19.080442+0000","last_undegraded":"2024-03-04T12:23:19.080442+0000","last_fullsized":"2024-03-04T12:23:19.080442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:44:14.545925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678736+0000","last_change":"2024-03-04T12:23:16.941841+0000","last_active":"2024-03-04T12:23:18.678736+0000","last_peered":"2024-03-04T12:23:18.678736+0000","last_clean":"2024-03-04T12:23:18.678736+0000","last_became_active":"2024-03-04T12:23:16.941449+0000","last_became_peered":"2024-03-04T12:23:16.941449+0000","last_unstale":"2024-03-04T12:23:18.678736+0000","last_undegraded":"2024-03-04T12:23:18.678736+0000","last_fullsized":"2024-03-04T12:23:18.678736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:50:35.451834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079453+0000","last_change":"2024-03-04T12:23:11.906723+0000","last_active":"2024-03-04T12:23:19.079453+0000","last_peered":"2024-03-04T12:23:19.079453+0000","last_clean":"2024-03-04T12:23:19.079453+0000","last_became_active":"2024-03-04T12:23:11.906357+0000","last_became_peered":"2024-03-04T12:23:11.906357+0000","last_unstale":"2024-03-04T12:23:19.079453+0000","last_undegraded":"2024-03-04T12:23:19.079453+0000","last_fullsized":"2024-03-04T12:23:19.079453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:09.592236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816387+0000","last_change":"2024-03-04T12:23:12.916435+0000","last_active":"2024-03-04T12:23:18.816387+0000","last_peered":"2024-03-04T12:23:18.816387+0000","last_clean":"2024-03-04T12:23:18.816387+0000","last_became_active":"2024-03-04T12:23:12.915607+0000","last_became_peered":"2024-03-04T12:23:12.915607+0000","last_unstale":"2024-03-04T12:23:18.816387+0000","last_undegraded":"2024-03-04T12:23:18.816387+0000","last_fullsized":"2024-03-04T12:23:18.816387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:41:32.900344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302993+0000","last_change":"2024-03-04T12:23:16.944342+0000","last_active":"2024-03-04T12:23:19.302993+0000","last_peered":"2024-03-04T12:23:19.302993+0000","last_clean":"2024-03-04T12:23:19.302993+0000","last_became_active":"2024-03-04T12:23:16.943683+0000","last_became_peered":"2024-03-04T12:23:16.943683+0000","last_unstale":"2024-03-04T12:23:19.302993+0000","last_undegraded":"2024-03-04T12:23:19.302993+0000","last_fullsized":"2024-03-04T12:23:19.302993+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:19.658860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079889+0000","last_change":"2024-03-04T12:23:11.899194+0000","last_active":"2024-03-04T12:23:19.079889+0000","last_peered":"2024-03-04T12:23:19.079889+0000","last_clean":"2024-03-04T12:23:19.079889+0000","last_became_active":"2024-03-04T12:23:11.898109+0000","last_became_peered":"2024-03-04T12:23:11.898109+0000","last_unstale":"2024-03-04T12:23:19.079889+0000","last_undegraded":"2024-03-04T12:23:19.079889+0000","last_fullsized":"2024-03-04T12:23:19.079889+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:08.808202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929727+0000","last_change":"2024-03-04T12:23:12.916549+0000","last_active":"2024-03-04T12:23:17.929727+0000","last_peered":"2024-03-04T12:23:17.929727+0000","last_clean":"2024-03-04T12:23:17.929727+0000","last_became_active":"2024-03-04T12:23:12.916204+0000","last_became_peered":"2024-03-04T12:23:12.916204+0000","last_unstale":"2024-03-04T12:23:17.929727+0000","last_undegraded":"2024-03-04T12:23:17.929727+0000","last_fullsized":"2024-03-04T12:23:17.929727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:56.412600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816839+0000","last_change":"2024-03-04T12:23:16.945903+0000","last_active":"2024-03-04T12:23:18.816839+0000","last_peered":"2024-03-04T12:23:18.816839+0000","last_clean":"2024-03-04T12:23:18.816839+0000","last_became_active":"2024-03-04T12:23:16.945535+0000","last_became_peered":"2024-03-04T12:23:16.945535+0000","last_unstale":"2024-03-04T12:23:18.816839+0000","last_undegraded":"2024-03-04T12:23:18.816839+0000","last_fullsized":"2024-03-04T12:23:18.816839+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:45:01.178370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080259+0000","last_change":"2024-03-04T12:23:11.901570+0000","last_active":"2024-03-04T12:23:19.080259+0000","last_peered":"2024-03-04T12:23:19.080259+0000","last_clean":"2024-03-04T12:23:19.080259+0000","last_became_active":"2024-03-04T12:23:11.900702+0000","last_became_peered":"2024-03-04T12:23:11.900702+0000","last_unstale":"2024-03-04T12:23:19.080259+0000","last_undegraded":"2024-03-04T12:23:19.080259+0000","last_fullsized":"2024-03-04T12:23:19.080259+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:18.758453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080223+0000","last_change":"2024-03-04T12:23:12.919571+0000","last_active":"2024-03-04T12:23:19.080223+0000","last_peered":"2024-03-04T12:23:19.080223+0000","last_clean":"2024-03-04T12:23:19.080223+0000","last_became_active":"2024-03-04T12:23:12.919143+0000","last_became_peered":"2024-03-04T12:23:12.919143+0000","last_unstale":"2024-03-04T12:23:19.080223+0000","last_undegraded":"2024-03-04T12:23:19.080223+0000","last_fullsized":"2024-03-04T12:23:19.080223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:25.459417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929914+0000","last_change":"2024-03-04T12:23:16.946380+0000","last_active":"2024-03-04T12:23:17.929914+0000","last_peered":"2024-03-04T12:23:17.929914+0000","last_clean":"2024-03-04T12:23:17.929914+0000","last_became_active":"2024-03-04T12:23:16.945983+0000","last_became_peered":"2024-03-04T12:23:16.945983+0000","last_unstale":"2024-03-04T12:23:17.929914+0000","last_undegraded":"2024-03-04T12:23:17.929914+0000","last_fullsized":"2024-03-04T12:23:17.929914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:39:43.297619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:38.006808+0000","last_change":"2024-03-04T12:23:11.897176+0000","last_active":"2024-03-04T12:23:38.006808+0000","last_peered":"2024-03-04T12:23:38.006808+0000","last_clean":"2024-03-04T12:23:38.006808+0000","last_became_active":"2024-03-04T12:23:11.896747+0000","last_became_peered":"2024-03-04T12:23:11.896747+0000","last_unstale":"2024-03-04T12:23:38.006808+0000","last_undegraded":"2024-03-04T12:23:38.006808+0000","last_fullsized":"2024-03-04T12:23:38.006808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:02.206704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302794+0000","last_change":"2024-03-04T12:23:12.922021+0000","last_active":"2024-03-04T12:23:19.302794+0000","last_peered":"2024-03-04T12:23:19.302794+0000","last_clean":"2024-03-04T12:23:19.302794+0000","last_became_active":"2024-03-04T12:23:12.918724+0000","last_became_peered":"2024-03-04T12:23:12.918724+0000","last_unstale":"2024-03-04T12:23:19.302794+0000","last_undegraded":"2024-03-04T12:23:19.302794+0000","last_fullsized":"2024-03-04T12:23:19.302794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:49:09.017979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979459+0000","last_change":"2024-03-04T12:23:16.943271+0000","last_active":"2024-03-04T12:23:17.979459+0000","last_peered":"2024-03-04T12:23:17.979459+0000","last_clean":"2024-03-04T12:23:17.979459+0000","last_became_active":"2024-03-04T12:23:16.942400+0000","last_became_peered":"2024-03-04T12:23:16.942400+0000","last_unstale":"2024-03-04T12:23:17.979459+0000","last_undegraded":"2024-03-04T12:23:17.979459+0000","last_fullsized":"2024-03-04T12:23:17.979459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:06.010960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080291+0000","last_change":"2024-03-04T12:23:11.896706+0000","last_active":"2024-03-04T12:23:19.080291+0000","last_peered":"2024-03-04T12:23:19.080291+0000","last_clean":"2024-03-04T12:23:19.080291+0000","last_became_active":"2024-03-04T12:23:11.896100+0000","last_became_peered":"2024-03-04T12:23:11.896100+0000","last_unstale":"2024-03-04T12:23:19.080291+0000","last_undegraded":"2024-03-04T12:23:19.080291+0000","last_fullsized":"2024-03-04T12:23:19.080291+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:42:19.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080422+0000","last_change":"2024-03-04T12:23:12.923631+0000","last_active":"2024-03-04T12:23:19.080422+0000","last_peered":"2024-03-04T12:23:19.080422+0000","last_clean":"2024-03-04T12:23:19.080422+0000","last_became_active":"2024-03-04T12:23:12.922487+0000","last_became_peered":"2024-03-04T12:23:12.922487+0000","last_unstale":"2024-03-04T12:23:19.080422+0000","last_undegraded":"2024-03-04T12:23:19.080422+0000","last_fullsized":"2024-03-04T12:23:19.080422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:30:33.923064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929999+0000","last_change":"2024-03-04T12:23:16.948423+0000","last_active":"2024-03-04T12:23:17.929999+0000","last_peered":"2024-03-04T12:23:17.929999+0000","last_clean":"2024-03-04T12:23:17.929999+0000","last_became_active":"2024-03-04T12:23:16.947922+0000","last_became_peered":"2024-03-04T12:23:16.947922+0000","last_unstale":"2024-03-04T12:23:17.929999+0000","last_undegraded":"2024-03-04T12:23:17.929999+0000","last_fullsized":"2024-03-04T12:23:17.929999+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:13:05.145310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:29.803757+0000","last_change":"2024-03-04T12:23:11.898903+0000","last_active":"2024-03-04T12:23:29.803757+0000","last_peered":"2024-03-04T12:23:29.803757+0000","last_clean":"2024-03-04T12:23:29.803757+0000","last_became_active":"2024-03-04T12:23:11.898477+0000","last_became_peered":"2024-03-04T12:23:11.898477+0000","last_unstale":"2024-03-04T12:23:29.803757+0000","last_undegraded":"2024-03-04T12:23:29.803757+0000","last_fullsized":"2024-03-04T12:23:29.803757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:36:58.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080486+0000","last_change":"2024-03-04T12:23:12.908497+0000","last_active":"2024-03-04T12:23:19.080486+0000","last_peered":"2024-03-04T12:23:19.080486+0000","last_clean":"2024-03-04T12:23:19.080486+0000","last_became_active":"2024-03-04T12:23:12.907847+0000","last_became_peered":"2024-03-04T12:23:12.907847+0000","last_unstale":"2024-03-04T12:23:19.080486+0000","last_undegraded":"2024-03-04T12:23:19.080486+0000","last_fullsized":"2024-03-04T12:23:19.080486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:51:44.592851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.081090+0000","last_change":"2024-03-04T12:23:16.951844+0000","last_active":"2024-03-04T12:23:19.081090+0000","last_peered":"2024-03-04T12:23:19.081090+0000","last_clean":"2024-03-04T12:23:19.081090+0000","last_became_active":"2024-03-04T12:23:16.951491+0000","last_became_peered":"2024-03-04T12:23:16.951491+0000","last_unstale":"2024-03-04T12:23:19.081090+0000","last_undegraded":"2024-03-04T12:23:19.081090+0000","last_fullsized":"2024-03-04T12:23:19.081090+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:54.098239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929707+0000","last_change":"2024-03-04T12:23:11.899592+0000","last_active":"2024-03-04T12:23:17.929707+0000","last_peered":"2024-03-04T12:23:17.929707+0000","last_clean":"2024-03-04T12:23:17.929707+0000","last_became_active":"2024-03-04T12:23:11.899275+0000","last_became_peered":"2024-03-04T12:23:11.899275+0000","last_unstale":"2024-03-04T12:23:17.929707+0000","last_undegraded":"2024-03-04T12:23:17.929707+0000","last_fullsized":"2024-03-04T12:23:17.929707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:02.184268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079473+0000","last_change":"2024-03-04T12:23:12.910084+0000","last_active":"2024-03-04T12:23:19.079473+0000","last_peered":"2024-03-04T12:23:19.079473+0000","last_clean":"2024-03-04T12:23:19.079473+0000","last_became_active":"2024-03-04T12:23:12.909719+0000","last_became_peered":"2024-03-04T12:23:12.909719+0000","last_unstale":"2024-03-04T12:23:19.079473+0000","last_undegraded":"2024-03-04T12:23:19.079473+0000","last_fullsized":"2024-03-04T12:23:19.079473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:16:41.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678458+0000","last_change":"2024-03-04T12:23:16.942680+0000","last_active":"2024-03-04T12:23:18.678458+0000","last_peered":"2024-03-04T12:23:18.678458+0000","last_clean":"2024-03-04T12:23:18.678458+0000","last_became_active":"2024-03-04T12:23:16.942307+0000","last_became_peered":"2024-03-04T12:23:16.942307+0000","last_unstale":"2024-03-04T12:23:18.678458+0000","last_undegraded":"2024-03-04T12:23:18.678458+0000","last_fullsized":"2024-03-04T12:23:18.678458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:18.390621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080671+0000","last_change":"2024-03-04T12:23:11.897506+0000","last_active":"2024-03-04T12:23:19.080671+0000","last_peered":"2024-03-04T12:23:19.080671+0000","last_clean":"2024-03-04T12:23:19.080671+0000","last_became_active":"2024-03-04T12:23:11.897090+0000","last_became_peered":"2024-03-04T12:23:11.897090+0000","last_unstale":"2024-03-04T12:23:19.080671+0000","last_undegraded":"2024-03-04T12:23:19.080671+0000","last_fullsized":"2024-03-04T12:23:19.080671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:10:27.868713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930408+0000","last_change":"2024-03-04T12:23:12.921825+0000","last_active":"2024-03-04T12:23:17.930408+0000","last_peered":"2024-03-04T12:23:17.930408+0000","last_clean":"2024-03-04T12:23:17.930408+0000","last_became_active":"2024-03-04T12:23:12.921491+0000","last_became_peered":"2024-03-04T12:23:12.921491+0000","last_unstale":"2024-03-04T12:23:17.930408+0000","last_undegraded":"2024-03-04T12:23:17.930408+0000","last_fullsized":"2024-03-04T12:23:17.930408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:17:08.203502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930232+0000","last_change":"2024-03-04T12:23:16.951920+0000","last_active":"2024-03-04T12:23:17.930232+0000","last_peered":"2024-03-04T12:23:17.930232+0000","last_clean":"2024-03-04T12:23:17.930232+0000","last_became_active":"2024-03-04T12:23:16.951507+0000","last_became_peered":"2024-03-04T12:23:16.951507+0000","last_unstale":"2024-03-04T12:23:17.930232+0000","last_undegraded":"2024-03-04T12:23:17.930232+0000","last_fullsized":"2024-03-04T12:23:17.930232+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:27.673692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.978889+0000","last_change":"2024-03-04T12:23:11.897722+0000","last_active":"2024-03-04T12:23:17.978889+0000","last_peered":"2024-03-04T12:23:17.978889+0000","last_clean":"2024-03-04T12:23:17.978889+0000","last_became_active":"2024-03-04T12:23:11.897276+0000","last_became_peered":"2024-03-04T12:23:11.897276+0000","last_unstale":"2024-03-04T12:23:17.978889+0000","last_undegraded":"2024-03-04T12:23:17.978889+0000","last_fullsized":"2024-03-04T12:23:17.978889+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:37.968804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930042+0000","last_change":"2024-03-04T12:23:12.914773+0000","last_active":"2024-03-04T12:23:17.930042+0000","last_peered":"2024-03-04T12:23:17.930042+0000","last_clean":"2024-03-04T12:23:17.930042+0000","last_became_active":"2024-03-04T12:23:12.914431+0000","last_became_peered":"2024-03-04T12:23:12.914431+0000","last_unstale":"2024-03-04T12:23:17.930042+0000","last_undegraded":"2024-03-04T12:23:17.930042+0000","last_fullsized":"2024-03-04T12:23:17.930042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:50.234491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079976+0000","last_change":"2024-03-04T12:23:16.952405+0000","last_active":"2024-03-04T12:23:19.079976+0000","last_peered":"2024-03-04T12:23:19.079976+0000","last_clean":"2024-03-04T12:23:19.079976+0000","last_became_active":"2024-03-04T12:23:16.951915+0000","last_became_peered":"2024-03-04T12:23:16.951915+0000","last_unstale":"2024-03-04T12:23:19.079976+0000","last_undegraded":"2024-03-04T12:23:19.079976+0000","last_fullsized":"2024-03-04T12:23:19.079976+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:38:03.765883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:39.016559+0000","last_change":"2024-03-04T12:23:11.903267+0000","last_active":"2024-03-04T12:23:39.016559+0000","last_peered":"2024-03-04T12:23:39.016559+0000","last_clean":"2024-03-04T12:23:39.016559+0000","last_became_active":"2024-03-04T12:23:11.902870+0000","last_became_peered":"2024-03-04T12:23:11.902870+0000","last_unstale":"2024-03-04T12:23:39.016559+0000","last_undegraded":"2024-03-04T12:23:39.016559+0000","last_fullsized":"2024-03-04T12:23:39.016559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:33:43.679869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080445+0000","last_change":"2024-03-04T12:23:12.925303+0000","last_active":"2024-03-04T12:23:19.080445+0000","last_peered":"2024-03-04T12:23:19.080445+0000","last_clean":"2024-03-04T12:23:19.080445+0000","last_became_active":"2024-03-04T12:23:12.924945+0000","last_became_peered":"2024-03-04T12:23:12.924945+0000","last_unstale":"2024-03-04T12:23:19.080445+0000","last_undegraded":"2024-03-04T12:23:19.080445+0000","last_fullsized":"2024-03-04T12:23:19.080445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:49.680925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080355+0000","last_change":"2024-03-04T12:23:16.951457+0000","last_active":"2024-03-04T12:23:19.080355+0000","last_peered":"2024-03-04T12:23:19.080355+0000","last_clean":"2024-03-04T12:23:19.080355+0000","last_became_active":"2024-03-04T12:23:16.950835+0000","last_became_peered":"2024-03-04T12:23:16.950835+0000","last_unstale":"2024-03-04T12:23:19.080355+0000","last_undegraded":"2024-03-04T12:23:19.080355+0000","last_fullsized":"2024-03-04T12:23:19.080355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:39.337275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080614+0000","last_change":"2024-03-04T12:23:11.903718+0000","last_active":"2024-03-04T12:23:19.080614+0000","last_peered":"2024-03-04T12:23:19.080614+0000","last_clean":"2024-03-04T12:23:19.080614+0000","last_became_active":"2024-03-04T12:23:11.902814+0000","last_became_peered":"2024-03-04T12:23:11.902814+0000","last_unstale":"2024-03-04T12:23:19.080614+0000","last_undegraded":"2024-03-04T12:23:19.080614+0000","last_fullsized":"2024-03-04T12:23:19.080614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:40:30.978576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678460+0000","last_change":"2024-03-04T12:23:12.919020+0000","last_active":"2024-03-04T12:23:18.678460+0000","last_peered":"2024-03-04T12:23:18.678460+0000","last_clean":"2024-03-04T12:23:18.678460+0000","last_became_active":"2024-03-04T12:23:12.918661+0000","last_became_peered":"2024-03-04T12:23:12.918661+0000","last_unstale":"2024-03-04T12:23:18.678460+0000","last_undegraded":"2024-03-04T12:23:18.678460+0000","last_fullsized":"2024-03-04T12:23:18.678460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:27:15.869681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080398+0000","last_change":"2024-03-04T12:23:16.950046+0000","last_active":"2024-03-04T12:23:19.080398+0000","last_peered":"2024-03-04T12:23:19.080398+0000","last_clean":"2024-03-04T12:23:19.080398+0000","last_became_active":"2024-03-04T12:23:16.949632+0000","last_became_peered":"2024-03-04T12:23:16.949632+0000","last_unstale":"2024-03-04T12:23:19.080398+0000","last_undegraded":"2024-03-04T12:23:19.080398+0000","last_fullsized":"2024-03-04T12:23:19.080398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:25.817212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930210+0000","last_change":"2024-03-04T12:23:11.897016+0000","last_active":"2024-03-04T12:23:17.930210+0000","last_peered":"2024-03-04T12:23:17.930210+0000","last_clean":"2024-03-04T12:23:17.930210+0000","last_became_active":"2024-03-04T12:23:11.896668+0000","last_became_peered":"2024-03-04T12:23:11.896668+0000","last_unstale":"2024-03-04T12:23:17.930210+0000","last_undegraded":"2024-03-04T12:23:17.930210+0000","last_fullsized":"2024-03-04T12:23:17.930210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:50:01.089658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080037+0000","last_change":"2024-03-04T12:23:12.908169+0000","last_active":"2024-03-04T12:23:19.080037+0000","last_peered":"2024-03-04T12:23:19.080037+0000","last_clean":"2024-03-04T12:23:19.080037+0000","last_became_active":"2024-03-04T12:23:12.907786+0000","last_became_peered":"2024-03-04T12:23:12.907786+0000","last_unstale":"2024-03-04T12:23:19.080037+0000","last_undegraded":"2024-03-04T12:23:19.080037+0000","last_fullsized":"2024-03-04T12:23:19.080037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:10.602216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079873+0000","last_change":"2024-03-04T12:23:16.944232+0000","last_active":"2024-03-04T12:23:19.079873+0000","last_peered":"2024-03-04T12:23:19.079873+0000","last_clean":"2024-03-04T12:23:19.079873+0000","last_became_active":"2024-03-04T12:23:16.943855+0000","last_became_peered":"2024-03-04T12:23:16.943855+0000","last_unstale":"2024-03-04T12:23:19.079873+0000","last_undegraded":"2024-03-04T12:23:19.079873+0000","last_fullsized":"2024-03-04T12:23:19.079873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:37:12.083622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930109+0000","last_change":"2024-03-04T12:23:11.899742+0000","last_active":"2024-03-04T12:23:17.930109+0000","last_peered":"2024-03-04T12:23:17.930109+0000","last_clean":"2024-03-04T12:23:17.930109+0000","last_became_active":"2024-03-04T12:23:11.898445+0000","last_became_peered":"2024-03-04T12:23:11.898445+0000","last_unstale":"2024-03-04T12:23:17.930109+0000","last_undegraded":"2024-03-04T12:23:17.930109+0000","last_fullsized":"2024-03-04T12:23:17.930109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:13.128348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079830+0000","last_change":"2024-03-04T12:23:12.921321+0000","last_active":"2024-03-04T12:23:19.079830+0000","last_peered":"2024-03-04T12:23:19.079830+0000","last_clean":"2024-03-04T12:23:19.079830+0000","last_became_active":"2024-03-04T12:23:12.918119+0000","last_became_peered":"2024-03-04T12:23:12.918119+0000","last_unstale":"2024-03-04T12:23:19.079830+0000","last_undegraded":"2024-03-04T12:23:19.079830+0000","last_fullsized":"2024-03-04T12:23:19.079830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:31.693583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079873+0000","last_change":"2024-03-04T12:23:16.943887+0000","last_active":"2024-03-04T12:23:19.079873+0000","last_peered":"2024-03-04T12:23:19.079873+0000","last_clean":"2024-03-04T12:23:19.079873+0000","last_became_active":"2024-03-04T12:23:16.943487+0000","last_became_peered":"2024-03-04T12:23:16.943487+0000","last_unstale":"2024-03-04T12:23:19.079873+0000","last_undegraded":"2024-03-04T12:23:19.079873+0000","last_fullsized":"2024-03-04T12:23:19.079873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:12.451176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080043+0000","last_change":"2024-03-04T12:23:12.918363+0000","last_active":"2024-03-04T12:23:19.080043+0000","last_peered":"2024-03-04T12:23:19.080043+0000","last_clean":"2024-03-04T12:23:19.080043+0000","last_became_active":"2024-03-04T12:23:12.917563+0000","last_became_peered":"2024-03-04T12:23:12.917563+0000","last_unstale":"2024-03-04T12:23:19.080043+0000","last_undegraded":"2024-03-04T12:23:19.080043+0000","last_fullsized":"2024-03-04T12:23:19.080043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:26:35.297513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079397+0000","last_change":"2024-03-04T12:23:11.904072+0000","last_active":"2024-03-04T12:23:19.079397+0000","last_peered":"2024-03-04T12:23:19.079397+0000","last_clean":"2024-03-04T12:23:19.079397+0000","last_became_active":"2024-03-04T12:23:11.903097+0000","last_became_peered":"2024-03-04T12:23:11.903097+0000","last_unstale":"2024-03-04T12:23:19.079397+0000","last_undegraded":"2024-03-04T12:23:19.079397+0000","last_fullsized":"2024-03-04T12:23:19.079397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:36.606667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930670+0000","last_change":"2024-03-04T12:23:16.951019+0000","last_active":"2024-03-04T12:23:17.930670+0000","last_peered":"2024-03-04T12:23:17.930670+0000","last_clean":"2024-03-04T12:23:17.930670+0000","last_became_active":"2024-03-04T12:23:16.950524+0000","last_became_peered":"2024-03-04T12:23:16.950524+0000","last_unstale":"2024-03-04T12:23:17.930670+0000","last_undegraded":"2024-03-04T12:23:17.930670+0000","last_fullsized":"2024-03-04T12:23:17.930670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:48.376539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816413+0000","last_change":"2024-03-04T12:23:12.927615+0000","last_active":"2024-03-04T12:23:18.816413+0000","last_peered":"2024-03-04T12:23:18.816413+0000","last_clean":"2024-03-04T12:23:18.816413+0000","last_became_active":"2024-03-04T12:23:12.927190+0000","last_became_peered":"2024-03-04T12:23:12.927190+0000","last_unstale":"2024-03-04T12:23:18.816413+0000","last_undegraded":"2024-03-04T12:23:18.816413+0000","last_fullsized":"2024-03-04T12:23:18.816413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:54:19.019030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979154+0000","last_change":"2024-03-04T12:23:11.893542+0000","last_active":"2024-03-04T12:23:17.979154+0000","last_peered":"2024-03-04T12:23:17.979154+0000","last_clean":"2024-03-04T12:23:17.979154+0000","last_became_active":"2024-03-04T12:23:11.893166+0000","last_became_peered":"2024-03-04T12:23:11.893166+0000","last_unstale":"2024-03-04T12:23:17.979154+0000","last_undegraded":"2024-03-04T12:23:17.979154+0000","last_fullsized":"2024-03-04T12:23:17.979154+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:32.117863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929900+0000","last_change":"2024-03-04T12:21:55.865103+0000","last_active":"2024-03-04T12:23:17.929900+0000","last_peered":"2024-03-04T12:23:17.929900+0000","last_clean":"2024-03-04T12:23:17.929900+0000","last_became_active":"2024-03-04T12:21:55.864553+0000","last_became_peered":"2024-03-04T12:21:55.864553+0000","last_unstale":"2024-03-04T12:23:17.929900+0000","last_undegraded":"2024-03-04T12:23:17.929900+0000","last_fullsized":"2024-03-04T12:23:17.929900+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_clean_scrub_stamp":"2024-03-04T12:18:28.442423+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:31.052782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.081061+0000","last_change":"2024-03-04T12:23:16.945852+0000","last_active":"2024-03-04T12:23:19.081061+0000","last_peered":"2024-03-04T12:23:19.081061+0000","last_clean":"2024-03-04T12:23:19.081061+0000","last_became_active":"2024-03-04T12:23:16.945496+0000","last_became_peered":"2024-03-04T12:23:16.945496+0000","last_unstale":"2024-03-04T12:23:19.081061+0000","last_undegraded":"2024-03-04T12:23:19.081061+0000","last_fullsized":"2024-03-04T12:23:19.081061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:35:10.880757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079323+0000","last_change":"2024-03-04T12:23:12.919219+0000","last_active":"2024-03-04T12:23:19.079323+0000","last_peered":"2024-03-04T12:23:19.079323+0000","last_clean":"2024-03-04T12:23:19.079323+0000","last_became_active":"2024-03-04T12:23:12.918897+0000","last_became_peered":"2024-03-04T12:23:12.918897+0000","last_unstale":"2024-03-04T12:23:19.079323+0000","last_undegraded":"2024-03-04T12:23:19.079323+0000","last_fullsized":"2024-03-04T12:23:19.079323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:24:32.594172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930044+0000","last_change":"2024-03-04T12:23:11.894544+0000","last_active":"2024-03-04T12:23:17.930044+0000","last_peered":"2024-03-04T12:23:17.930044+0000","last_clean":"2024-03-04T12:23:17.930044+0000","last_became_active":"2024-03-04T12:23:11.894188+0000","last_became_peered":"2024-03-04T12:23:11.894188+0000","last_unstale":"2024-03-04T12:23:17.930044+0000","last_undegraded":"2024-03-04T12:23:17.930044+0000","last_fullsized":"2024-03-04T12:23:17.930044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:28:14.687530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678468+0000","last_change":"2024-03-04T12:23:16.945510+0000","last_active":"2024-03-04T12:23:18.678468+0000","last_peered":"2024-03-04T12:23:18.678468+0000","last_clean":"2024-03-04T12:23:18.678468+0000","last_became_active":"2024-03-04T12:23:16.945140+0000","last_became_peered":"2024-03-04T12:23:16.945140+0000","last_unstale":"2024-03-04T12:23:18.678468+0000","last_undegraded":"2024-03-04T12:23:18.678468+0000","last_fullsized":"2024-03-04T12:23:18.678468+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:44.102503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080574+0000","last_change":"2024-03-04T12:23:11.904515+0000","last_active":"2024-03-04T12:23:19.080574+0000","last_peered":"2024-03-04T12:23:19.080574+0000","last_clean":"2024-03-04T12:23:19.080574+0000","last_became_active":"2024-03-04T12:23:11.904149+0000","last_became_peered":"2024-03-04T12:23:11.904149+0000","last_unstale":"2024-03-04T12:23:19.080574+0000","last_undegraded":"2024-03-04T12:23:19.080574+0000","last_fullsized":"2024-03-04T12:23:19.080574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:04:51.382626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079956+0000","last_change":"2024-03-04T12:23:12.908749+0000","last_active":"2024-03-04T12:23:19.079956+0000","last_peered":"2024-03-04T12:23:19.079956+0000","last_clean":"2024-03-04T12:23:19.079956+0000","last_became_active":"2024-03-04T12:23:12.908405+0000","last_became_peered":"2024-03-04T12:23:12.908405+0000","last_unstale":"2024-03-04T12:23:19.079956+0000","last_undegraded":"2024-03-04T12:23:19.079956+0000","last_fullsized":"2024-03-04T12:23:19.079956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:10.472953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816434+0000","last_change":"2024-03-04T12:23:16.951145+0000","last_active":"2024-03-04T12:23:18.816434+0000","last_peered":"2024-03-04T12:23:18.816434+0000","last_clean":"2024-03-04T12:23:18.816434+0000","last_became_active":"2024-03-04T12:23:16.950702+0000","last_became_peered":"2024-03-04T12:23:16.950702+0000","last_unstale":"2024-03-04T12:23:18.816434+0000","last_undegraded":"2024-03-04T12:23:18.816434+0000","last_fullsized":"2024-03-04T12:23:18.816434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:42.720616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:39.898638+0000","last_change":"2024-03-04T12:23:11.893810+0000","last_active":"2024-03-04T12:23:39.898638+0000","last_peered":"2024-03-04T12:23:39.898638+0000","last_clean":"2024-03-04T12:23:39.898638+0000","last_became_active":"2024-03-04T12:23:11.893465+0000","last_became_peered":"2024-03-04T12:23:11.893465+0000","last_unstale":"2024-03-04T12:23:39.898638+0000","last_undegraded":"2024-03-04T12:23:39.898638+0000","last_fullsized":"2024-03-04T12:23:39.898638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:27:41.199786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816750+0000","last_change":"2024-03-04T12:23:12.925763+0000","last_active":"2024-03-04T12:23:18.816750+0000","last_peered":"2024-03-04T12:23:18.816750+0000","last_clean":"2024-03-04T12:23:18.816750+0000","last_became_active":"2024-03-04T12:23:12.925429+0000","last_became_peered":"2024-03-04T12:23:12.925429+0000","last_unstale":"2024-03-04T12:23:18.816750+0000","last_undegraded":"2024-03-04T12:23:18.816750+0000","last_fullsized":"2024-03-04T12:23:18.816750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:46.814459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929720+0000","last_change":"2024-03-04T12:23:16.942624+0000","last_active":"2024-03-04T12:23:17.929720+0000","last_peered":"2024-03-04T12:23:17.929720+0000","last_clean":"2024-03-04T12:23:17.929720+0000","last_became_active":"2024-03-04T12:23:16.942287+0000","last_became_peered":"2024-03-04T12:23:16.942287+0000","last_unstale":"2024-03-04T12:23:17.929720+0000","last_undegraded":"2024-03-04T12:23:17.929720+0000","last_fullsized":"2024-03-04T12:23:17.929720+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:59.479611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979526+0000","last_change":"2024-03-04T12:23:11.891504+0000","last_active":"2024-03-04T12:23:17.979526+0000","last_peered":"2024-03-04T12:23:17.979526+0000","last_clean":"2024-03-04T12:23:17.979526+0000","last_became_active":"2024-03-04T12:23:11.891134+0000","last_became_peered":"2024-03-04T12:23:11.891134+0000","last_unstale":"2024-03-04T12:23:17.979526+0000","last_undegraded":"2024-03-04T12:23:17.979526+0000","last_fullsized":"2024-03-04T12:23:17.979526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:41.481003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.303017+0000","last_change":"2024-03-04T12:23:12.928989+0000","last_active":"2024-03-04T12:23:19.303017+0000","last_peered":"2024-03-04T12:23:19.303017+0000","last_clean":"2024-03-04T12:23:19.303017+0000","last_became_active":"2024-03-04T12:23:12.928609+0000","last_became_peered":"2024-03-04T12:23:12.928609+0000","last_unstale":"2024-03-04T12:23:19.303017+0000","last_undegraded":"2024-03-04T12:23:19.303017+0000","last_fullsized":"2024-03-04T12:23:19.303017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:39.979546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979327+0000","last_change":"2024-03-04T12:23:16.949590+0000","last_active":"2024-03-04T12:23:17.979327+0000","last_peered":"2024-03-04T12:23:17.979327+0000","last_clean":"2024-03-04T12:23:17.979327+0000","last_became_active":"2024-03-04T12:23:16.949086+0000","last_became_peered":"2024-03-04T12:23:16.949086+0000","last_unstale":"2024-03-04T12:23:17.979327+0000","last_undegraded":"2024-03-04T12:23:17.979327+0000","last_fullsized":"2024-03-04T12:23:17.979327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:07:09.819097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:38.005174+0000","last_change":"2024-03-04T12:23:11.894110+0000","last_active":"2024-03-04T12:23:38.005174+0000","last_peered":"2024-03-04T12:23:38.005174+0000","last_clean":"2024-03-04T12:23:38.005174+0000","last_became_active":"2024-03-04T12:23:11.893419+0000","last_became_peered":"2024-03-04T12:23:11.893419+0000","last_unstale":"2024-03-04T12:23:38.005174+0000","last_undegraded":"2024-03-04T12:23:38.005174+0000","last_fullsized":"2024-03-04T12:23:38.005174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:53.963989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079749+0000","last_change":"2024-03-04T12:23:12.907403+0000","last_active":"2024-03-04T12:23:19.079749+0000","last_peered":"2024-03-04T12:23:19.079749+0000","last_clean":"2024-03-04T12:23:19.079749+0000","last_became_active":"2024-03-04T12:23:12.906674+0000","last_became_peered":"2024-03-04T12:23:12.906674+0000","last_unstale":"2024-03-04T12:23:19.079749+0000","last_undegraded":"2024-03-04T12:23:19.079749+0000","last_fullsized":"2024-03-04T12:23:19.079749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:49:01.133676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930542+0000","last_change":"2024-03-04T12:23:16.955113+0000","last_active":"2024-03-04T12:23:17.930542+0000","last_peered":"2024-03-04T12:23:17.930542+0000","last_clean":"2024-03-04T12:23:17.930542+0000","last_became_active":"2024-03-04T12:23:16.951624+0000","last_became_peered":"2024-03-04T12:23:16.951624+0000","last_unstale":"2024-03-04T12:23:17.930542+0000","last_undegraded":"2024-03-04T12:23:17.930542+0000","last_fullsized":"2024-03-04T12:23:17.930542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:22:23.935460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816401+0000","last_change":"2024-03-04T12:23:11.900213+0000","last_active":"2024-03-04T12:23:18.816401+0000","last_peered":"2024-03-04T12:23:18.816401+0000","last_clean":"2024-03-04T12:23:18.816401+0000","last_became_active":"2024-03-04T12:23:11.899832+0000","last_became_peered":"2024-03-04T12:23:11.899832+0000","last_unstale":"2024-03-04T12:23:18.816401+0000","last_undegraded":"2024-03-04T12:23:18.816401+0000","last_fullsized":"2024-03-04T12:23:18.816401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:55:24.553309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678159+0000","last_change":"2024-03-04T12:23:12.913315+0000","last_active":"2024-03-04T12:23:18.678159+0000","last_peered":"2024-03-04T12:23:18.678159+0000","last_clean":"2024-03-04T12:23:18.678159+0000","last_became_active":"2024-03-04T12:23:12.912850+0000","last_became_peered":"2024-03-04T12:23:12.912850+0000","last_unstale":"2024-03-04T12:23:18.678159+0000","last_undegraded":"2024-03-04T12:23:18.678159+0000","last_fullsized":"2024-03-04T12:23:18.678159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:55.140866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979207+0000","last_change":"2024-03-04T12:23:16.943295+0000","last_active":"2024-03-04T12:23:17.979207+0000","last_peered":"2024-03-04T12:23:17.979207+0000","last_clean":"2024-03-04T12:23:17.979207+0000","last_became_active":"2024-03-04T12:23:16.942853+0000","last_became_peered":"2024-03-04T12:23:16.942853+0000","last_unstale":"2024-03-04T12:23:17.979207+0000","last_undegraded":"2024-03-04T12:23:17.979207+0000","last_fullsized":"2024-03-04T12:23:17.979207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:05:51.162179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929672+0000","last_change":"2024-03-04T12:23:11.897870+0000","last_active":"2024-03-04T12:23:17.929672+0000","last_peered":"2024-03-04T12:23:17.929672+0000","last_clean":"2024-03-04T12:23:17.929672+0000","last_became_active":"2024-03-04T12:23:11.897443+0000","last_became_peered":"2024-03-04T12:23:11.897443+0000","last_unstale":"2024-03-04T12:23:17.929672+0000","last_undegraded":"2024-03-04T12:23:17.929672+0000","last_fullsized":"2024-03-04T12:23:17.929672+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:49:37.875541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816672+0000","last_change":"2024-03-04T12:23:12.916441+0000","last_active":"2024-03-04T12:23:18.816672+0000","last_peered":"2024-03-04T12:23:18.816672+0000","last_clean":"2024-03-04T12:23:18.816672+0000","last_became_active":"2024-03-04T12:23:12.916078+0000","last_became_peered":"2024-03-04T12:23:12.916078+0000","last_unstale":"2024-03-04T12:23:18.816672+0000","last_undegraded":"2024-03-04T12:23:18.816672+0000","last_fullsized":"2024-03-04T12:23:18.816672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:45:23.663627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979389+0000","last_change":"2024-03-04T12:23:16.953341+0000","last_active":"2024-03-04T12:23:17.979389+0000","last_peered":"2024-03-04T12:23:17.979389+0000","last_clean":"2024-03-04T12:23:17.979389+0000","last_became_active":"2024-03-04T12:23:16.952878+0000","last_became_peered":"2024-03-04T12:23:16.952878+0000","last_unstale":"2024-03-04T12:23:17.979389+0000","last_undegraded":"2024-03-04T12:23:17.979389+0000","last_fullsized":"2024-03-04T12:23:17.979389+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:08:55.575263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:38.006943+0000","last_change":"2024-03-04T12:23:11.903182+0000","last_active":"2024-03-04T12:23:38.006943+0000","last_peered":"2024-03-04T12:23:38.006943+0000","last_clean":"2024-03-04T12:23:38.006943+0000","last_became_active":"2024-03-04T12:23:11.902709+0000","last_became_peered":"2024-03-04T12:23:11.902709+0000","last_unstale":"2024-03-04T12:23:38.006943+0000","last_undegraded":"2024-03-04T12:23:38.006943+0000","last_fullsized":"2024-03-04T12:23:38.006943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:13:33.438929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930288+0000","last_change":"2024-03-04T12:23:12.931339+0000","last_active":"2024-03-04T12:23:17.930288+0000","last_peered":"2024-03-04T12:23:17.930288+0000","last_clean":"2024-03-04T12:23:17.930288+0000","last_became_active":"2024-03-04T12:23:12.930213+0000","last_became_peered":"2024-03-04T12:23:12.930213+0000","last_unstale":"2024-03-04T12:23:17.930288+0000","last_undegraded":"2024-03-04T12:23:17.930288+0000","last_fullsized":"2024-03-04T12:23:17.930288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:55:23.084139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079581+0000","last_change":"2024-03-04T12:23:16.948723+0000","last_active":"2024-03-04T12:23:19.079581+0000","last_peered":"2024-03-04T12:23:19.079581+0000","last_clean":"2024-03-04T12:23:19.079581+0000","last_became_active":"2024-03-04T12:23:16.948256+0000","last_became_peered":"2024-03-04T12:23:16.948256+0000","last_unstale":"2024-03-04T12:23:19.079581+0000","last_undegraded":"2024-03-04T12:23:19.079581+0000","last_fullsized":"2024-03-04T12:23:19.079581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:22.060517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930042+0000","last_change":"2024-03-04T12:23:11.898475+0000","last_active":"2024-03-04T12:23:17.930042+0000","last_peered":"2024-03-04T12:23:17.930042+0000","last_clean":"2024-03-04T12:23:17.930042+0000","last_became_active":"2024-03-04T12:23:11.898117+0000","last_became_peered":"2024-03-04T12:23:11.898117+0000","last_unstale":"2024-03-04T12:23:17.930042+0000","last_undegraded":"2024-03-04T12:23:17.930042+0000","last_fullsized":"2024-03-04T12:23:17.930042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:21:25.119591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930746+0000","last_change":"2024-03-04T12:23:12.929772+0000","last_active":"2024-03-04T12:23:17.930746+0000","last_peered":"2024-03-04T12:23:17.930746+0000","last_clean":"2024-03-04T12:23:17.930746+0000","last_became_active":"2024-03-04T12:23:12.929405+0000","last_became_peered":"2024-03-04T12:23:12.929405+0000","last_unstale":"2024-03-04T12:23:17.930746+0000","last_undegraded":"2024-03-04T12:23:17.930746+0000","last_fullsized":"2024-03-04T12:23:17.930746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:22:42.874989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.303039+0000","last_change":"2024-03-04T12:23:16.941715+0000","last_active":"2024-03-04T12:23:19.303039+0000","last_peered":"2024-03-04T12:23:19.303039+0000","last_clean":"2024-03-04T12:23:19.303039+0000","last_became_active":"2024-03-04T12:23:16.941298+0000","last_became_peered":"2024-03-04T12:23:16.941298+0000","last_unstale":"2024-03-04T12:23:19.303039+0000","last_undegraded":"2024-03-04T12:23:19.303039+0000","last_fullsized":"2024-03-04T12:23:19.303039+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:09:02.019531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080638+0000","last_change":"2024-03-04T12:23:11.896413+0000","last_active":"2024-03-04T12:23:19.080638+0000","last_peered":"2024-03-04T12:23:19.080638+0000","last_clean":"2024-03-04T12:23:19.080638+0000","last_became_active":"2024-03-04T12:23:11.895679+0000","last_became_peered":"2024-03-04T12:23:11.895679+0000","last_unstale":"2024-03-04T12:23:19.080638+0000","last_undegraded":"2024-03-04T12:23:19.080638+0000","last_fullsized":"2024-03-04T12:23:19.080638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:25:41.967530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930490+0000","last_change":"2024-03-04T12:23:12.920468+0000","last_active":"2024-03-04T12:23:17.930490+0000","last_peered":"2024-03-04T12:23:17.930490+0000","last_clean":"2024-03-04T12:23:17.930490+0000","last_became_active":"2024-03-04T12:23:12.920113+0000","last_became_peered":"2024-03-04T12:23:12.920113+0000","last_unstale":"2024-03-04T12:23:17.930490+0000","last_undegraded":"2024-03-04T12:23:17.930490+0000","last_fullsized":"2024-03-04T12:23:17.930490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:00:02.283518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079761+0000","last_change":"2024-03-04T12:23:16.944840+0000","last_active":"2024-03-04T12:23:19.079761+0000","last_peered":"2024-03-04T12:23:19.079761+0000","last_clean":"2024-03-04T12:23:19.079761+0000","last_became_active":"2024-03-04T12:23:16.944333+0000","last_became_peered":"2024-03-04T12:23:16.944333+0000","last_unstale":"2024-03-04T12:23:19.079761+0000","last_undegraded":"2024-03-04T12:23:19.079761+0000","last_fullsized":"2024-03-04T12:23:19.079761+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:06:05.535789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079858+0000","last_change":"2024-03-04T12:23:11.903323+0000","last_active":"2024-03-04T12:23:19.079858+0000","last_peered":"2024-03-04T12:23:19.079858+0000","last_clean":"2024-03-04T12:23:19.079858+0000","last_became_active":"2024-03-04T12:23:11.902987+0000","last_became_peered":"2024-03-04T12:23:11.902987+0000","last_unstale":"2024-03-04T12:23:19.079858+0000","last_undegraded":"2024-03-04T12:23:19.079858+0000","last_fullsized":"2024-03-04T12:23:19.079858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:14.044391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930749+0000","last_change":"2024-03-04T12:23:12.913491+0000","last_active":"2024-03-04T12:23:17.930749+0000","last_peered":"2024-03-04T12:23:17.930749+0000","last_clean":"2024-03-04T12:23:17.930749+0000","last_became_active":"2024-03-04T12:23:12.912103+0000","last_became_peered":"2024-03-04T12:23:12.912103+0000","last_unstale":"2024-03-04T12:23:17.930749+0000","last_undegraded":"2024-03-04T12:23:17.930749+0000","last_fullsized":"2024-03-04T12:23:17.930749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:05:38.033596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678282+0000","last_change":"2024-03-04T12:23:12.908437+0000","last_active":"2024-03-04T12:23:18.678282+0000","last_peered":"2024-03-04T12:23:18.678282+0000","last_clean":"2024-03-04T12:23:18.678282+0000","last_became_active":"2024-03-04T12:23:12.908071+0000","last_became_peered":"2024-03-04T12:23:12.908071+0000","last_unstale":"2024-03-04T12:23:18.678282+0000","last_undegraded":"2024-03-04T12:23:18.678282+0000","last_fullsized":"2024-03-04T12:23:18.678282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:20.112645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929913+0000","last_change":"2024-03-04T12:23:12.921645+0000","last_active":"2024-03-04T12:23:17.929913+0000","last_peered":"2024-03-04T12:23:17.929913+0000","last_clean":"2024-03-04T12:23:17.929913+0000","last_became_active":"2024-03-04T12:23:12.921275+0000","last_became_peered":"2024-03-04T12:23:12.921275+0000","last_unstale":"2024-03-04T12:23:17.929913+0000","last_undegraded":"2024-03-04T12:23:17.929913+0000","last_fullsized":"2024-03-04T12:23:17.929913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:56:17.740782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929592+0000","last_change":"2024-03-04T12:23:12.919813+0000","last_active":"2024-03-04T12:23:17.929592+0000","last_peered":"2024-03-04T12:23:17.929592+0000","last_clean":"2024-03-04T12:23:17.929592+0000","last_became_active":"2024-03-04T12:23:12.919469+0000","last_became_peered":"2024-03-04T12:23:12.919469+0000","last_unstale":"2024-03-04T12:23:17.929592+0000","last_undegraded":"2024-03-04T12:23:17.929592+0000","last_fullsized":"2024-03-04T12:23:17.929592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:46.720268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302719+0000","last_change":"2024-03-04T12:23:12.918476+0000","last_active":"2024-03-04T12:23:19.302719+0000","last_peered":"2024-03-04T12:23:19.302719+0000","last_clean":"2024-03-04T12:23:19.302719+0000","last_became_active":"2024-03-04T12:23:12.917743+0000","last_became_peered":"2024-03-04T12:23:12.917743+0000","last_unstale":"2024-03-04T12:23:19.302719+0000","last_undegraded":"2024-03-04T12:23:19.302719+0000","last_fullsized":"2024-03-04T12:23:19.302719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:20.246769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079778+0000","last_change":"2024-03-04T12:23:12.918254+0000","last_active":"2024-03-04T12:23:19.079778+0000","last_peered":"2024-03-04T12:23:19.079778+0000","last_clean":"2024-03-04T12:23:19.079778+0000","last_became_active":"2024-03-04T12:23:12.917922+0000","last_became_peered":"2024-03-04T12:23:12.917922+0000","last_unstale":"2024-03-04T12:23:19.079778+0000","last_undegraded":"2024-03-04T12:23:19.079778+0000","last_fullsized":"2024-03-04T12:23:19.079778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:48:20.847984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678592+0000","last_change":"2024-03-04T12:23:12.910368+0000","last_active":"2024-03-04T12:23:18.678592+0000","last_peered":"2024-03-04T12:23:18.678592+0000","last_clean":"2024-03-04T12:23:18.678592+0000","last_became_active":"2024-03-04T12:23:12.910017+0000","last_became_peered":"2024-03-04T12:23:12.910017+0000","last_unstale":"2024-03-04T12:23:18.678592+0000","last_undegraded":"2024-03-04T12:23:18.678592+0000","last_fullsized":"2024-03-04T12:23:18.678592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:15:22.093644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930173+0000","last_change":"2024-03-04T12:23:12.921021+0000","last_active":"2024-03-04T12:23:17.930173+0000","last_peered":"2024-03-04T12:23:17.930173+0000","last_clean":"2024-03-04T12:23:17.930173+0000","last_became_active":"2024-03-04T12:23:12.920600+0000","last_became_peered":"2024-03-04T12:23:12.920600+0000","last_unstale":"2024-03-04T12:23:17.930173+0000","last_undegraded":"2024-03-04T12:23:17.930173+0000","last_fullsized":"2024-03-04T12:23:17.930173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:00:31.706900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816860+0000","last_change":"2024-03-04T12:23:12.915236+0000","last_active":"2024-03-04T12:23:18.816860+0000","last_peered":"2024-03-04T12:23:18.816860+0000","last_clean":"2024-03-04T12:23:18.816860+0000","last_became_active":"2024-03-04T12:23:12.912133+0000","last_became_peered":"2024-03-04T12:23:12.912133+0000","last_unstale":"2024-03-04T12:23:18.816860+0000","last_undegraded":"2024-03-04T12:23:18.816860+0000","last_fullsized":"2024-03-04T12:23:18.816860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:00:10.765556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080486+0000","last_change":"2024-03-04T12:23:12.911394+0000","last_active":"2024-03-04T12:23:19.080486+0000","last_peered":"2024-03-04T12:23:19.080486+0000","last_clean":"2024-03-04T12:23:19.080486+0000","last_became_active":"2024-03-04T12:23:12.911061+0000","last_became_peered":"2024-03-04T12:23:12.911061+0000","last_unstale":"2024-03-04T12:23:19.080486+0000","last_undegraded":"2024-03-04T12:23:19.080486+0000","last_fullsized":"2024-03-04T12:23:19.080486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:39.880383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816399+0000","last_change":"2024-03-04T12:23:12.911356+0000","last_active":"2024-03-04T12:23:18.816399+0000","last_peered":"2024-03-04T12:23:18.816399+0000","last_clean":"2024-03-04T12:23:18.816399+0000","last_became_active":"2024-03-04T12:23:12.911029+0000","last_became_peered":"2024-03-04T12:23:12.911029+0000","last_unstale":"2024-03-04T12:23:18.816399+0000","last_undegraded":"2024-03-04T12:23:18.816399+0000","last_fullsized":"2024-03-04T12:23:18.816399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:52.611625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080428+0000","last_change":"2024-03-04T12:23:12.924921+0000","last_active":"2024-03-04T12:23:19.080428+0000","last_peered":"2024-03-04T12:23:19.080428+0000","last_clean":"2024-03-04T12:23:19.080428+0000","last_became_active":"2024-03-04T12:23:12.924558+0000","last_became_peered":"2024-03-04T12:23:12.924558+0000","last_unstale":"2024-03-04T12:23:19.080428+0000","last_undegraded":"2024-03-04T12:23:19.080428+0000","last_fullsized":"2024-03-04T12:23:19.080428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:10:53.238440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079682+0000","last_change":"2024-03-04T12:23:12.913106+0000","last_active":"2024-03-04T12:23:19.079682+0000","last_peered":"2024-03-04T12:23:19.079682+0000","last_clean":"2024-03-04T12:23:19.079682+0000","last_became_active":"2024-03-04T12:23:12.912741+0000","last_became_peered":"2024-03-04T12:23:12.912741+0000","last_unstale":"2024-03-04T12:23:19.079682+0000","last_undegraded":"2024-03-04T12:23:19.079682+0000","last_fullsized":"2024-03-04T12:23:19.079682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:46:21.406383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678333+0000","last_change":"2024-03-04T12:23:12.918082+0000","last_active":"2024-03-04T12:23:18.678333+0000","last_peered":"2024-03-04T12:23:18.678333+0000","last_clean":"2024-03-04T12:23:18.678333+0000","last_became_active":"2024-03-04T12:23:12.917754+0000","last_became_peered":"2024-03-04T12:23:12.917754+0000","last_unstale":"2024-03-04T12:23:18.678333+0000","last_undegraded":"2024-03-04T12:23:18.678333+0000","last_fullsized":"2024-03-04T12:23:18.678333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:56.601779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286257,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4774,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":87,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":16972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383699,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1071619,"data_compressed":51081,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":478123,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070499,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":477841,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":51081,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":478693,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":481091,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":477319,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":51081,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T12:23:42.688 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T12:23:42.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.820+0000 7f7833f13640 1 -- 172.21.15.60:0/3103394844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f782c101450 msgr2=0x7f782c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.828 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.820+0000 7f7833f13640 1 --2- 172.21.15.60:0/3103394844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f782c101450 0x7f782c1018d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7820009900 tx=0x7f782002bc30 comp rx=0 tx=0).stop 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.820+0000 7f7833f13640 1 -- 172.21.15.60:0/3103394844 shutdown_connections 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.820+0000 7f7833f13640 1 --2- 172.21.15.60:0/3103394844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f782c101e10 0x7f782c1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.820+0000 7f7833f13640 1 --2- 172.21.15.60:0/3103394844 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f782c101450 0x7f782c1018d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.820+0000 7f7833f13640 1 --2- 172.21.15.60:0/3103394844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f782c100260 0x7f782c100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.820+0000 7f7833f13640 1 -- 172.21.15.60:0/3103394844 >> 172.21.15.60:0/3103394844 conn(0x7f782c0fb7f0 msgr2=0x7f782c0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f7833f13640 1 -- 172.21.15.60:0/3103394844 shutdown_connections 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f7833f13640 1 -- 172.21.15.60:0/3103394844 wait complete. 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f7833f13640 1 Processor -- start 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f7833f13640 1 -- start start 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f7833f13640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f782c100260 0x7f782c1041f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f7833f13640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f782c101450 0x7f782c102830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f7833f13640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f782c101e10 0x7f782c102d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f7833f13640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c10bef0 con 0x7f782c101e10 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f7833f13640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c10bd70 con 0x7f782c101450 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f7833f13640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782c10c070 con 0x7f782c100260 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f78324b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f782c101e10 0x7f782c102d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f78324b0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f782c101e10 0x7f782c102d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38058/0 (socket says 172.21.15.60:38058) 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.821+0000 7f78324b0640 1 -- 172.21.15.60:0/1949226242 learned_addr learned my addr 172.21.15.60:0/1949226242 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f7831caf640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f782c100260 0x7f782c1041f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f78314ae640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f782c101450 0x7f782c102830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f78324b0640 1 -- 172.21.15.60:0/1949226242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f782c100260 msgr2=0x7f782c1041f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f78324b0640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f782c100260 0x7f782c1041f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f78324b0640 1 -- 172.21.15.60:0/1949226242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f782c101450 msgr2=0x7f782c102830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f78324b0640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f782c101450 0x7f782c102830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f78324b0640 1 -- 172.21.15.60:0/1949226242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c103600 con 0x7f782c101e10 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f78324b0640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f782c101e10 0x7f782c102d70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f781c00eaa0 tx=0x7f781c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:42.830 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f782affd640 1 -- 172.21.15.60:0/1949226242 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c00cdb0 con 0x7f782c101e10 2024-03-04T12:23:42.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f782affd640 1 -- 172.21.15.60:0/1949226242 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f781c005020 con 0x7f782c101e10 2024-03-04T12:23:42.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f782affd640 1 -- 172.21.15.60:0/1949226242 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781c0116b0 con 0x7f782c101e10 2024-03-04T12:23:42.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f7833f13640 1 -- 172.21.15.60:0/1949226242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782c1a1790 con 0x7f782c101e10 2024-03-04T12:23:42.831 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.822+0000 7f7833f13640 1 -- 172.21.15.60:0/1949226242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782c1a1cb0 con 0x7f782c101e10 2024-03-04T12:23:42.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.823+0000 7f7833f13640 1 -- 172.21.15.60:0/1949226242 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c068600 con 0x7f782c101e10 2024-03-04T12:23:42.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.824+0000 7f782affd640 1 -- 172.21.15.60:0/1949226242 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f781c005560 con 0x7f782c101e10 2024-03-04T12:23:42.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.825+0000 7f782affd640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f78140a1e10 0x7f78140a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:42.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.825+0000 7f7831caf640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f78140a1e10 0x7f78140a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:42.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.825+0000 7f782affd640 1 -- 172.21.15.60:0/1949226242 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f781c0ce7f0 con 0x7f782c101e10 2024-03-04T12:23:42.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.825+0000 7f7831caf640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f78140a1e10 0x7f78140a42d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f782c1012b0 tx=0x7f7824005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:42.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:42.828+0000 7f782affd640 1 -- 172.21.15.60:0/1949226242 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f781c016030 con 0x7f782c101e10 2024-03-04T12:23:43.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.012+0000 7f7833f13640 1 -- 172.21.15.60:0/1949226242 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f782c0fd950 con 0x7f78140a1e10 2024-03-04T12:23:43.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.021+0000 7f782affd640 1 -- 172.21.15.60:0/1949226242 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340564 (secure 0 0 0) 0x7f782c0fd950 con 0x7f78140a1e10 2024-03-04T12:23:43.031 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:43.032 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-04T12:23:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.025+0000 7f7833f13640 1 -- 172.21.15.60:0/1949226242 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f78140a1e10 msgr2=0x7f78140a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.025+0000 7f7833f13640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f78140a1e10 0x7f78140a42d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f782c1012b0 tx=0x7f7824005f90 comp rx=0 tx=0).stop 2024-03-04T12:23:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.025+0000 7f7833f13640 1 -- 172.21.15.60:0/1949226242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f782c101e10 msgr2=0x7f782c102d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.025+0000 7f7833f13640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f782c101e10 0x7f782c102d70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f781c00eaa0 tx=0x7f781c00ef60 comp rx=0 tx=0).stop 2024-03-04T12:23:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.025+0000 7f7833f13640 1 -- 172.21.15.60:0/1949226242 shutdown_connections 2024-03-04T12:23:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.025+0000 7f7833f13640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f78140a1e10 0x7f78140a42d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:43.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.025+0000 7f7833f13640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f782c101e10 0x7f782c102d70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:43.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.025+0000 7f7833f13640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f782c101450 0x7f782c102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:43.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.025+0000 7f7833f13640 1 --2- 172.21.15.60:0/1949226242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f782c100260 0x7f782c1041f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:43.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.025+0000 7f7833f13640 1 -- 172.21.15.60:0/1949226242 >> 172.21.15.60:0/1949226242 conn(0x7f782c0fb7f0 msgr2=0x7f782c0fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:43.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.025+0000 7f7833f13640 1 -- 172.21.15.60:0/1949226242 shutdown_connections 2024-03-04T12:23:43.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:43.025+0000 7f7833f13640 1 -- 172.21.15.60:0/1949226242 wait complete. 2024-03-04T12:23:43.040 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:42 smithi060 ceph-mon[32745]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:43.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:42 smithi060 ceph-mon[32745]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:43.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:42 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] up:active 2024-03-04T12:23:43.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:42 smithi060 ceph-mon[32745]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-04T12:23:43.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:42 smithi060 ceph-mon[32745]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:23:43.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:42 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/797153292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:43.041 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:42 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1222187001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:43.042 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-03-04T12:23:42.159947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595142,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":137,"num_read_kb":120,"num_write":268,"num_write_kb":2966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381556,"kb_used_data":11572,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560268,"statfs":{"total":1151940427776,"available":1151549714432,"internally_reserved":0,"allocated":11849728,"data_stored":7513922,"data_compressed":285408,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":275,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":40,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003371"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678736+0000","last_change":"2024-03-04T12:23:12.926673+0000","last_active":"2024-03-04T12:23:18.678736+0000","last_peered":"2024-03-04T12:23:18.678736+0000","last_clean":"2024-03-04T12:23:18.678736+0000","last_became_active":"2024-03-04T12:23:12.926299+0000","last_became_peered":"2024-03-04T12:23:12.926299+0000","last_unstale":"2024-03-04T12:23:18.678736+0000","last_undegraded":"2024-03-04T12:23:18.678736+0000","last_fullsized":"2024-03-04T12:23:18.678736+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:19.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302937+0000","last_change":"2024-03-04T12:23:12.921758+0000","last_active":"2024-03-04T12:23:19.302937+0000","last_peered":"2024-03-04T12:23:19.302937+0000","last_clean":"2024-03-04T12:23:19.302937+0000","last_became_active":"2024-03-04T12:23:12.921317+0000","last_became_peered":"2024-03-04T12:23:12.921317+0000","last_unstale":"2024-03-04T12:23:19.302937+0000","last_undegraded":"2024-03-04T12:23:19.302937+0000","last_fullsized":"2024-03-04T12:23:19.302937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:20:25.179004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302592+0000","last_change":"2024-03-04T12:23:12.925771+0000","last_active":"2024-03-04T12:23:19.302592+0000","last_peered":"2024-03-04T12:23:19.302592+0000","last_clean":"2024-03-04T12:23:19.302592+0000","last_became_active":"2024-03-04T12:23:12.925447+0000","last_became_peered":"2024-03-04T12:23:12.925447+0000","last_unstale":"2024-03-04T12:23:19.302592+0000","last_undegraded":"2024-03-04T12:23:19.302592+0000","last_fullsized":"2024-03-04T12:23:19.302592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:12.380831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979569+0000","last_change":"2024-03-04T12:23:12.915973+0000","last_active":"2024-03-04T12:23:17.979569+0000","last_peered":"2024-03-04T12:23:17.979569+0000","last_clean":"2024-03-04T12:23:17.979569+0000","last_became_active":"2024-03-04T12:23:12.914914+0000","last_became_peered":"2024-03-04T12:23:12.914914+0000","last_unstale":"2024-03-04T12:23:17.979569+0000","last_undegraded":"2024-03-04T12:23:17.979569+0000","last_fullsized":"2024-03-04T12:23:17.979569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:52:09.908889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079975+0000","last_change":"2024-03-04T12:23:12.921118+0000","last_active":"2024-03-04T12:23:19.079975+0000","last_peered":"2024-03-04T12:23:19.079975+0000","last_clean":"2024-03-04T12:23:19.079975+0000","last_became_active":"2024-03-04T12:23:12.920127+0000","last_became_peered":"2024-03-04T12:23:12.920127+0000","last_unstale":"2024-03-04T12:23:19.079975+0000","last_undegraded":"2024-03-04T12:23:19.079975+0000","last_fullsized":"2024-03-04T12:23:19.079975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:22:10.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079522+0000","last_change":"2024-03-04T12:23:12.916751+0000","last_active":"2024-03-04T12:23:19.079522+0000","last_peered":"2024-03-04T12:23:19.079522+0000","last_clean":"2024-03-04T12:23:19.079522+0000","last_became_active":"2024-03-04T12:23:12.916404+0000","last_became_peered":"2024-03-04T12:23:12.916404+0000","last_unstale":"2024-03-04T12:23:19.079522+0000","last_undegraded":"2024-03-04T12:23:19.079522+0000","last_fullsized":"2024-03-04T12:23:19.079522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:39.998992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930022+0000","last_change":"2024-03-04T12:23:12.909105+0000","last_active":"2024-03-04T12:23:17.930022+0000","last_peered":"2024-03-04T12:23:17.930022+0000","last_clean":"2024-03-04T12:23:17.930022+0000","last_became_active":"2024-03-04T12:23:12.908743+0000","last_became_peered":"2024-03-04T12:23:12.908743+0000","last_unstale":"2024-03-04T12:23:17.930022+0000","last_undegraded":"2024-03-04T12:23:17.930022+0000","last_fullsized":"2024-03-04T12:23:17.930022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:02:44.159796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816254+0000","last_change":"2024-03-04T12:23:12.917627+0000","last_active":"2024-03-04T12:23:18.816254+0000","last_peered":"2024-03-04T12:23:18.816254+0000","last_clean":"2024-03-04T12:23:18.816254+0000","last_became_active":"2024-03-04T12:23:12.917279+0000","last_became_peered":"2024-03-04T12:23:12.917279+0000","last_unstale":"2024-03-04T12:23:18.816254+0000","last_undegraded":"2024-03-04T12:23:18.816254+0000","last_fullsized":"2024-03-04T12:23:18.816254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:33:03.487273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079679+0000","last_change":"2024-03-04T12:23:12.922601+0000","last_active":"2024-03-04T12:23:19.079679+0000","last_peered":"2024-03-04T12:23:19.079679+0000","last_clean":"2024-03-04T12:23:19.079679+0000","last_became_active":"2024-03-04T12:23:12.922037+0000","last_became_peered":"2024-03-04T12:23:12.922037+0000","last_unstale":"2024-03-04T12:23:19.079679+0000","last_undegraded":"2024-03-04T12:23:19.079679+0000","last_fullsized":"2024-03-04T12:23:19.079679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:32.076439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080427+0000","last_change":"2024-03-04T12:23:12.916308+0000","last_active":"2024-03-04T12:23:19.080427+0000","last_peered":"2024-03-04T12:23:19.080427+0000","last_clean":"2024-03-04T12:23:19.080427+0000","last_became_active":"2024-03-04T12:23:12.915897+0000","last_became_peered":"2024-03-04T12:23:12.915897+0000","last_unstale":"2024-03-04T12:23:19.080427+0000","last_undegraded":"2024-03-04T12:23:19.080427+0000","last_fullsized":"2024-03-04T12:23:19.080427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:15:53.133053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930704+0000","last_change":"2024-03-04T12:23:12.930700+0000","last_active":"2024-03-04T12:23:17.930704+0000","last_peered":"2024-03-04T12:23:17.930704+0000","last_clean":"2024-03-04T12:23:17.930704+0000","last_became_active":"2024-03-04T12:23:12.930328+0000","last_became_peered":"2024-03-04T12:23:12.930328+0000","last_unstale":"2024-03-04T12:23:17.930704+0000","last_undegraded":"2024-03-04T12:23:17.930704+0000","last_fullsized":"2024-03-04T12:23:17.930704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:29.990527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979581+0000","last_change":"2024-03-04T12:23:12.915471+0000","last_active":"2024-03-04T12:23:17.979581+0000","last_peered":"2024-03-04T12:23:17.979581+0000","last_clean":"2024-03-04T12:23:17.979581+0000","last_became_active":"2024-03-04T12:23:12.913627+0000","last_became_peered":"2024-03-04T12:23:12.913627+0000","last_unstale":"2024-03-04T12:23:17.979581+0000","last_undegraded":"2024-03-04T12:23:17.979581+0000","last_fullsized":"2024-03-04T12:23:17.979581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:42:56.762782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079647+0000","last_change":"2024-03-04T12:23:12.899143+0000","last_active":"2024-03-04T12:23:19.079647+0000","last_peered":"2024-03-04T12:23:19.079647+0000","last_clean":"2024-03-04T12:23:19.079647+0000","last_became_active":"2024-03-04T12:23:12.898498+0000","last_became_peered":"2024-03-04T12:23:12.898498+0000","last_unstale":"2024-03-04T12:23:19.079647+0000","last_undegraded":"2024-03-04T12:23:19.079647+0000","last_fullsized":"2024-03-04T12:23:19.079647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:33.840683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302557+0000","last_change":"2024-03-04T12:23:12.920873+0000","last_active":"2024-03-04T12:23:19.302557+0000","last_peered":"2024-03-04T12:23:19.302557+0000","last_clean":"2024-03-04T12:23:19.302557+0000","last_became_active":"2024-03-04T12:23:12.919080+0000","last_became_peered":"2024-03-04T12:23:12.919080+0000","last_unstale":"2024-03-04T12:23:19.302557+0000","last_undegraded":"2024-03-04T12:23:19.302557+0000","last_fullsized":"2024-03-04T12:23:19.302557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:47:26.710624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302576+0000","last_change":"2024-03-04T12:23:12.918345+0000","last_active":"2024-03-04T12:23:19.302576+0000","last_peered":"2024-03-04T12:23:19.302576+0000","last_clean":"2024-03-04T12:23:19.302576+0000","last_became_active":"2024-03-04T12:23:12.917967+0000","last_became_peered":"2024-03-04T12:23:12.917967+0000","last_unstale":"2024-03-04T12:23:19.302576+0000","last_undegraded":"2024-03-04T12:23:19.302576+0000","last_fullsized":"2024-03-04T12:23:19.302576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:15.888300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079588+0000","last_change":"2024-03-04T12:23:12.923860+0000","last_active":"2024-03-04T12:23:19.079588+0000","last_peered":"2024-03-04T12:23:19.079588+0000","last_clean":"2024-03-04T12:23:19.079588+0000","last_became_active":"2024-03-04T12:23:12.923530+0000","last_became_peered":"2024-03-04T12:23:12.923530+0000","last_unstale":"2024-03-04T12:23:19.079588+0000","last_undegraded":"2024-03-04T12:23:19.079588+0000","last_fullsized":"2024-03-04T12:23:19.079588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:30:01.466458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678633+0000","last_change":"2024-03-04T12:23:12.927159+0000","last_active":"2024-03-04T12:23:18.678633+0000","last_peered":"2024-03-04T12:23:18.678633+0000","last_clean":"2024-03-04T12:23:18.678633+0000","last_became_active":"2024-03-04T12:23:12.926800+0000","last_became_peered":"2024-03-04T12:23:12.926800+0000","last_unstale":"2024-03-04T12:23:18.678633+0000","last_undegraded":"2024-03-04T12:23:18.678633+0000","last_fullsized":"2024-03-04T12:23:18.678633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:52.633648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930475+0000","last_change":"2024-03-04T12:23:12.920607+0000","last_active":"2024-03-04T12:23:17.930475+0000","last_peered":"2024-03-04T12:23:17.930475+0000","last_clean":"2024-03-04T12:23:17.930475+0000","last_became_active":"2024-03-04T12:23:12.920257+0000","last_became_peered":"2024-03-04T12:23:12.920257+0000","last_unstale":"2024-03-04T12:23:17.930475+0000","last_undegraded":"2024-03-04T12:23:17.930475+0000","last_fullsized":"2024-03-04T12:23:17.930475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:39.353673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930163+0000","last_change":"2024-03-04T12:23:12.906635+0000","last_active":"2024-03-04T12:23:17.930163+0000","last_peered":"2024-03-04T12:23:17.930163+0000","last_clean":"2024-03-04T12:23:17.930163+0000","last_became_active":"2024-03-04T12:23:12.906243+0000","last_became_peered":"2024-03-04T12:23:12.906243+0000","last_unstale":"2024-03-04T12:23:17.930163+0000","last_undegraded":"2024-03-04T12:23:17.930163+0000","last_fullsized":"2024-03-04T12:23:17.930163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:44.375934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929791+0000","last_change":"2024-03-04T12:23:16.951685+0000","last_active":"2024-03-04T12:23:17.929791+0000","last_peered":"2024-03-04T12:23:17.929791+0000","last_clean":"2024-03-04T12:23:17.929791+0000","last_became_active":"2024-03-04T12:23:16.951329+0000","last_became_peered":"2024-03-04T12:23:16.951329+0000","last_unstale":"2024-03-04T12:23:17.929791+0000","last_undegraded":"2024-03-04T12:23:17.929791+0000","last_fullsized":"2024-03-04T12:23:17.929791+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:03:27.233466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'11","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:39.897732+0000","last_change":"2024-03-04T12:23:11.900681+0000","last_active":"2024-03-04T12:23:39.897732+0000","last_peered":"2024-03-04T12:23:39.897732+0000","last_clean":"2024-03-04T12:23:39.897732+0000","last_became_active":"2024-03-04T12:23:11.900365+0000","last_became_peered":"2024-03-04T12:23:11.900365+0000","last_unstale":"2024-03-04T12:23:39.897732+0000","last_undegraded":"2024-03-04T12:23:39.897732+0000","last_fullsized":"2024-03-04T12:23:39.897732+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:53.052513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816892+0000","last_change":"2024-03-04T12:23:12.908525+0000","last_active":"2024-03-04T12:23:18.816892+0000","last_peered":"2024-03-04T12:23:18.816892+0000","last_clean":"2024-03-04T12:23:18.816892+0000","last_became_active":"2024-03-04T12:23:12.908074+0000","last_became_peered":"2024-03-04T12:23:12.908074+0000","last_unstale":"2024-03-04T12:23:18.816892+0000","last_undegraded":"2024-03-04T12:23:18.816892+0000","last_fullsized":"2024-03-04T12:23:18.816892+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:44:43.061750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.978887+0000","last_change":"2024-03-04T12:23:16.953704+0000","last_active":"2024-03-04T12:23:17.978887+0000","last_peered":"2024-03-04T12:23:17.978887+0000","last_clean":"2024-03-04T12:23:17.978887+0000","last_became_active":"2024-03-04T12:23:16.953177+0000","last_became_peered":"2024-03-04T12:23:16.953177+0000","last_unstale":"2024-03-04T12:23:17.978887+0000","last_undegraded":"2024-03-04T12:23:17.978887+0000","last_fullsized":"2024-03-04T12:23:17.978887+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:37:00.469767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.817006+0000","last_change":"2024-03-04T12:23:11.904271+0000","last_active":"2024-03-04T12:23:18.817006+0000","last_peered":"2024-03-04T12:23:18.817006+0000","last_clean":"2024-03-04T12:23:18.817006+0000","last_became_active":"2024-03-04T12:23:11.903963+0000","last_became_peered":"2024-03-04T12:23:11.903963+0000","last_unstale":"2024-03-04T12:23:18.817006+0000","last_undegraded":"2024-03-04T12:23:18.817006+0000","last_fullsized":"2024-03-04T12:23:18.817006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:44.774621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302762+0000","last_change":"2024-03-04T12:23:12.925750+0000","last_active":"2024-03-04T12:23:19.302762+0000","last_peered":"2024-03-04T12:23:19.302762+0000","last_clean":"2024-03-04T12:23:19.302762+0000","last_became_active":"2024-03-04T12:23:12.925380+0000","last_became_peered":"2024-03-04T12:23:12.925380+0000","last_unstale":"2024-03-04T12:23:19.302762+0000","last_undegraded":"2024-03-04T12:23:19.302762+0000","last_fullsized":"2024-03-04T12:23:19.302762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:14.665127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979103+0000","last_change":"2024-03-04T12:23:16.952303+0000","last_active":"2024-03-04T12:23:17.979103+0000","last_peered":"2024-03-04T12:23:17.979103+0000","last_clean":"2024-03-04T12:23:17.979103+0000","last_became_active":"2024-03-04T12:23:16.951666+0000","last_became_peered":"2024-03-04T12:23:16.951666+0000","last_unstale":"2024-03-04T12:23:17.979103+0000","last_undegraded":"2024-03-04T12:23:17.979103+0000","last_fullsized":"2024-03-04T12:23:17.979103+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:41.326808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:39.016126+0000","last_change":"2024-03-04T12:23:11.903871+0000","last_active":"2024-03-04T12:23:39.016126+0000","last_peered":"2024-03-04T12:23:39.016126+0000","last_clean":"2024-03-04T12:23:39.016126+0000","last_became_active":"2024-03-04T12:23:11.902735+0000","last_became_peered":"2024-03-04T12:23:11.902735+0000","last_unstale":"2024-03-04T12:23:39.016126+0000","last_undegraded":"2024-03-04T12:23:39.016126+0000","last_fullsized":"2024-03-04T12:23:39.016126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:45:09.897740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080316+0000","last_change":"2024-03-04T12:23:12.924504+0000","last_active":"2024-03-04T12:23:19.080316+0000","last_peered":"2024-03-04T12:23:19.080316+0000","last_clean":"2024-03-04T12:23:19.080316+0000","last_became_active":"2024-03-04T12:23:12.922934+0000","last_became_peered":"2024-03-04T12:23:12.922934+0000","last_unstale":"2024-03-04T12:23:19.080316+0000","last_undegraded":"2024-03-04T12:23:19.080316+0000","last_fullsized":"2024-03-04T12:23:19.080316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:28.974601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.978937+0000","last_change":"2024-03-04T12:23:16.952324+0000","last_active":"2024-03-04T12:23:17.978937+0000","last_peered":"2024-03-04T12:23:17.978937+0000","last_clean":"2024-03-04T12:23:17.978937+0000","last_became_active":"2024-03-04T12:23:16.951145+0000","last_became_peered":"2024-03-04T12:23:16.951145+0000","last_unstale":"2024-03-04T12:23:17.978937+0000","last_undegraded":"2024-03-04T12:23:17.978937+0000","last_fullsized":"2024-03-04T12:23:17.978937+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:20:47.029523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080345+0000","last_change":"2024-03-04T12:23:11.904391+0000","last_active":"2024-03-04T12:23:19.080345+0000","last_peered":"2024-03-04T12:23:19.080345+0000","last_clean":"2024-03-04T12:23:19.080345+0000","last_became_active":"2024-03-04T12:23:11.903244+0000","last_became_peered":"2024-03-04T12:23:11.903244+0000","last_unstale":"2024-03-04T12:23:19.080345+0000","last_undegraded":"2024-03-04T12:23:19.080345+0000","last_fullsized":"2024-03-04T12:23:19.080345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:27.590418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080535+0000","last_change":"2024-03-04T12:23:12.925623+0000","last_active":"2024-03-04T12:23:19.080535+0000","last_peered":"2024-03-04T12:23:19.080535+0000","last_clean":"2024-03-04T12:23:19.080535+0000","last_became_active":"2024-03-04T12:23:12.925243+0000","last_became_peered":"2024-03-04T12:23:12.925243+0000","last_unstale":"2024-03-04T12:23:19.080535+0000","last_undegraded":"2024-03-04T12:23:19.080535+0000","last_fullsized":"2024-03-04T12:23:19.080535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:11:04.142655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930053+0000","last_change":"2024-03-04T12:23:16.946064+0000","last_active":"2024-03-04T12:23:17.930053+0000","last_peered":"2024-03-04T12:23:17.930053+0000","last_clean":"2024-03-04T12:23:17.930053+0000","last_became_active":"2024-03-04T12:23:16.945342+0000","last_became_peered":"2024-03-04T12:23:16.945342+0000","last_unstale":"2024-03-04T12:23:17.930053+0000","last_undegraded":"2024-03-04T12:23:17.930053+0000","last_fullsized":"2024-03-04T12:23:17.930053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:43:37.500216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080124+0000","last_change":"2024-03-04T12:23:11.899478+0000","last_active":"2024-03-04T12:23:19.080124+0000","last_peered":"2024-03-04T12:23:19.080124+0000","last_clean":"2024-03-04T12:23:19.080124+0000","last_became_active":"2024-03-04T12:23:11.899153+0000","last_became_peered":"2024-03-04T12:23:11.899153+0000","last_unstale":"2024-03-04T12:23:19.080124+0000","last_undegraded":"2024-03-04T12:23:19.080124+0000","last_fullsized":"2024-03-04T12:23:19.080124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:10.709054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080459+0000","last_change":"2024-03-04T12:23:12.908560+0000","last_active":"2024-03-04T12:23:19.080459+0000","last_peered":"2024-03-04T12:23:19.080459+0000","last_clean":"2024-03-04T12:23:19.080459+0000","last_became_active":"2024-03-04T12:23:12.908221+0000","last_became_peered":"2024-03-04T12:23:12.908221+0000","last_unstale":"2024-03-04T12:23:19.080459+0000","last_undegraded":"2024-03-04T12:23:19.080459+0000","last_fullsized":"2024-03-04T12:23:19.080459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:28:06.545945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929928+0000","last_change":"2024-03-04T12:23:16.950976+0000","last_active":"2024-03-04T12:23:17.929928+0000","last_peered":"2024-03-04T12:23:17.929928+0000","last_clean":"2024-03-04T12:23:17.929928+0000","last_became_active":"2024-03-04T12:23:16.950589+0000","last_became_peered":"2024-03-04T12:23:16.950589+0000","last_unstale":"2024-03-04T12:23:17.929928+0000","last_undegraded":"2024-03-04T12:23:17.929928+0000","last_fullsized":"2024-03-04T12:23:17.929928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:23:47.395349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079520+0000","last_change":"2024-03-04T12:23:11.898360+0000","last_active":"2024-03-04T12:23:19.079520+0000","last_peered":"2024-03-04T12:23:19.079520+0000","last_clean":"2024-03-04T12:23:19.079520+0000","last_became_active":"2024-03-04T12:23:11.897988+0000","last_became_peered":"2024-03-04T12:23:11.897988+0000","last_unstale":"2024-03-04T12:23:19.079520+0000","last_undegraded":"2024-03-04T12:23:19.079520+0000","last_fullsized":"2024-03-04T12:23:19.079520+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:17.448032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080442+0000","last_change":"2024-03-04T12:23:12.918392+0000","last_active":"2024-03-04T12:23:19.080442+0000","last_peered":"2024-03-04T12:23:19.080442+0000","last_clean":"2024-03-04T12:23:19.080442+0000","last_became_active":"2024-03-04T12:23:12.917656+0000","last_became_peered":"2024-03-04T12:23:12.917656+0000","last_unstale":"2024-03-04T12:23:19.080442+0000","last_undegraded":"2024-03-04T12:23:19.080442+0000","last_fullsized":"2024-03-04T12:23:19.080442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:44:14.545925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678736+0000","last_change":"2024-03-04T12:23:16.941841+0000","last_active":"2024-03-04T12:23:18.678736+0000","last_peered":"2024-03-04T12:23:18.678736+0000","last_clean":"2024-03-04T12:23:18.678736+0000","last_became_active":"2024-03-04T12:23:16.941449+0000","last_became_peered":"2024-03-04T12:23:16.941449+0000","last_unstale":"2024-03-04T12:23:18.678736+0000","last_undegraded":"2024-03-04T12:23:18.678736+0000","last_fullsized":"2024-03-04T12:23:18.678736+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:50:35.451834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079453+0000","last_change":"2024-03-04T12:23:11.906723+0000","last_active":"2024-03-04T12:23:19.079453+0000","last_peered":"2024-03-04T12:23:19.079453+0000","last_clean":"2024-03-04T12:23:19.079453+0000","last_became_active":"2024-03-04T12:23:11.906357+0000","last_became_peered":"2024-03-04T12:23:11.906357+0000","last_unstale":"2024-03-04T12:23:19.079453+0000","last_undegraded":"2024-03-04T12:23:19.079453+0000","last_fullsized":"2024-03-04T12:23:19.079453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:09.592236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816387+0000","last_change":"2024-03-04T12:23:12.916435+0000","last_active":"2024-03-04T12:23:18.816387+0000","last_peered":"2024-03-04T12:23:18.816387+0000","last_clean":"2024-03-04T12:23:18.816387+0000","last_became_active":"2024-03-04T12:23:12.915607+0000","last_became_peered":"2024-03-04T12:23:12.915607+0000","last_unstale":"2024-03-04T12:23:18.816387+0000","last_undegraded":"2024-03-04T12:23:18.816387+0000","last_fullsized":"2024-03-04T12:23:18.816387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:41:32.900344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302993+0000","last_change":"2024-03-04T12:23:16.944342+0000","last_active":"2024-03-04T12:23:19.302993+0000","last_peered":"2024-03-04T12:23:19.302993+0000","last_clean":"2024-03-04T12:23:19.302993+0000","last_became_active":"2024-03-04T12:23:16.943683+0000","last_became_peered":"2024-03-04T12:23:16.943683+0000","last_unstale":"2024-03-04T12:23:19.302993+0000","last_undegraded":"2024-03-04T12:23:19.302993+0000","last_fullsized":"2024-03-04T12:23:19.302993+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:19.658860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079889+0000","last_change":"2024-03-04T12:23:11.899194+0000","last_active":"2024-03-04T12:23:19.079889+0000","last_peered":"2024-03-04T12:23:19.079889+0000","last_clean":"2024-03-04T12:23:19.079889+0000","last_became_active":"2024-03-04T12:23:11.898109+0000","last_became_peered":"2024-03-04T12:23:11.898109+0000","last_unstale":"2024-03-04T12:23:19.079889+0000","last_undegraded":"2024-03-04T12:23:19.079889+0000","last_fullsized":"2024-03-04T12:23:19.079889+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:08.808202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929727+0000","last_change":"2024-03-04T12:23:12.916549+0000","last_active":"2024-03-04T12:23:17.929727+0000","last_peered":"2024-03-04T12:23:17.929727+0000","last_clean":"2024-03-04T12:23:17.929727+0000","last_became_active":"2024-03-04T12:23:12.916204+0000","last_became_peered":"2024-03-04T12:23:12.916204+0000","last_unstale":"2024-03-04T12:23:17.929727+0000","last_undegraded":"2024-03-04T12:23:17.929727+0000","last_fullsized":"2024-03-04T12:23:17.929727+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:56.412600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816839+0000","last_change":"2024-03-04T12:23:16.945903+0000","last_active":"2024-03-04T12:23:18.816839+0000","last_peered":"2024-03-04T12:23:18.816839+0000","last_clean":"2024-03-04T12:23:18.816839+0000","last_became_active":"2024-03-04T12:23:16.945535+0000","last_became_peered":"2024-03-04T12:23:16.945535+0000","last_unstale":"2024-03-04T12:23:18.816839+0000","last_undegraded":"2024-03-04T12:23:18.816839+0000","last_fullsized":"2024-03-04T12:23:18.816839+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:45:01.178370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080259+0000","last_change":"2024-03-04T12:23:11.901570+0000","last_active":"2024-03-04T12:23:19.080259+0000","last_peered":"2024-03-04T12:23:19.080259+0000","last_clean":"2024-03-04T12:23:19.080259+0000","last_became_active":"2024-03-04T12:23:11.900702+0000","last_became_peered":"2024-03-04T12:23:11.900702+0000","last_unstale":"2024-03-04T12:23:19.080259+0000","last_undegraded":"2024-03-04T12:23:19.080259+0000","last_fullsized":"2024-03-04T12:23:19.080259+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:18.758453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080223+0000","last_change":"2024-03-04T12:23:12.919571+0000","last_active":"2024-03-04T12:23:19.080223+0000","last_peered":"2024-03-04T12:23:19.080223+0000","last_clean":"2024-03-04T12:23:19.080223+0000","last_became_active":"2024-03-04T12:23:12.919143+0000","last_became_peered":"2024-03-04T12:23:12.919143+0000","last_unstale":"2024-03-04T12:23:19.080223+0000","last_undegraded":"2024-03-04T12:23:19.080223+0000","last_fullsized":"2024-03-04T12:23:19.080223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:25.459417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929914+0000","last_change":"2024-03-04T12:23:16.946380+0000","last_active":"2024-03-04T12:23:17.929914+0000","last_peered":"2024-03-04T12:23:17.929914+0000","last_clean":"2024-03-04T12:23:17.929914+0000","last_became_active":"2024-03-04T12:23:16.945983+0000","last_became_peered":"2024-03-04T12:23:16.945983+0000","last_unstale":"2024-03-04T12:23:17.929914+0000","last_undegraded":"2024-03-04T12:23:17.929914+0000","last_fullsized":"2024-03-04T12:23:17.929914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:39:43.297619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:38.006808+0000","last_change":"2024-03-04T12:23:11.897176+0000","last_active":"2024-03-04T12:23:38.006808+0000","last_peered":"2024-03-04T12:23:38.006808+0000","last_clean":"2024-03-04T12:23:38.006808+0000","last_became_active":"2024-03-04T12:23:11.896747+0000","last_became_peered":"2024-03-04T12:23:11.896747+0000","last_unstale":"2024-03-04T12:23:38.006808+0000","last_undegraded":"2024-03-04T12:23:38.006808+0000","last_fullsized":"2024-03-04T12:23:38.006808+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:02.206704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302794+0000","last_change":"2024-03-04T12:23:12.922021+0000","last_active":"2024-03-04T12:23:19.302794+0000","last_peered":"2024-03-04T12:23:19.302794+0000","last_clean":"2024-03-04T12:23:19.302794+0000","last_became_active":"2024-03-04T12:23:12.918724+0000","last_became_peered":"2024-03-04T12:23:12.918724+0000","last_unstale":"2024-03-04T12:23:19.302794+0000","last_undegraded":"2024-03-04T12:23:19.302794+0000","last_fullsized":"2024-03-04T12:23:19.302794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:49:09.017979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979459+0000","last_change":"2024-03-04T12:23:16.943271+0000","last_active":"2024-03-04T12:23:17.979459+0000","last_peered":"2024-03-04T12:23:17.979459+0000","last_clean":"2024-03-04T12:23:17.979459+0000","last_became_active":"2024-03-04T12:23:16.942400+0000","last_became_peered":"2024-03-04T12:23:16.942400+0000","last_unstale":"2024-03-04T12:23:17.979459+0000","last_undegraded":"2024-03-04T12:23:17.979459+0000","last_fullsized":"2024-03-04T12:23:17.979459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:06.010960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080291+0000","last_change":"2024-03-04T12:23:11.896706+0000","last_active":"2024-03-04T12:23:19.080291+0000","last_peered":"2024-03-04T12:23:19.080291+0000","last_clean":"2024-03-04T12:23:19.080291+0000","last_became_active":"2024-03-04T12:23:11.896100+0000","last_became_peered":"2024-03-04T12:23:11.896100+0000","last_unstale":"2024-03-04T12:23:19.080291+0000","last_undegraded":"2024-03-04T12:23:19.080291+0000","last_fullsized":"2024-03-04T12:23:19.080291+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:42:19.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080422+0000","last_change":"2024-03-04T12:23:12.923631+0000","last_active":"2024-03-04T12:23:19.080422+0000","last_peered":"2024-03-04T12:23:19.080422+0000","last_clean":"2024-03-04T12:23:19.080422+0000","last_became_active":"2024-03-04T12:23:12.922487+0000","last_became_peered":"2024-03-04T12:23:12.922487+0000","last_unstale":"2024-03-04T12:23:19.080422+0000","last_undegraded":"2024-03-04T12:23:19.080422+0000","last_fullsized":"2024-03-04T12:23:19.080422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:30:33.923064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929999+0000","last_change":"2024-03-04T12:23:16.948423+0000","last_active":"2024-03-04T12:23:17.929999+0000","last_peered":"2024-03-04T12:23:17.929999+0000","last_clean":"2024-03-04T12:23:17.929999+0000","last_became_active":"2024-03-04T12:23:16.947922+0000","last_became_peered":"2024-03-04T12:23:16.947922+0000","last_unstale":"2024-03-04T12:23:17.929999+0000","last_undegraded":"2024-03-04T12:23:17.929999+0000","last_fullsized":"2024-03-04T12:23:17.929999+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:13:05.145310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:29.803757+0000","last_change":"2024-03-04T12:23:11.898903+0000","last_active":"2024-03-04T12:23:29.803757+0000","last_peered":"2024-03-04T12:23:29.803757+0000","last_clean":"2024-03-04T12:23:29.803757+0000","last_became_active":"2024-03-04T12:23:11.898477+0000","last_became_peered":"2024-03-04T12:23:11.898477+0000","last_unstale":"2024-03-04T12:23:29.803757+0000","last_undegraded":"2024-03-04T12:23:29.803757+0000","last_fullsized":"2024-03-04T12:23:29.803757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:36:58.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080486+0000","last_change":"2024-03-04T12:23:12.908497+0000","last_active":"2024-03-04T12:23:19.080486+0000","last_peered":"2024-03-04T12:23:19.080486+0000","last_clean":"2024-03-04T12:23:19.080486+0000","last_became_active":"2024-03-04T12:23:12.907847+0000","last_became_peered":"2024-03-04T12:23:12.907847+0000","last_unstale":"2024-03-04T12:23:19.080486+0000","last_undegraded":"2024-03-04T12:23:19.080486+0000","last_fullsized":"2024-03-04T12:23:19.080486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:51:44.592851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.081090+0000","last_change":"2024-03-04T12:23:16.951844+0000","last_active":"2024-03-04T12:23:19.081090+0000","last_peered":"2024-03-04T12:23:19.081090+0000","last_clean":"2024-03-04T12:23:19.081090+0000","last_became_active":"2024-03-04T12:23:16.951491+0000","last_became_peered":"2024-03-04T12:23:16.951491+0000","last_unstale":"2024-03-04T12:23:19.081090+0000","last_undegraded":"2024-03-04T12:23:19.081090+0000","last_fullsized":"2024-03-04T12:23:19.081090+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:54.098239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929707+0000","last_change":"2024-03-04T12:23:11.899592+0000","last_active":"2024-03-04T12:23:17.929707+0000","last_peered":"2024-03-04T12:23:17.929707+0000","last_clean":"2024-03-04T12:23:17.929707+0000","last_became_active":"2024-03-04T12:23:11.899275+0000","last_became_peered":"2024-03-04T12:23:11.899275+0000","last_unstale":"2024-03-04T12:23:17.929707+0000","last_undegraded":"2024-03-04T12:23:17.929707+0000","last_fullsized":"2024-03-04T12:23:17.929707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:02.184268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079473+0000","last_change":"2024-03-04T12:23:12.910084+0000","last_active":"2024-03-04T12:23:19.079473+0000","last_peered":"2024-03-04T12:23:19.079473+0000","last_clean":"2024-03-04T12:23:19.079473+0000","last_became_active":"2024-03-04T12:23:12.909719+0000","last_became_peered":"2024-03-04T12:23:12.909719+0000","last_unstale":"2024-03-04T12:23:19.079473+0000","last_undegraded":"2024-03-04T12:23:19.079473+0000","last_fullsized":"2024-03-04T12:23:19.079473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:16:41.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678458+0000","last_change":"2024-03-04T12:23:16.942680+0000","last_active":"2024-03-04T12:23:18.678458+0000","last_peered":"2024-03-04T12:23:18.678458+0000","last_clean":"2024-03-04T12:23:18.678458+0000","last_became_active":"2024-03-04T12:23:16.942307+0000","last_became_peered":"2024-03-04T12:23:16.942307+0000","last_unstale":"2024-03-04T12:23:18.678458+0000","last_undegraded":"2024-03-04T12:23:18.678458+0000","last_fullsized":"2024-03-04T12:23:18.678458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:33:18.390621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080671+0000","last_change":"2024-03-04T12:23:11.897506+0000","last_active":"2024-03-04T12:23:19.080671+0000","last_peered":"2024-03-04T12:23:19.080671+0000","last_clean":"2024-03-04T12:23:19.080671+0000","last_became_active":"2024-03-04T12:23:11.897090+0000","last_became_peered":"2024-03-04T12:23:11.897090+0000","last_unstale":"2024-03-04T12:23:19.080671+0000","last_undegraded":"2024-03-04T12:23:19.080671+0000","last_fullsized":"2024-03-04T12:23:19.080671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:10:27.868713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930408+0000","last_change":"2024-03-04T12:23:12.921825+0000","last_active":"2024-03-04T12:23:17.930408+0000","last_peered":"2024-03-04T12:23:17.930408+0000","last_clean":"2024-03-04T12:23:17.930408+0000","last_became_active":"2024-03-04T12:23:12.921491+0000","last_became_peered":"2024-03-04T12:23:12.921491+0000","last_unstale":"2024-03-04T12:23:17.930408+0000","last_undegraded":"2024-03-04T12:23:17.930408+0000","last_fullsized":"2024-03-04T12:23:17.930408+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:17:08.203502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930232+0000","last_change":"2024-03-04T12:23:16.951920+0000","last_active":"2024-03-04T12:23:17.930232+0000","last_peered":"2024-03-04T12:23:17.930232+0000","last_clean":"2024-03-04T12:23:17.930232+0000","last_became_active":"2024-03-04T12:23:16.951507+0000","last_became_peered":"2024-03-04T12:23:16.951507+0000","last_unstale":"2024-03-04T12:23:17.930232+0000","last_undegraded":"2024-03-04T12:23:17.930232+0000","last_fullsized":"2024-03-04T12:23:17.930232+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:27.673692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.978889+0000","last_change":"2024-03-04T12:23:11.897722+0000","last_active":"2024-03-04T12:23:17.978889+0000","last_peered":"2024-03-04T12:23:17.978889+0000","last_clean":"2024-03-04T12:23:17.978889+0000","last_became_active":"2024-03-04T12:23:11.897276+0000","last_became_peered":"2024-03-04T12:23:11.897276+0000","last_unstale":"2024-03-04T12:23:17.978889+0000","last_undegraded":"2024-03-04T12:23:17.978889+0000","last_fullsized":"2024-03-04T12:23:17.978889+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:37.968804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930042+0000","last_change":"2024-03-04T12:23:12.914773+0000","last_active":"2024-03-04T12:23:17.930042+0000","last_peered":"2024-03-04T12:23:17.930042+0000","last_clean":"2024-03-04T12:23:17.930042+0000","last_became_active":"2024-03-04T12:23:12.914431+0000","last_became_peered":"2024-03-04T12:23:12.914431+0000","last_unstale":"2024-03-04T12:23:17.930042+0000","last_undegraded":"2024-03-04T12:23:17.930042+0000","last_fullsized":"2024-03-04T12:23:17.930042+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:50.234491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079976+0000","last_change":"2024-03-04T12:23:16.952405+0000","last_active":"2024-03-04T12:23:19.079976+0000","last_peered":"2024-03-04T12:23:19.079976+0000","last_clean":"2024-03-04T12:23:19.079976+0000","last_became_active":"2024-03-04T12:23:16.951915+0000","last_became_peered":"2024-03-04T12:23:16.951915+0000","last_unstale":"2024-03-04T12:23:19.079976+0000","last_undegraded":"2024-03-04T12:23:19.079976+0000","last_fullsized":"2024-03-04T12:23:19.079976+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:38:03.765883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:39.016559+0000","last_change":"2024-03-04T12:23:11.903267+0000","last_active":"2024-03-04T12:23:39.016559+0000","last_peered":"2024-03-04T12:23:39.016559+0000","last_clean":"2024-03-04T12:23:39.016559+0000","last_became_active":"2024-03-04T12:23:11.902870+0000","last_became_peered":"2024-03-04T12:23:11.902870+0000","last_unstale":"2024-03-04T12:23:39.016559+0000","last_undegraded":"2024-03-04T12:23:39.016559+0000","last_fullsized":"2024-03-04T12:23:39.016559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:33:43.679869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080445+0000","last_change":"2024-03-04T12:23:12.925303+0000","last_active":"2024-03-04T12:23:19.080445+0000","last_peered":"2024-03-04T12:23:19.080445+0000","last_clean":"2024-03-04T12:23:19.080445+0000","last_became_active":"2024-03-04T12:23:12.924945+0000","last_became_peered":"2024-03-04T12:23:12.924945+0000","last_unstale":"2024-03-04T12:23:19.080445+0000","last_undegraded":"2024-03-04T12:23:19.080445+0000","last_fullsized":"2024-03-04T12:23:19.080445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:49.680925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080355+0000","last_change":"2024-03-04T12:23:16.951457+0000","last_active":"2024-03-04T12:23:19.080355+0000","last_peered":"2024-03-04T12:23:19.080355+0000","last_clean":"2024-03-04T12:23:19.080355+0000","last_became_active":"2024-03-04T12:23:16.950835+0000","last_became_peered":"2024-03-04T12:23:16.950835+0000","last_unstale":"2024-03-04T12:23:19.080355+0000","last_undegraded":"2024-03-04T12:23:19.080355+0000","last_fullsized":"2024-03-04T12:23:19.080355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:39.337275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080614+0000","last_change":"2024-03-04T12:23:11.903718+0000","last_active":"2024-03-04T12:23:19.080614+0000","last_peered":"2024-03-04T12:23:19.080614+0000","last_clean":"2024-03-04T12:23:19.080614+0000","last_became_active":"2024-03-04T12:23:11.902814+0000","last_became_peered":"2024-03-04T12:23:11.902814+0000","last_unstale":"2024-03-04T12:23:19.080614+0000","last_undegraded":"2024-03-04T12:23:19.080614+0000","last_fullsized":"2024-03-04T12:23:19.080614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:40:30.978576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678460+0000","last_change":"2024-03-04T12:23:12.919020+0000","last_active":"2024-03-04T12:23:18.678460+0000","last_peered":"2024-03-04T12:23:18.678460+0000","last_clean":"2024-03-04T12:23:18.678460+0000","last_became_active":"2024-03-04T12:23:12.918661+0000","last_became_peered":"2024-03-04T12:23:12.918661+0000","last_unstale":"2024-03-04T12:23:18.678460+0000","last_undegraded":"2024-03-04T12:23:18.678460+0000","last_fullsized":"2024-03-04T12:23:18.678460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:27:15.869681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080398+0000","last_change":"2024-03-04T12:23:16.950046+0000","last_active":"2024-03-04T12:23:19.080398+0000","last_peered":"2024-03-04T12:23:19.080398+0000","last_clean":"2024-03-04T12:23:19.080398+0000","last_became_active":"2024-03-04T12:23:16.949632+0000","last_became_peered":"2024-03-04T12:23:16.949632+0000","last_unstale":"2024-03-04T12:23:19.080398+0000","last_undegraded":"2024-03-04T12:23:19.080398+0000","last_fullsized":"2024-03-04T12:23:19.080398+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:25.817212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930210+0000","last_change":"2024-03-04T12:23:11.897016+0000","last_active":"2024-03-04T12:23:17.930210+0000","last_peered":"2024-03-04T12:23:17.930210+0000","last_clean":"2024-03-04T12:23:17.930210+0000","last_became_active":"2024-03-04T12:23:11.896668+0000","last_became_peered":"2024-03-04T12:23:11.896668+0000","last_unstale":"2024-03-04T12:23:17.930210+0000","last_undegraded":"2024-03-04T12:23:17.930210+0000","last_fullsized":"2024-03-04T12:23:17.930210+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:50:01.089658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080037+0000","last_change":"2024-03-04T12:23:12.908169+0000","last_active":"2024-03-04T12:23:19.080037+0000","last_peered":"2024-03-04T12:23:19.080037+0000","last_clean":"2024-03-04T12:23:19.080037+0000","last_became_active":"2024-03-04T12:23:12.907786+0000","last_became_peered":"2024-03-04T12:23:12.907786+0000","last_unstale":"2024-03-04T12:23:19.080037+0000","last_undegraded":"2024-03-04T12:23:19.080037+0000","last_fullsized":"2024-03-04T12:23:19.080037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:10.602216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079873+0000","last_change":"2024-03-04T12:23:16.944232+0000","last_active":"2024-03-04T12:23:19.079873+0000","last_peered":"2024-03-04T12:23:19.079873+0000","last_clean":"2024-03-04T12:23:19.079873+0000","last_became_active":"2024-03-04T12:23:16.943855+0000","last_became_peered":"2024-03-04T12:23:16.943855+0000","last_unstale":"2024-03-04T12:23:19.079873+0000","last_undegraded":"2024-03-04T12:23:19.079873+0000","last_fullsized":"2024-03-04T12:23:19.079873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:37:12.083622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930109+0000","last_change":"2024-03-04T12:23:11.899742+0000","last_active":"2024-03-04T12:23:17.930109+0000","last_peered":"2024-03-04T12:23:17.930109+0000","last_clean":"2024-03-04T12:23:17.930109+0000","last_became_active":"2024-03-04T12:23:11.898445+0000","last_became_peered":"2024-03-04T12:23:11.898445+0000","last_unstale":"2024-03-04T12:23:17.930109+0000","last_undegraded":"2024-03-04T12:23:17.930109+0000","last_fullsized":"2024-03-04T12:23:17.930109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:13.128348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079830+0000","last_change":"2024-03-04T12:23:12.921321+0000","last_active":"2024-03-04T12:23:19.079830+0000","last_peered":"2024-03-04T12:23:19.079830+0000","last_clean":"2024-03-04T12:23:19.079830+0000","last_became_active":"2024-03-04T12:23:12.918119+0000","last_became_peered":"2024-03-04T12:23:12.918119+0000","last_unstale":"2024-03-04T12:23:19.079830+0000","last_undegraded":"2024-03-04T12:23:19.079830+0000","last_fullsized":"2024-03-04T12:23:19.079830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:31.693583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079873+0000","last_change":"2024-03-04T12:23:16.943887+0000","last_active":"2024-03-04T12:23:19.079873+0000","last_peered":"2024-03-04T12:23:19.079873+0000","last_clean":"2024-03-04T12:23:19.079873+0000","last_became_active":"2024-03-04T12:23:16.943487+0000","last_became_peered":"2024-03-04T12:23:16.943487+0000","last_unstale":"2024-03-04T12:23:19.079873+0000","last_undegraded":"2024-03-04T12:23:19.079873+0000","last_fullsized":"2024-03-04T12:23:19.079873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:12.451176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080043+0000","last_change":"2024-03-04T12:23:12.918363+0000","last_active":"2024-03-04T12:23:19.080043+0000","last_peered":"2024-03-04T12:23:19.080043+0000","last_clean":"2024-03-04T12:23:19.080043+0000","last_became_active":"2024-03-04T12:23:12.917563+0000","last_became_peered":"2024-03-04T12:23:12.917563+0000","last_unstale":"2024-03-04T12:23:19.080043+0000","last_undegraded":"2024-03-04T12:23:19.080043+0000","last_fullsized":"2024-03-04T12:23:19.080043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:26:35.297513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079397+0000","last_change":"2024-03-04T12:23:11.904072+0000","last_active":"2024-03-04T12:23:19.079397+0000","last_peered":"2024-03-04T12:23:19.079397+0000","last_clean":"2024-03-04T12:23:19.079397+0000","last_became_active":"2024-03-04T12:23:11.903097+0000","last_became_peered":"2024-03-04T12:23:11.903097+0000","last_unstale":"2024-03-04T12:23:19.079397+0000","last_undegraded":"2024-03-04T12:23:19.079397+0000","last_fullsized":"2024-03-04T12:23:19.079397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:36.606667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930670+0000","last_change":"2024-03-04T12:23:16.951019+0000","last_active":"2024-03-04T12:23:17.930670+0000","last_peered":"2024-03-04T12:23:17.930670+0000","last_clean":"2024-03-04T12:23:17.930670+0000","last_became_active":"2024-03-04T12:23:16.950524+0000","last_became_peered":"2024-03-04T12:23:16.950524+0000","last_unstale":"2024-03-04T12:23:17.930670+0000","last_undegraded":"2024-03-04T12:23:17.930670+0000","last_fullsized":"2024-03-04T12:23:17.930670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:48.376539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816413+0000","last_change":"2024-03-04T12:23:12.927615+0000","last_active":"2024-03-04T12:23:18.816413+0000","last_peered":"2024-03-04T12:23:18.816413+0000","last_clean":"2024-03-04T12:23:18.816413+0000","last_became_active":"2024-03-04T12:23:12.927190+0000","last_became_peered":"2024-03-04T12:23:12.927190+0000","last_unstale":"2024-03-04T12:23:18.816413+0000","last_undegraded":"2024-03-04T12:23:18.816413+0000","last_fullsized":"2024-03-04T12:23:18.816413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:54:19.019030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979154+0000","last_change":"2024-03-04T12:23:11.893542+0000","last_active":"2024-03-04T12:23:17.979154+0000","last_peered":"2024-03-04T12:23:17.979154+0000","last_clean":"2024-03-04T12:23:17.979154+0000","last_became_active":"2024-03-04T12:23:11.893166+0000","last_became_peered":"2024-03-04T12:23:11.893166+0000","last_unstale":"2024-03-04T12:23:17.979154+0000","last_undegraded":"2024-03-04T12:23:17.979154+0000","last_fullsized":"2024-03-04T12:23:17.979154+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:32.117863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929900+0000","last_change":"2024-03-04T12:21:55.865103+0000","last_active":"2024-03-04T12:23:17.929900+0000","last_peered":"2024-03-04T12:23:17.929900+0000","last_clean":"2024-03-04T12:23:17.929900+0000","last_became_active":"2024-03-04T12:21:55.864553+0000","last_became_peered":"2024-03-04T12:21:55.864553+0000","last_unstale":"2024-03-04T12:23:17.929900+0000","last_undegraded":"2024-03-04T12:23:17.929900+0000","last_fullsized":"2024-03-04T12:23:17.929900+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_clean_scrub_stamp":"2024-03-04T12:18:28.442423+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:31.052782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.081061+0000","last_change":"2024-03-04T12:23:16.945852+0000","last_active":"2024-03-04T12:23:19.081061+0000","last_peered":"2024-03-04T12:23:19.081061+0000","last_clean":"2024-03-04T12:23:19.081061+0000","last_became_active":"2024-03-04T12:23:16.945496+0000","last_became_peered":"2024-03-04T12:23:16.945496+0000","last_unstale":"2024-03-04T12:23:19.081061+0000","last_undegraded":"2024-03-04T12:23:19.081061+0000","last_fullsized":"2024-03-04T12:23:19.081061+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:35:10.880757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079323+0000","last_change":"2024-03-04T12:23:12.919219+0000","last_active":"2024-03-04T12:23:19.079323+0000","last_peered":"2024-03-04T12:23:19.079323+0000","last_clean":"2024-03-04T12:23:19.079323+0000","last_became_active":"2024-03-04T12:23:12.918897+0000","last_became_peered":"2024-03-04T12:23:12.918897+0000","last_unstale":"2024-03-04T12:23:19.079323+0000","last_undegraded":"2024-03-04T12:23:19.079323+0000","last_fullsized":"2024-03-04T12:23:19.079323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:24:32.594172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930044+0000","last_change":"2024-03-04T12:23:11.894544+0000","last_active":"2024-03-04T12:23:17.930044+0000","last_peered":"2024-03-04T12:23:17.930044+0000","last_clean":"2024-03-04T12:23:17.930044+0000","last_became_active":"2024-03-04T12:23:11.894188+0000","last_became_peered":"2024-03-04T12:23:11.894188+0000","last_unstale":"2024-03-04T12:23:17.930044+0000","last_undegraded":"2024-03-04T12:23:17.930044+0000","last_fullsized":"2024-03-04T12:23:17.930044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:28:14.687530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678468+0000","last_change":"2024-03-04T12:23:16.945510+0000","last_active":"2024-03-04T12:23:18.678468+0000","last_peered":"2024-03-04T12:23:18.678468+0000","last_clean":"2024-03-04T12:23:18.678468+0000","last_became_active":"2024-03-04T12:23:16.945140+0000","last_became_peered":"2024-03-04T12:23:16.945140+0000","last_unstale":"2024-03-04T12:23:18.678468+0000","last_undegraded":"2024-03-04T12:23:18.678468+0000","last_fullsized":"2024-03-04T12:23:18.678468+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:44.102503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080574+0000","last_change":"2024-03-04T12:23:11.904515+0000","last_active":"2024-03-04T12:23:19.080574+0000","last_peered":"2024-03-04T12:23:19.080574+0000","last_clean":"2024-03-04T12:23:19.080574+0000","last_became_active":"2024-03-04T12:23:11.904149+0000","last_became_peered":"2024-03-04T12:23:11.904149+0000","last_unstale":"2024-03-04T12:23:19.080574+0000","last_undegraded":"2024-03-04T12:23:19.080574+0000","last_fullsized":"2024-03-04T12:23:19.080574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:04:51.382626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079956+0000","last_change":"2024-03-04T12:23:12.908749+0000","last_active":"2024-03-04T12:23:19.079956+0000","last_peered":"2024-03-04T12:23:19.079956+0000","last_clean":"2024-03-04T12:23:19.079956+0000","last_became_active":"2024-03-04T12:23:12.908405+0000","last_became_peered":"2024-03-04T12:23:12.908405+0000","last_unstale":"2024-03-04T12:23:19.079956+0000","last_undegraded":"2024-03-04T12:23:19.079956+0000","last_fullsized":"2024-03-04T12:23:19.079956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:10.472953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816434+0000","last_change":"2024-03-04T12:23:16.951145+0000","last_active":"2024-03-04T12:23:18.816434+0000","last_peered":"2024-03-04T12:23:18.816434+0000","last_clean":"2024-03-04T12:23:18.816434+0000","last_became_active":"2024-03-04T12:23:16.950702+0000","last_became_peered":"2024-03-04T12:23:16.950702+0000","last_unstale":"2024-03-04T12:23:18.816434+0000","last_undegraded":"2024-03-04T12:23:18.816434+0000","last_fullsized":"2024-03-04T12:23:18.816434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:42.720616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:39.898638+0000","last_change":"2024-03-04T12:23:11.893810+0000","last_active":"2024-03-04T12:23:39.898638+0000","last_peered":"2024-03-04T12:23:39.898638+0000","last_clean":"2024-03-04T12:23:39.898638+0000","last_became_active":"2024-03-04T12:23:11.893465+0000","last_became_peered":"2024-03-04T12:23:11.893465+0000","last_unstale":"2024-03-04T12:23:39.898638+0000","last_undegraded":"2024-03-04T12:23:39.898638+0000","last_fullsized":"2024-03-04T12:23:39.898638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:27:41.199786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816750+0000","last_change":"2024-03-04T12:23:12.925763+0000","last_active":"2024-03-04T12:23:18.816750+0000","last_peered":"2024-03-04T12:23:18.816750+0000","last_clean":"2024-03-04T12:23:18.816750+0000","last_became_active":"2024-03-04T12:23:12.925429+0000","last_became_peered":"2024-03-04T12:23:12.925429+0000","last_unstale":"2024-03-04T12:23:18.816750+0000","last_undegraded":"2024-03-04T12:23:18.816750+0000","last_fullsized":"2024-03-04T12:23:18.816750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:46.814459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929720+0000","last_change":"2024-03-04T12:23:16.942624+0000","last_active":"2024-03-04T12:23:17.929720+0000","last_peered":"2024-03-04T12:23:17.929720+0000","last_clean":"2024-03-04T12:23:17.929720+0000","last_became_active":"2024-03-04T12:23:16.942287+0000","last_became_peered":"2024-03-04T12:23:16.942287+0000","last_unstale":"2024-03-04T12:23:17.929720+0000","last_undegraded":"2024-03-04T12:23:17.929720+0000","last_fullsized":"2024-03-04T12:23:17.929720+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:59.479611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979526+0000","last_change":"2024-03-04T12:23:11.891504+0000","last_active":"2024-03-04T12:23:17.979526+0000","last_peered":"2024-03-04T12:23:17.979526+0000","last_clean":"2024-03-04T12:23:17.979526+0000","last_became_active":"2024-03-04T12:23:11.891134+0000","last_became_peered":"2024-03-04T12:23:11.891134+0000","last_unstale":"2024-03-04T12:23:17.979526+0000","last_undegraded":"2024-03-04T12:23:17.979526+0000","last_fullsized":"2024-03-04T12:23:17.979526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:41.481003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.303017+0000","last_change":"2024-03-04T12:23:12.928989+0000","last_active":"2024-03-04T12:23:19.303017+0000","last_peered":"2024-03-04T12:23:19.303017+0000","last_clean":"2024-03-04T12:23:19.303017+0000","last_became_active":"2024-03-04T12:23:12.928609+0000","last_became_peered":"2024-03-04T12:23:12.928609+0000","last_unstale":"2024-03-04T12:23:19.303017+0000","last_undegraded":"2024-03-04T12:23:19.303017+0000","last_fullsized":"2024-03-04T12:23:19.303017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:39.979546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979327+0000","last_change":"2024-03-04T12:23:16.949590+0000","last_active":"2024-03-04T12:23:17.979327+0000","last_peered":"2024-03-04T12:23:17.979327+0000","last_clean":"2024-03-04T12:23:17.979327+0000","last_became_active":"2024-03-04T12:23:16.949086+0000","last_became_peered":"2024-03-04T12:23:16.949086+0000","last_unstale":"2024-03-04T12:23:17.979327+0000","last_undegraded":"2024-03-04T12:23:17.979327+0000","last_fullsized":"2024-03-04T12:23:17.979327+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:07:09.819097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:38.005174+0000","last_change":"2024-03-04T12:23:11.894110+0000","last_active":"2024-03-04T12:23:38.005174+0000","last_peered":"2024-03-04T12:23:38.005174+0000","last_clean":"2024-03-04T12:23:38.005174+0000","last_became_active":"2024-03-04T12:23:11.893419+0000","last_became_peered":"2024-03-04T12:23:11.893419+0000","last_unstale":"2024-03-04T12:23:38.005174+0000","last_undegraded":"2024-03-04T12:23:38.005174+0000","last_fullsized":"2024-03-04T12:23:38.005174+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:53.963989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079749+0000","last_change":"2024-03-04T12:23:12.907403+0000","last_active":"2024-03-04T12:23:19.079749+0000","last_peered":"2024-03-04T12:23:19.079749+0000","last_clean":"2024-03-04T12:23:19.079749+0000","last_became_active":"2024-03-04T12:23:12.906674+0000","last_became_peered":"2024-03-04T12:23:12.906674+0000","last_unstale":"2024-03-04T12:23:19.079749+0000","last_undegraded":"2024-03-04T12:23:19.079749+0000","last_fullsized":"2024-03-04T12:23:19.079749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:49:01.133676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930542+0000","last_change":"2024-03-04T12:23:16.955113+0000","last_active":"2024-03-04T12:23:17.930542+0000","last_peered":"2024-03-04T12:23:17.930542+0000","last_clean":"2024-03-04T12:23:17.930542+0000","last_became_active":"2024-03-04T12:23:16.951624+0000","last_became_peered":"2024-03-04T12:23:16.951624+0000","last_unstale":"2024-03-04T12:23:17.930542+0000","last_undegraded":"2024-03-04T12:23:17.930542+0000","last_fullsized":"2024-03-04T12:23:17.930542+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:22:23.935460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816401+0000","last_change":"2024-03-04T12:23:11.900213+0000","last_active":"2024-03-04T12:23:18.816401+0000","last_peered":"2024-03-04T12:23:18.816401+0000","last_clean":"2024-03-04T12:23:18.816401+0000","last_became_active":"2024-03-04T12:23:11.899832+0000","last_became_peered":"2024-03-04T12:23:11.899832+0000","last_unstale":"2024-03-04T12:23:18.816401+0000","last_undegraded":"2024-03-04T12:23:18.816401+0000","last_fullsized":"2024-03-04T12:23:18.816401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:55:24.553309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678159+0000","last_change":"2024-03-04T12:23:12.913315+0000","last_active":"2024-03-04T12:23:18.678159+0000","last_peered":"2024-03-04T12:23:18.678159+0000","last_clean":"2024-03-04T12:23:18.678159+0000","last_became_active":"2024-03-04T12:23:12.912850+0000","last_became_peered":"2024-03-04T12:23:12.912850+0000","last_unstale":"2024-03-04T12:23:18.678159+0000","last_undegraded":"2024-03-04T12:23:18.678159+0000","last_fullsized":"2024-03-04T12:23:18.678159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:55.140866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979207+0000","last_change":"2024-03-04T12:23:16.943295+0000","last_active":"2024-03-04T12:23:17.979207+0000","last_peered":"2024-03-04T12:23:17.979207+0000","last_clean":"2024-03-04T12:23:17.979207+0000","last_became_active":"2024-03-04T12:23:16.942853+0000","last_became_peered":"2024-03-04T12:23:16.942853+0000","last_unstale":"2024-03-04T12:23:17.979207+0000","last_undegraded":"2024-03-04T12:23:17.979207+0000","last_fullsized":"2024-03-04T12:23:17.979207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:05:51.162179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929672+0000","last_change":"2024-03-04T12:23:11.897870+0000","last_active":"2024-03-04T12:23:17.929672+0000","last_peered":"2024-03-04T12:23:17.929672+0000","last_clean":"2024-03-04T12:23:17.929672+0000","last_became_active":"2024-03-04T12:23:11.897443+0000","last_became_peered":"2024-03-04T12:23:11.897443+0000","last_unstale":"2024-03-04T12:23:17.929672+0000","last_undegraded":"2024-03-04T12:23:17.929672+0000","last_fullsized":"2024-03-04T12:23:17.929672+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:49:37.875541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816672+0000","last_change":"2024-03-04T12:23:12.916441+0000","last_active":"2024-03-04T12:23:18.816672+0000","last_peered":"2024-03-04T12:23:18.816672+0000","last_clean":"2024-03-04T12:23:18.816672+0000","last_became_active":"2024-03-04T12:23:12.916078+0000","last_became_peered":"2024-03-04T12:23:12.916078+0000","last_unstale":"2024-03-04T12:23:18.816672+0000","last_undegraded":"2024-03-04T12:23:18.816672+0000","last_fullsized":"2024-03-04T12:23:18.816672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:45:23.663627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.979389+0000","last_change":"2024-03-04T12:23:16.953341+0000","last_active":"2024-03-04T12:23:17.979389+0000","last_peered":"2024-03-04T12:23:17.979389+0000","last_clean":"2024-03-04T12:23:17.979389+0000","last_became_active":"2024-03-04T12:23:16.952878+0000","last_became_peered":"2024-03-04T12:23:16.952878+0000","last_unstale":"2024-03-04T12:23:17.979389+0000","last_undegraded":"2024-03-04T12:23:17.979389+0000","last_fullsized":"2024-03-04T12:23:17.979389+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:08:55.575263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:38.006943+0000","last_change":"2024-03-04T12:23:11.903182+0000","last_active":"2024-03-04T12:23:38.006943+0000","last_peered":"2024-03-04T12:23:38.006943+0000","last_clean":"2024-03-04T12:23:38.006943+0000","last_became_active":"2024-03-04T12:23:11.902709+0000","last_became_peered":"2024-03-04T12:23:11.902709+0000","last_unstale":"2024-03-04T12:23:38.006943+0000","last_undegraded":"2024-03-04T12:23:38.006943+0000","last_fullsized":"2024-03-04T12:23:38.006943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:13:33.438929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930288+0000","last_change":"2024-03-04T12:23:12.931339+0000","last_active":"2024-03-04T12:23:17.930288+0000","last_peered":"2024-03-04T12:23:17.930288+0000","last_clean":"2024-03-04T12:23:17.930288+0000","last_became_active":"2024-03-04T12:23:12.930213+0000","last_became_peered":"2024-03-04T12:23:12.930213+0000","last_unstale":"2024-03-04T12:23:17.930288+0000","last_undegraded":"2024-03-04T12:23:17.930288+0000","last_fullsized":"2024-03-04T12:23:17.930288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:55:23.084139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079581+0000","last_change":"2024-03-04T12:23:16.948723+0000","last_active":"2024-03-04T12:23:19.079581+0000","last_peered":"2024-03-04T12:23:19.079581+0000","last_clean":"2024-03-04T12:23:19.079581+0000","last_became_active":"2024-03-04T12:23:16.948256+0000","last_became_peered":"2024-03-04T12:23:16.948256+0000","last_unstale":"2024-03-04T12:23:19.079581+0000","last_undegraded":"2024-03-04T12:23:19.079581+0000","last_fullsized":"2024-03-04T12:23:19.079581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:22.060517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930042+0000","last_change":"2024-03-04T12:23:11.898475+0000","last_active":"2024-03-04T12:23:17.930042+0000","last_peered":"2024-03-04T12:23:17.930042+0000","last_clean":"2024-03-04T12:23:17.930042+0000","last_became_active":"2024-03-04T12:23:11.898117+0000","last_became_peered":"2024-03-04T12:23:11.898117+0000","last_unstale":"2024-03-04T12:23:17.930042+0000","last_undegraded":"2024-03-04T12:23:17.930042+0000","last_fullsized":"2024-03-04T12:23:17.930042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:21:25.119591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930746+0000","last_change":"2024-03-04T12:23:12.929772+0000","last_active":"2024-03-04T12:23:17.930746+0000","last_peered":"2024-03-04T12:23:17.930746+0000","last_clean":"2024-03-04T12:23:17.930746+0000","last_became_active":"2024-03-04T12:23:12.929405+0000","last_became_peered":"2024-03-04T12:23:12.929405+0000","last_unstale":"2024-03-04T12:23:17.930746+0000","last_undegraded":"2024-03-04T12:23:17.930746+0000","last_fullsized":"2024-03-04T12:23:17.930746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:22:42.874989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.303039+0000","last_change":"2024-03-04T12:23:16.941715+0000","last_active":"2024-03-04T12:23:19.303039+0000","last_peered":"2024-03-04T12:23:19.303039+0000","last_clean":"2024-03-04T12:23:19.303039+0000","last_became_active":"2024-03-04T12:23:16.941298+0000","last_became_peered":"2024-03-04T12:23:16.941298+0000","last_unstale":"2024-03-04T12:23:19.303039+0000","last_undegraded":"2024-03-04T12:23:19.303039+0000","last_fullsized":"2024-03-04T12:23:19.303039+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:09:02.019531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080638+0000","last_change":"2024-03-04T12:23:11.896413+0000","last_active":"2024-03-04T12:23:19.080638+0000","last_peered":"2024-03-04T12:23:19.080638+0000","last_clean":"2024-03-04T12:23:19.080638+0000","last_became_active":"2024-03-04T12:23:11.895679+0000","last_became_peered":"2024-03-04T12:23:11.895679+0000","last_unstale":"2024-03-04T12:23:19.080638+0000","last_undegraded":"2024-03-04T12:23:19.080638+0000","last_fullsized":"2024-03-04T12:23:19.080638+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:25:41.967530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930490+0000","last_change":"2024-03-04T12:23:12.920468+0000","last_active":"2024-03-04T12:23:17.930490+0000","last_peered":"2024-03-04T12:23:17.930490+0000","last_clean":"2024-03-04T12:23:17.930490+0000","last_became_active":"2024-03-04T12:23:12.920113+0000","last_became_peered":"2024-03-04T12:23:12.920113+0000","last_unstale":"2024-03-04T12:23:17.930490+0000","last_undegraded":"2024-03-04T12:23:17.930490+0000","last_fullsized":"2024-03-04T12:23:17.930490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:00:02.283518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079761+0000","last_change":"2024-03-04T12:23:16.944840+0000","last_active":"2024-03-04T12:23:19.079761+0000","last_peered":"2024-03-04T12:23:19.079761+0000","last_clean":"2024-03-04T12:23:19.079761+0000","last_became_active":"2024-03-04T12:23:16.944333+0000","last_became_peered":"2024-03-04T12:23:16.944333+0000","last_unstale":"2024-03-04T12:23:19.079761+0000","last_undegraded":"2024-03-04T12:23:19.079761+0000","last_fullsized":"2024-03-04T12:23:19.079761+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:06:05.535789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079858+0000","last_change":"2024-03-04T12:23:11.903323+0000","last_active":"2024-03-04T12:23:19.079858+0000","last_peered":"2024-03-04T12:23:19.079858+0000","last_clean":"2024-03-04T12:23:19.079858+0000","last_became_active":"2024-03-04T12:23:11.902987+0000","last_became_peered":"2024-03-04T12:23:11.902987+0000","last_unstale":"2024-03-04T12:23:19.079858+0000","last_undegraded":"2024-03-04T12:23:19.079858+0000","last_fullsized":"2024-03-04T12:23:19.079858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:14.044391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930749+0000","last_change":"2024-03-04T12:23:12.913491+0000","last_active":"2024-03-04T12:23:17.930749+0000","last_peered":"2024-03-04T12:23:17.930749+0000","last_clean":"2024-03-04T12:23:17.930749+0000","last_became_active":"2024-03-04T12:23:12.912103+0000","last_became_peered":"2024-03-04T12:23:12.912103+0000","last_unstale":"2024-03-04T12:23:17.930749+0000","last_undegraded":"2024-03-04T12:23:17.930749+0000","last_fullsized":"2024-03-04T12:23:17.930749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:05:38.033596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678282+0000","last_change":"2024-03-04T12:23:12.908437+0000","last_active":"2024-03-04T12:23:18.678282+0000","last_peered":"2024-03-04T12:23:18.678282+0000","last_clean":"2024-03-04T12:23:18.678282+0000","last_became_active":"2024-03-04T12:23:12.908071+0000","last_became_peered":"2024-03-04T12:23:12.908071+0000","last_unstale":"2024-03-04T12:23:18.678282+0000","last_undegraded":"2024-03-04T12:23:18.678282+0000","last_fullsized":"2024-03-04T12:23:18.678282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:20.112645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929913+0000","last_change":"2024-03-04T12:23:12.921645+0000","last_active":"2024-03-04T12:23:17.929913+0000","last_peered":"2024-03-04T12:23:17.929913+0000","last_clean":"2024-03-04T12:23:17.929913+0000","last_became_active":"2024-03-04T12:23:12.921275+0000","last_became_peered":"2024-03-04T12:23:12.921275+0000","last_unstale":"2024-03-04T12:23:17.929913+0000","last_undegraded":"2024-03-04T12:23:17.929913+0000","last_fullsized":"2024-03-04T12:23:17.929913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:56:17.740782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.929592+0000","last_change":"2024-03-04T12:23:12.919813+0000","last_active":"2024-03-04T12:23:17.929592+0000","last_peered":"2024-03-04T12:23:17.929592+0000","last_clean":"2024-03-04T12:23:17.929592+0000","last_became_active":"2024-03-04T12:23:12.919469+0000","last_became_peered":"2024-03-04T12:23:12.919469+0000","last_unstale":"2024-03-04T12:23:17.929592+0000","last_undegraded":"2024-03-04T12:23:17.929592+0000","last_fullsized":"2024-03-04T12:23:17.929592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:46.720268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.302719+0000","last_change":"2024-03-04T12:23:12.918476+0000","last_active":"2024-03-04T12:23:19.302719+0000","last_peered":"2024-03-04T12:23:19.302719+0000","last_clean":"2024-03-04T12:23:19.302719+0000","last_became_active":"2024-03-04T12:23:12.917743+0000","last_became_peered":"2024-03-04T12:23:12.917743+0000","last_unstale":"2024-03-04T12:23:19.302719+0000","last_undegraded":"2024-03-04T12:23:19.302719+0000","last_fullsized":"2024-03-04T12:23:19.302719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:20.246769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079778+0000","last_change":"2024-03-04T12:23:12.918254+0000","last_active":"2024-03-04T12:23:19.079778+0000","last_peered":"2024-03-04T12:23:19.079778+0000","last_clean":"2024-03-04T12:23:19.079778+0000","last_became_active":"2024-03-04T12:23:12.917922+0000","last_became_peered":"2024-03-04T12:23:12.917922+0000","last_unstale":"2024-03-04T12:23:19.079778+0000","last_undegraded":"2024-03-04T12:23:19.079778+0000","last_fullsized":"2024-03-04T12:23:19.079778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:48:20.847984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678592+0000","last_change":"2024-03-04T12:23:12.910368+0000","last_active":"2024-03-04T12:23:18.678592+0000","last_peered":"2024-03-04T12:23:18.678592+0000","last_clean":"2024-03-04T12:23:18.678592+0000","last_became_active":"2024-03-04T12:23:12.910017+0000","last_became_peered":"2024-03-04T12:23:12.910017+0000","last_unstale":"2024-03-04T12:23:18.678592+0000","last_undegraded":"2024-03-04T12:23:18.678592+0000","last_fullsized":"2024-03-04T12:23:18.678592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:15:22.093644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:17.930173+0000","last_change":"2024-03-04T12:23:12.921021+0000","last_active":"2024-03-04T12:23:17.930173+0000","last_peered":"2024-03-04T12:23:17.930173+0000","last_clean":"2024-03-04T12:23:17.930173+0000","last_became_active":"2024-03-04T12:23:12.920600+0000","last_became_peered":"2024-03-04T12:23:12.920600+0000","last_unstale":"2024-03-04T12:23:17.930173+0000","last_undegraded":"2024-03-04T12:23:17.930173+0000","last_fullsized":"2024-03-04T12:23:17.930173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:00:31.706900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816860+0000","last_change":"2024-03-04T12:23:12.915236+0000","last_active":"2024-03-04T12:23:18.816860+0000","last_peered":"2024-03-04T12:23:18.816860+0000","last_clean":"2024-03-04T12:23:18.816860+0000","last_became_active":"2024-03-04T12:23:12.912133+0000","last_became_peered":"2024-03-04T12:23:12.912133+0000","last_unstale":"2024-03-04T12:23:18.816860+0000","last_undegraded":"2024-03-04T12:23:18.816860+0000","last_fullsized":"2024-03-04T12:23:18.816860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:00:10.765556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080486+0000","last_change":"2024-03-04T12:23:12.911394+0000","last_active":"2024-03-04T12:23:19.080486+0000","last_peered":"2024-03-04T12:23:19.080486+0000","last_clean":"2024-03-04T12:23:19.080486+0000","last_became_active":"2024-03-04T12:23:12.911061+0000","last_became_peered":"2024-03-04T12:23:12.911061+0000","last_unstale":"2024-03-04T12:23:19.080486+0000","last_undegraded":"2024-03-04T12:23:19.080486+0000","last_fullsized":"2024-03-04T12:23:19.080486+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:39.880383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.816399+0000","last_change":"2024-03-04T12:23:12.911356+0000","last_active":"2024-03-04T12:23:18.816399+0000","last_peered":"2024-03-04T12:23:18.816399+0000","last_clean":"2024-03-04T12:23:18.816399+0000","last_became_active":"2024-03-04T12:23:12.911029+0000","last_became_peered":"2024-03-04T12:23:12.911029+0000","last_unstale":"2024-03-04T12:23:18.816399+0000","last_undegraded":"2024-03-04T12:23:18.816399+0000","last_fullsized":"2024-03-04T12:23:18.816399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:52.611625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.080428+0000","last_change":"2024-03-04T12:23:12.924921+0000","last_active":"2024-03-04T12:23:19.080428+0000","last_peered":"2024-03-04T12:23:19.080428+0000","last_clean":"2024-03-04T12:23:19.080428+0000","last_became_active":"2024-03-04T12:23:12.924558+0000","last_became_peered":"2024-03-04T12:23:12.924558+0000","last_unstale":"2024-03-04T12:23:19.080428+0000","last_undegraded":"2024-03-04T12:23:19.080428+0000","last_fullsized":"2024-03-04T12:23:19.080428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:10:53.238440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:19.079682+0000","last_change":"2024-03-04T12:23:12.913106+0000","last_active":"2024-03-04T12:23:19.079682+0000","last_peered":"2024-03-04T12:23:19.079682+0000","last_clean":"2024-03-04T12:23:19.079682+0000","last_became_active":"2024-03-04T12:23:12.912741+0000","last_became_peered":"2024-03-04T12:23:12.912741+0000","last_unstale":"2024-03-04T12:23:19.079682+0000","last_undegraded":"2024-03-04T12:23:19.079682+0000","last_fullsized":"2024-03-04T12:23:19.079682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:46:21.406383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-04T12:23:18.678333+0000","last_change":"2024-03-04T12:23:12.918082+0000","last_active":"2024-03-04T12:23:18.678333+0000","last_peered":"2024-03-04T12:23:18.678333+0000","last_clean":"2024-03-04T12:23:18.678333+0000","last_became_active":"2024-03-04T12:23:12.917754+0000","last_became_peered":"2024-03-04T12:23:12.917754+0000","last_unstale":"2024-03-04T12:23:18.678333+0000","last_undegraded":"2024-03-04T12:23:18.678333+0000","last_fullsized":"2024-03-04T12:23:18.678333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:56.601779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286257,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4774,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":87,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":16972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383699,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1071619,"data_compressed":51081,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":478123,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070499,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":477841,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":51081,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":478693,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":481091,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":477319,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14685,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":51081,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T12:23:43.047 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-04T12:23:43.047 INFO:tasks.cephadm:Setting up client nodes... 2024-03-04T12:23:43.048 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T12:23:43.327 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:43.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:42 smithi086 ceph-mon[36851]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:43.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:42 smithi086 ceph-mon[36851]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:43.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:42 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] up:active 2024-03-04T12:23:43.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:42 smithi086 ceph-mon[36851]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-04T12:23:43.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:42 smithi086 ceph-mon[36851]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:23:43.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:42 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/797153292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:43.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:42 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1222187001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:43.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:42 smithi063 ceph-mon[36717]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:43.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:42 smithi063 ceph-mon[36717]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:23:43.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:42 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] up:active 2024-03-04T12:23:43.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:42 smithi063 ceph-mon[36717]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-04T12:23:43.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:42 smithi063 ceph-mon[36717]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:23:43.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:42 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/797153292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:23:43.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:42 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1222187001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:43 smithi060 ceph-mon[32745]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 938 B/s wr, 3 op/s 2024-03-04T12:23:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:43 smithi060 ceph-mon[32745]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:23:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:43 smithi060 ceph-mon[32745]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:23:44.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:43 smithi086 ceph-mon[36851]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 938 B/s wr, 3 op/s 2024-03-04T12:23:44.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:43 smithi086 ceph-mon[36851]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:23:44.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:43 smithi086 ceph-mon[36851]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:23:44.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:43 smithi063 ceph-mon[36717]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 938 B/s wr, 3 op/s 2024-03-04T12:23:44.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:43 smithi063 ceph-mon[36717]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:23:44.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:43 smithi063 ceph-mon[36717]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:23:45.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.167+0000 7fcc055b3700 1 -- 172.21.15.60:0/3934488905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc001019d0 msgr2=0x7fcc00109270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:45.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.167+0000 7fcc055b3700 1 --2- 172.21.15.60:0/3934488905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc001019d0 0x7fcc00109270 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800b0a0 tx=0x7fcbf800b3b0 comp rx=0 tx=0).stop 2024-03-04T12:23:45.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.167+0000 7fcc055b3700 1 -- 172.21.15.60:0/3934488905 shutdown_connections 2024-03-04T12:23:45.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.167+0000 7fcc055b3700 1 --2- 172.21.15.60:0/3934488905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc001019d0 0x7fcc00109270 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:45.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.167+0000 7fcc055b3700 1 --2- 172.21.15.60:0/3934488905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc001023b0 0x7fcc001027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:45.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.167+0000 7fcc055b3700 1 --2- 172.21.15.60:0/3934488905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc00101010 0x7fcc00101490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:45.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.167+0000 7fcc055b3700 1 -- 172.21.15.60:0/3934488905 >> 172.21.15.60:0/3934488905 conn(0x7fcc000fbac0 msgr2=0x7fcc000fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:45.175 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.167+0000 7fcc055b3700 1 -- 172.21.15.60:0/3934488905 shutdown_connections 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.167+0000 7fcc055b3700 1 -- 172.21.15.60:0/3934488905 wait complete. 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.168+0000 7fcc055b3700 1 Processor -- start 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.168+0000 7fcc055b3700 1 -- start start 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.168+0000 7fcc055b3700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc00101010 0x7fcc00197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.168+0000 7fcc055b3700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc001019d0 0x7fcc00198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.168+0000 7fcc055b3700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc001023b0 0x7fcc0019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.168+0000 7fcc055b3700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0010c950 con 0x7fcc001023b0 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.168+0000 7fcc055b3700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0010c7d0 con 0x7fcc001019d0 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.168+0000 7fcc055b3700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0010cad0 con 0x7fcc00101010 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.168+0000 7fcbff7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc001023b0 0x7fcc0019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.168+0000 7fcbff7fe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc001023b0 0x7fcc0019f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:38074/0 (socket says 172.21.15.60:38074) 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.168+0000 7fcbff7fe700 1 -- 172.21.15.60:0/957930668 learned_addr learned my addr 172.21.15.60:0/957930668 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.168+0000 7fcbfeffd700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc00101010 0x7fcc00197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:45.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.169+0000 7fcbfe7fc700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc001019d0 0x7fcc00198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:45.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.169+0000 7fcbff7fe700 1 -- 172.21.15.60:0/957930668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc00101010 msgr2=0x7fcc00197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:45.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.169+0000 7fcbff7fe700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc00101010 0x7fcc00197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:45.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.169+0000 7fcbff7fe700 1 -- 172.21.15.60:0/957930668 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc001019d0 msgr2=0x7fcc00198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:45.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.169+0000 7fcbff7fe700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc001019d0 0x7fcc00198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:45.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.169+0000 7fcbff7fe700 1 -- 172.21.15.60:0/957930668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0019fb50 con 0x7fcc001023b0 2024-03-04T12:23:45.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.169+0000 7fcbff7fe700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc001023b0 0x7fcc0019f4b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8004d00 tx=0x7fcbf8004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:45.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.169+0000 7fcbe7fff700 1 -- 172.21.15.60:0/957930668 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8021070 con 0x7fcc001023b0 2024-03-04T12:23:45.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.169+0000 7fcbe7fff700 1 -- 172.21.15.60:0/957930668 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbf8007b80 con 0x7fcc001023b0 2024-03-04T12:23:45.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.169+0000 7fcc055b3700 1 -- 172.21.15.60:0/957930668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0019fde0 con 0x7fcc001023b0 2024-03-04T12:23:45.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.169+0000 7fcbe7fff700 1 -- 172.21.15.60:0/957930668 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf80052a0 con 0x7fcc001023b0 2024-03-04T12:23:45.177 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.169+0000 7fcc055b3700 1 -- 172.21.15.60:0/957930668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc001a01e0 con 0x7fcc001023b0 2024-03-04T12:23:45.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.170+0000 7fcc055b3700 1 -- 172.21.15.60:0/957930668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc00050360 con 0x7fcc001023b0 2024-03-04T12:23:45.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.171+0000 7fcbe7fff700 1 -- 172.21.15.60:0/957930668 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fcbf8024070 con 0x7fcc001023b0 2024-03-04T12:23:45.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.172+0000 7fcbe7fff700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcbe80a1f50 0x7fcbe80a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:45.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.172+0000 7fcbfeffd700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcbe80a1f50 0x7fcbe80a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:45.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.172+0000 7fcbe7fff700 1 -- 172.21.15.60:0/957930668 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcbf80cf210 con 0x7fcc001023b0 2024-03-04T12:23:45.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.172+0000 7fcbfeffd700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcbe80a1f50 0x7fcbe80a4410 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcc0006a380 tx=0x7fcbf0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:45.182 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.176+0000 7fcbe7fff700 1 -- 172.21.15.60:0/957930668 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcc00050360 con 0x7fcc001023b0 2024-03-04T12:23:45.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.459+0000 7fcc055b3700 1 -- 172.21.15.60:0/957930668 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fcc001a06d0 con 0x7fcc001023b0 2024-03-04T12:23:45.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.461+0000 7fcbe7fff700 1 -- 172.21.15.60:0/957930668 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7fcbf809c330 con 0x7fcc001023b0 2024-03-04T12:23:45.467 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2024-03-04T12:23:45.467 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBRveVl8RZyGxAA43/aML0FgX35yiNHZEMuMw== 2024-03-04T12:23:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.463+0000 7fcc055b3700 1 -- 172.21.15.60:0/957930668 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcbe80a1f50 msgr2=0x7fcbe80a4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.463+0000 7fcc055b3700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcbe80a1f50 0x7fcbe80a4410 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcc0006a380 tx=0x7fcbf0009260 comp rx=0 tx=0).stop 2024-03-04T12:23:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.463+0000 7fcc055b3700 1 -- 172.21.15.60:0/957930668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc001023b0 msgr2=0x7fcc0019f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:45.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.463+0000 7fcc055b3700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc001023b0 0x7fcc0019f4b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8004d00 tx=0x7fcbf8004d30 comp rx=0 tx=0).stop 2024-03-04T12:23:45.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.463+0000 7fcc055b3700 1 -- 172.21.15.60:0/957930668 shutdown_connections 2024-03-04T12:23:45.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.463+0000 7fcc055b3700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcbe80a1f50 0x7fcbe80a4410 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:45.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.463+0000 7fcc055b3700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcc001023b0 0x7fcc0019f4b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:45.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.463+0000 7fcc055b3700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc00101010 0x7fcc00197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:45.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.463+0000 7fcc055b3700 1 --2- 172.21.15.60:0/957930668 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc001019d0 0x7fcc00198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:45.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.463+0000 7fcc055b3700 1 -- 172.21.15.60:0/957930668 >> 172.21.15.60:0/957930668 conn(0x7fcc000fbac0 msgr2=0x7fcc000fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:45.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.464+0000 7fcc055b3700 1 -- 172.21.15.60:0/957930668 shutdown_connections 2024-03-04T12:23:45.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:45.464+0000 7fcc055b3700 1 -- 172.21.15.60:0/957930668 wait complete. 2024-03-04T12:23:46.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:45 smithi060 ceph-mon[32745]: pgmap v309: 129 pgs: 129 active+clean; 628 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.1 KiB/s wr, 17 op/s 2024-03-04T12:23:46.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:45 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/957930668' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T12:23:46.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:45 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/957930668' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T12:23:46.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:45 smithi086 ceph-mon[36851]: pgmap v309: 129 pgs: 129 active+clean; 628 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.1 KiB/s wr, 17 op/s 2024-03-04T12:23:46.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:45 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/957930668' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T12:23:46.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:45 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/957930668' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T12:23:46.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:45 smithi063 ceph-mon[36717]: pgmap v309: 129 pgs: 129 active+clean; 628 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.1 KiB/s wr, 17 op/s 2024-03-04T12:23:46.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:45 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/957930668' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T12:23:46.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:45 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/957930668' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T12:23:46.418 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:23:46.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-04T12:23:46.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-04T12:23:46.465 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-04T12:23:46.465 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-04T12:23:46.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph mgr dump --format=json 2024-03-04T12:23:46.752 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:48.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:47 smithi060 ceph-mon[32745]: pgmap v310: 129 pgs: 129 active+clean; 628 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 8.3 KiB/s wr, 20 op/s 2024-03-04T12:23:48.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:47 smithi086 ceph-mon[36851]: pgmap v310: 129 pgs: 129 active+clean; 628 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 8.3 KiB/s wr, 20 op/s 2024-03-04T12:23:48.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:47 smithi063 ceph-mon[36717]: pgmap v310: 129 pgs: 129 active+clean; 628 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 8.3 KiB/s wr, 20 op/s 2024-03-04T12:23:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.595+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3127625143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e58106d50 msgr2=0x7f0e58107130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.595+0000 7f0e5d0a5700 1 --2- 172.21.15.60:0/3127625143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e58106d50 0x7f0e58107130 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0e48009990 tx=0x7f0e48009ca0 comp rx=0 tx=0).stop 2024-03-04T12:23:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.595+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3127625143 shutdown_connections 2024-03-04T12:23:48.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.595+0000 7f0e5d0a5700 1 --2- 172.21.15.60:0/3127625143 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e581016d0 0x7f0e5810db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.595+0000 7f0e5d0a5700 1 --2- 172.21.15.60:0/3127625143 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e58106d50 0x7f0e58107130 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.595+0000 7f0e5d0a5700 1 --2- 172.21.15.60:0/3127625143 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e58100d10 0x7f0e58101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.595+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3127625143 >> 172.21.15.60:0/3127625143 conn(0x7f0e58077cc0 msgr2=0x7f0e580780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.595+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3127625143 shutdown_connections 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.595+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3127625143 wait complete. 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.596+0000 7f0e5d0a5700 1 Processor -- start 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.596+0000 7f0e5d0a5700 1 -- start start 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.596+0000 7f0e5d0a5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e58100d10 0x7f0e581984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.596+0000 7f0e5d0a5700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e581016d0 0x7f0e58198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.596+0000 7f0e5d0a5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e58106d50 0x7f0e5819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.596+0000 7f0e5d0a5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e58111020 con 0x7f0e58100d10 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.596+0000 7f0e5d0a5700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e58110ea0 con 0x7f0e581016d0 2024-03-04T12:23:48.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e5d0a5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e581111a0 con 0x7f0e58106d50 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e56d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e58100d10 0x7f0e581984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e56d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e58100d10 0x7f0e581984e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51928/0 (socket says 172.21.15.60:51928) 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e56d9d700 1 -- 172.21.15.60:0/3468155823 learned_addr learned my addr 172.21.15.60:0/3468155823 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e5659c700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e581016d0 0x7f0e58198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e5759e700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e58106d50 0x7f0e5819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e56d9d700 1 -- 172.21.15.60:0/3468155823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e58106d50 msgr2=0x7f0e5819cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e56d9d700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e58106d50 0x7f0e5819cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e56d9d700 1 -- 172.21.15.60:0/3468155823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e581016d0 msgr2=0x7f0e58198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e56d9d700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e581016d0 0x7f0e58198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e56d9d700 1 -- 172.21.15.60:0/3468155823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5819d4f0 con 0x7f0e58100d10 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e56d9d700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e58100d10 0x7f0e581984e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0e48005910 tx=0x7f0e480047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e4ffff700 1 -- 172.21.15.60:0/3468155823 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48027070 con 0x7f0e58100d10 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e4ffff700 1 -- 172.21.15.60:0/3468155823 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e48004050 con 0x7f0e58100d10 2024-03-04T12:23:48.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3468155823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5819d780 con 0x7f0e58100d10 2024-03-04T12:23:48.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e4ffff700 1 -- 172.21.15.60:0/3468155823 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4800f460 con 0x7f0e58100d10 2024-03-04T12:23:48.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.597+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3468155823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e581a5060 con 0x7f0e58100d10 2024-03-04T12:23:48.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.598+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3468155823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e58102400 con 0x7f0e58100d10 2024-03-04T12:23:48.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.599+0000 7f0e4ffff700 1 -- 172.21.15.60:0/3468155823 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f0e4802a070 con 0x7f0e58100d10 2024-03-04T12:23:48.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.600+0000 7f0e4ffff700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0e440a1ea0 0x7f0e440a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:48.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.600+0000 7f0e5659c700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0e440a1ea0 0x7f0e440a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:48.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.600+0000 7f0e4ffff700 1 -- 172.21.15.60:0/3468155823 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0e480cf090 con 0x7f0e58100d10 2024-03-04T12:23:48.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.600+0000 7f0e5659c700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0e440a1ea0 0x7f0e440a4360 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0e58078230 tx=0x7f0e4000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:48.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.604+0000 7f0e4ffff700 1 -- 172.21.15.60:0/3468155823 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0e4805dd20 con 0x7f0e58100d10 2024-03-04T12:23:48.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.866+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3468155823 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0e581a5230 con 0x7f0e58100d10 2024-03-04T12:23:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.871+0000 7f0e4ffff700 1 -- 172.21.15.60:0/3468155823 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+261465 (secure 0 0 0) 0x7f0e48013070 con 0x7f0e58100d10 2024-03-04T12:23:48.884 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:48.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.873+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3468155823 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0e440a1ea0 msgr2=0x7f0e440a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:48.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.873+0000 7f0e5d0a5700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0e440a1ea0 0x7f0e440a4360 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0e58078230 tx=0x7f0e4000a300 comp rx=0 tx=0).stop 2024-03-04T12:23:48.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.873+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3468155823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e58100d10 msgr2=0x7f0e581984e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:48.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.873+0000 7f0e5d0a5700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e58100d10 0x7f0e581984e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0e48005910 tx=0x7f0e480047b0 comp rx=0 tx=0).stop 2024-03-04T12:23:48.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.874+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3468155823 shutdown_connections 2024-03-04T12:23:48.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.874+0000 7f0e5d0a5700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e58106d50 0x7f0e5819cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:48.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.874+0000 7f0e5d0a5700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0e440a1ea0 0x7f0e440a4360 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:48.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.874+0000 7f0e5d0a5700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e58100d10 0x7f0e581984e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:48.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.874+0000 7f0e5d0a5700 1 --2- 172.21.15.60:0/3468155823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e581016d0 0x7f0e58198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:48.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.874+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3468155823 >> 172.21.15.60:0/3468155823 conn(0x7f0e58077cc0 msgr2=0x7f0e580fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:48.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.874+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3468155823 shutdown_connections 2024-03-04T12:23:48.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:48.874+0000 7f0e5d0a5700 1 -- 172.21.15.60:0/3468155823 wait complete. 2024-03-04T12:23:49.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:48 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3468155823' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T12:23:49.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:48 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3468155823' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T12:23:49.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:48 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3468155823' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T12:23:49.658 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":1914863179},{"type":"v1","addr":"172.21.15.60:6801","nonce":1914863179}]},"active_addr":"172.21.15.60:6801/1914863179","active_change":"2024-03-04T12:16:04.073770+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.60:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":1342937290}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":3302898178}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":366921369}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":2604977412}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":741562320}]}]} 2024-03-04T12:23:49.663 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-04T12:23:49.663 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-04T12:23:49.663 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T12:23:49.909 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:50.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:49 smithi060 ceph-mon[32745]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.3 KiB/s wr, 23 op/s 2024-03-04T12:23:50.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:49 smithi086 ceph-mon[36851]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.3 KiB/s wr, 23 op/s 2024-03-04T12:23:50.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:49 smithi063 ceph-mon[36717]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 9.3 KiB/s wr, 23 op/s 2024-03-04T12:23:51.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.741+0000 7f93d8987700 1 -- 172.21.15.60:0/1874078251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d4107e70 msgr2=0x7f93d4108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:51.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.741+0000 7f93d8987700 1 --2- 172.21.15.60:0/1874078251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d4107e70 0x7f93d4108250 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f93c40099c0 tx=0x7f93c4009cd0 comp rx=0 tx=0).stop 2024-03-04T12:23:51.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.742+0000 7f93d8987700 1 -- 172.21.15.60:0/1874078251 shutdown_connections 2024-03-04T12:23:51.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.742+0000 7f93d8987700 1 --2- 172.21.15.60:0/1874078251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93d406a470 0x7f93d410ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:51.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.742+0000 7f93d8987700 1 --2- 172.21.15.60:0/1874078251 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d4107e70 0x7f93d4108250 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:51.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.742+0000 7f93d8987700 1 --2- 172.21.15.60:0/1874078251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93d4069ad0 0x7f93d4069f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:51.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.742+0000 7f93d8987700 1 -- 172.21.15.60:0/1874078251 >> 172.21.15.60:0/1874078251 conn(0x7f93d4077c50 msgr2=0x7f93d4078060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:51.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.742+0000 7f93d8987700 1 -- 172.21.15.60:0/1874078251 shutdown_connections 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.742+0000 7f93d8987700 1 -- 172.21.15.60:0/1874078251 wait complete. 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.742+0000 7f93d8987700 1 Processor -- start 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.742+0000 7f93d8987700 1 -- start start 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d8987700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d4069ad0 0x7f93d4100df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d8987700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93d406a470 0x7f93d4101330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d8987700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93d4107e70 0x7f93d4101870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d8987700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d4112130 con 0x7f93d4069ad0 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d8987700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d4111fb0 con 0x7f93d4107e70 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d8987700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d41122b0 con 0x7f93d406a470 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d4069ad0 0x7f93d4100df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d259c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d4069ad0 0x7f93d4100df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51936/0 (socket says 172.21.15.60:51936) 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d259c700 1 -- 172.21.15.60:0/759996201 learned_addr learned my addr 172.21.15.60:0/759996201 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d2d9d700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93d4107e70 0x7f93d4101870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:51.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d1d9b700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93d406a470 0x7f93d4101330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:51.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d259c700 1 -- 172.21.15.60:0/759996201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93d406a470 msgr2=0x7f93d4101330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:51.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d259c700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93d406a470 0x7f93d4101330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:51.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d259c700 1 -- 172.21.15.60:0/759996201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93d4107e70 msgr2=0x7f93d4101870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:51.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d259c700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93d4107e70 0x7f93d4101870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:51.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d259c700 1 -- 172.21.15.60:0/759996201 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d41a6b10 con 0x7f93d4069ad0 2024-03-04T12:23:51.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.743+0000 7f93d259c700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d4069ad0 0x7f93d4100df0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f93c4006010 tx=0x7f93c4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:51.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.744+0000 7f93c37fe700 1 -- 172.21.15.60:0/759996201 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c4027070 con 0x7f93d4069ad0 2024-03-04T12:23:51.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.744+0000 7f93c37fe700 1 -- 172.21.15.60:0/759996201 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93c4003780 con 0x7f93d4069ad0 2024-03-04T12:23:51.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.744+0000 7f93d8987700 1 -- 172.21.15.60:0/759996201 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d41a6da0 con 0x7f93d4069ad0 2024-03-04T12:23:51.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.744+0000 7f93c37fe700 1 -- 172.21.15.60:0/759996201 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c4022da0 con 0x7f93d4069ad0 2024-03-04T12:23:51.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.744+0000 7f93d8987700 1 -- 172.21.15.60:0/759996201 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d41a71f0 con 0x7f93d4069ad0 2024-03-04T12:23:51.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.745+0000 7f93d8987700 1 -- 172.21.15.60:0/759996201 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d4068ae0 con 0x7f93d4069ad0 2024-03-04T12:23:51.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.746+0000 7f93c37fe700 1 -- 172.21.15.60:0/759996201 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f93c402a070 con 0x7f93d4069ad0 2024-03-04T12:23:51.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.747+0000 7f93c37fe700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f93bc0a1f20 0x7f93bc0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:51.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.747+0000 7f93d1d9b700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f93bc0a1f20 0x7f93bc0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:51.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.747+0000 7f93c37fe700 1 -- 172.21.15.60:0/759996201 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f93c40cedd0 con 0x7f93d4069ad0 2024-03-04T12:23:51.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.748+0000 7f93d1d9b700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f93bc0a1f20 0x7f93bc0a43e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f93c8005950 tx=0x7f93c80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:51.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.751+0000 7f93c37fe700 1 -- 172.21.15.60:0/759996201 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f93c4013070 con 0x7f93d4069ad0 2024-03-04T12:23:51.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.957+0000 7f93d8987700 1 -- 172.21.15.60:0/759996201 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93d402da30 con 0x7f93d4069ad0 2024-03-04T12:23:51.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.958+0000 7f93c37fe700 1 -- 172.21.15.60:0/759996201 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23997 (secure 0 0 0) 0x7f93c409be90 con 0x7f93d4069ad0 2024-03-04T12:23:51.966 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:51.966 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":89,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:23:17.289925+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:23:51.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.960+0000 7f93d8987700 1 -- 172.21.15.60:0/759996201 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f93bc0a1f20 msgr2=0x7f93bc0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:51.967 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.960+0000 7f93d8987700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f93bc0a1f20 0x7f93bc0a43e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f93c8005950 tx=0x7f93c80058e0 comp rx=0 tx=0).stop 2024-03-04T12:23:51.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.960+0000 7f93d8987700 1 -- 172.21.15.60:0/759996201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d4069ad0 msgr2=0x7f93d4100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:51.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.960+0000 7f93d8987700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d4069ad0 0x7f93d4100df0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f93c4006010 tx=0x7f93c4009f80 comp rx=0 tx=0).stop 2024-03-04T12:23:51.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.960+0000 7f93d8987700 1 -- 172.21.15.60:0/759996201 shutdown_connections 2024-03-04T12:23:51.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.960+0000 7f93d8987700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93d4107e70 0x7f93d4101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:51.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.961+0000 7f93d8987700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f93bc0a1f20 0x7f93bc0a43e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:51.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.961+0000 7f93d8987700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f93d4069ad0 0x7f93d4100df0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:51.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.961+0000 7f93d8987700 1 --2- 172.21.15.60:0/759996201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93d406a470 0x7f93d4101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:51.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.961+0000 7f93d8987700 1 -- 172.21.15.60:0/759996201 >> 172.21.15.60:0/759996201 conn(0x7f93d4077c50 msgr2=0x7f93d40ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:51.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.961+0000 7f93d8987700 1 -- 172.21.15.60:0/759996201 shutdown_connections 2024-03-04T12:23:51.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:51.961+0000 7f93d8987700 1 -- 172.21.15.60:0/759996201 wait complete. 2024-03-04T12:23:52.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:51 smithi060 ceph-mon[32745]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 24 op/s 2024-03-04T12:23:52.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:51 smithi086 ceph-mon[36851]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 24 op/s 2024-03-04T12:23:52.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:51 smithi063 ceph-mon[36717]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 24 op/s 2024-03-04T12:23:52.735 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-04T12:23:52.735 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T12:23:52.986 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:53.107 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:52 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/759996201' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:53.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:52 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/759996201' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:52 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/759996201' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:54.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:53 smithi060 ceph-mon[32745]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 KiB/s wr, 23 op/s 2024-03-04T12:23:54.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:53 smithi086 ceph-mon[36851]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 KiB/s wr, 23 op/s 2024-03-04T12:23:54.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:53 smithi063 ceph-mon[36717]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 KiB/s wr, 23 op/s 2024-03-04T12:23:54.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.802+0000 7f600eeb5700 1 -- 172.21.15.60:0/1454667585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6008107140 msgr2=0x7f600810ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:54.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.802+0000 7f600eeb5700 1 --2- 172.21.15.60:0/1454667585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6008107140 0x7f600810ed10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f600000b0d0 tx=0x7f600000b3e0 comp rx=0 tx=0).stop 2024-03-04T12:23:54.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.802+0000 7f600eeb5700 1 -- 172.21.15.60:0/1454667585 shutdown_connections 2024-03-04T12:23:54.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.802+0000 7f600eeb5700 1 --2- 172.21.15.60:0/1454667585 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6008107140 0x7f600810ed10 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:54.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.802+0000 7f600eeb5700 1 --2- 172.21.15.60:0/1454667585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6008102060 0x7f6008102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:54.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.802+0000 7f600eeb5700 1 --2- 172.21.15.60:0/1454667585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6008102a10 0x7f6008106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:54.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.802+0000 7f600eeb5700 1 -- 172.21.15.60:0/1454667585 >> 172.21.15.60:0/1454667585 conn(0x7f60080fd8d0 msgr2=0x7f60080ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:54.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.802+0000 7f600eeb5700 1 -- 172.21.15.60:0/1454667585 shutdown_connections 2024-03-04T12:23:54.810 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.802+0000 7f600eeb5700 1 -- 172.21.15.60:0/1454667585 wait complete. 2024-03-04T12:23:54.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f600eeb5700 1 Processor -- start 2024-03-04T12:23:54.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f600eeb5700 1 -- start start 2024-03-04T12:23:54.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f600eeb5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6008102060 0x7f6008198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:54.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f600eeb5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6008102a10 0x7f6008198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:54.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f600eeb5700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6008107140 0x7f600819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:54.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f600eeb5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60081121a0 con 0x7f6008102a10 2024-03-04T12:23:54.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f600eeb5700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6008112020 con 0x7f6008107140 2024-03-04T12:23:54.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f600eeb5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6008112320 con 0x7f6008102060 2024-03-04T12:23:54.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f6007fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6008102a10 0x7f6008198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:54.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f6007fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6008102a10 0x7f6008198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51964/0 (socket says 172.21.15.60:51964) 2024-03-04T12:23:54.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f6007fff700 1 -- 172.21.15.60:0/2112651286 learned_addr learned my addr 172.21.15.60:0/2112651286 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:23:54.812 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f600d452700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6008107140 0x7f600819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.803+0000 7f600cc51700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6008102060 0x7f6008198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.804+0000 7f6007fff700 1 -- 172.21.15.60:0/2112651286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6008102060 msgr2=0x7f6008198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.804+0000 7f6007fff700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6008102060 0x7f6008198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.804+0000 7f6007fff700 1 -- 172.21.15.60:0/2112651286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6008107140 msgr2=0x7f600819ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.804+0000 7f6007fff700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6008107140 0x7f600819ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.804+0000 7f6007fff700 1 -- 172.21.15.60:0/2112651286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600819d5a0 con 0x7f6008102a10 2024-03-04T12:23:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.804+0000 7f6007fff700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6008102a10 0x7f6008198ad0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00b500 tx=0x7f5ffc00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.804+0000 7f6005ffb700 1 -- 172.21.15.60:0/2112651286 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc015020 con 0x7f6008102a10 2024-03-04T12:23:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.804+0000 7f6005ffb700 1 -- 172.21.15.60:0/2112651286 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ffc011d10 con 0x7f6008102a10 2024-03-04T12:23:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.804+0000 7f600eeb5700 1 -- 172.21.15.60:0/2112651286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600819d890 con 0x7f6008102a10 2024-03-04T12:23:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.804+0000 7f6005ffb700 1 -- 172.21.15.60:0/2112651286 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc010de0 con 0x7f6008102a10 2024-03-04T12:23:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.804+0000 7f600eeb5700 1 -- 172.21.15.60:0/2112651286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60081a5110 con 0x7f6008102a10 2024-03-04T12:23:54.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.805+0000 7f600eeb5700 1 -- 172.21.15.60:0/2112651286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6008068ae0 con 0x7f6008102a10 2024-03-04T12:23:54.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.806+0000 7f6005ffb700 1 -- 172.21.15.60:0/2112651286 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f5ffc018020 con 0x7f6008102a10 2024-03-04T12:23:54.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.807+0000 7f6005ffb700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5ff00a1f20 0x7f5ff00a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:23:54.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.807+0000 7f600cc51700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5ff00a1f20 0x7f5ff00a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:23:54.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.807+0000 7f6005ffb700 1 -- 172.21.15.60:0/2112651286 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f5ffc0cd6d0 con 0x7f6008102a10 2024-03-04T12:23:54.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.807+0000 7f600cc51700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5ff00a1f20 0x7f5ff00a43e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8005950 tx=0x7f5ff800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:23:54.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:54.811+0000 7f6005ffb700 1 -- 172.21.15.60:0/2112651286 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5ffc01f020 con 0x7f6008102a10 2024-03-04T12:23:55.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.015+0000 7f600eeb5700 1 -- 172.21.15.60:0/2112651286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60081998f0 con 0x7f6008102a10 2024-03-04T12:23:55.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.017+0000 7f6005ffb700 1 -- 172.21.15.60:0/2112651286 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23997 (secure 0 0 0) 0x7f5ffc09a870 con 0x7f6008102a10 2024-03-04T12:23:55.024 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:23:55.024 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":89,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:23:17.289925+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:23:55.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.019+0000 7f600eeb5700 1 -- 172.21.15.60:0/2112651286 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5ff00a1f20 msgr2=0x7f5ff00a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:55.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.019+0000 7f600eeb5700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5ff00a1f20 0x7f5ff00a43e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8005950 tx=0x7f5ff800b410 comp rx=0 tx=0).stop 2024-03-04T12:23:55.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.019+0000 7f600eeb5700 1 -- 172.21.15.60:0/2112651286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6008102a10 msgr2=0x7f6008198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:23:55.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.019+0000 7f600eeb5700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6008102a10 0x7f6008198ad0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00b500 tx=0x7f5ffc00b810 comp rx=0 tx=0).stop 2024-03-04T12:23:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.019+0000 7f600eeb5700 1 -- 172.21.15.60:0/2112651286 shutdown_connections 2024-03-04T12:23:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.019+0000 7f600eeb5700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6008107140 0x7f600819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.019+0000 7f600eeb5700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6008102060 0x7f6008198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.019+0000 7f600eeb5700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5ff00a1f20 0x7f5ff00a43e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.019+0000 7f600eeb5700 1 --2- 172.21.15.60:0/2112651286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6008102a10 0x7f6008198ad0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:23:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.019+0000 7f600eeb5700 1 -- 172.21.15.60:0/2112651286 >> 172.21.15.60:0/2112651286 conn(0x7f60080fd8d0 msgr2=0x7f60080fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:23:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.019+0000 7f600eeb5700 1 -- 172.21.15.60:0/2112651286 shutdown_connections 2024-03-04T12:23:55.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:23:55.020+0000 7f600eeb5700 1 -- 172.21.15.60:0/2112651286 wait complete. 2024-03-04T12:23:55.771 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-04T12:23:55.772 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-04T12:23:55.772 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-04T12:23:55.773 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-04T12:23:55.773 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-04T12:23:55.773 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-04T12:23:55.774 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-04T12:23:55.774 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-04T12:23:55.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-04T12:23:55.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-04T12:23:55.775 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-04T12:23:55.776 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-04T12:23:56.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:55 smithi060 ceph-mon[32745]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 25 op/s 2024-03-04T12:23:56.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:55 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2112651286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:56.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:55 smithi086 ceph-mon[36851]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 25 op/s 2024-03-04T12:23:56.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:55 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2112651286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:56.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:55 smithi063 ceph-mon[36717]: pgmap v314: 129 pgs: 129 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.7 KiB/s wr, 25 op/s 2024-03-04T12:23:56.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:55 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2112651286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:23:56.533 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:56.534 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:56.536 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:56.536 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:56.537 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:56.539 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:56.543 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:56.547 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:56.548 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:56.555 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:56.557 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:56.577 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:23:58.320 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:57 smithi060 ceph-mon[32745]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-04T12:23:58.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:57 smithi086 ceph-mon[36851]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-04T12:23:58.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:57 smithi063 ceph-mon[36717]: pgmap v315: 129 pgs: 129 active+clean; 639 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-04T12:24:00.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:23:59 smithi060 ceph-mon[32745]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-04T12:24:00.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:23:59 smithi086 ceph-mon[36851]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-04T12:24:00.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:23:59 smithi063 ceph-mon[36717]: pgmap v316: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-04T12:24:02.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:01 smithi060 ceph-mon[32745]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-04T12:24:02.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:01 smithi086 ceph-mon[36851]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-04T12:24:02.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:01 smithi063 ceph-mon[36717]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-04T12:24:04.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:03 smithi060 ceph-mon[32745]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-04T12:24:04.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:03 smithi086 ceph-mon[36851]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-04T12:24:04.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:03 smithi063 ceph-mon[36717]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-04T12:24:05.319 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-03-04T12:24:05.319 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-04T12:24:05.319 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 7]}]: dispatch 2024-03-04T12:24:05.319 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-03-04T12:24:05.319 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-04T12:24:05.319 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-04T12:24:05.319 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-03-04T12:24:05.319 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-04T12:24:05.319 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-03-04T12:24:05.319 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:04 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-04T12:24:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-03-04T12:24:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-04T12:24:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 7]}]: dispatch 2024-03-04T12:24:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-03-04T12:24:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-04T12:24:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-04T12:24:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-03-04T12:24:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-04T12:24:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-03-04T12:24:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:04 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-04T12:24:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-03-04T12:24:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2024-03-04T12:24:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 7]}]: dispatch 2024-03-04T12:24:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-03-04T12:24:05.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-03-04T12:24:05.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-03-04T12:24:05.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-03-04T12:24:05.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-04T12:24:05.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-03-04T12:24:05.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:04 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-03-04T12:24:06.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.030+0000 7f10a058e700 1 -- 172.21.15.60:0/2080767259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f109806a720 msgr2=0x7f109806ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.030+0000 7f10a058e700 1 --2- 172.21.15.60:0/2080767259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f109806a720 0x7f109806ab00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f10940099c0 tx=0x7f1094009cd0 comp rx=0 tx=0).stop 2024-03-04T12:24:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.030+0000 7f10a058e700 1 -- 172.21.15.60:0/2080767259 shutdown_connections 2024-03-04T12:24:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.030+0000 7f10a058e700 1 --2- 172.21.15.60:0/2080767259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1098107190 0x7f109810ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.030+0000 7f10a058e700 1 --2- 172.21.15.60:0/2080767259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f109806a720 0x7f109806ab00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.030+0000 7f10a058e700 1 --2- 172.21.15.60:0/2080767259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f109806b0d0 0x7f1098106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.030+0000 7f10a058e700 1 -- 172.21.15.60:0/2080767259 >> 172.21.15.60:0/2080767259 conn(0x7f1098077c90 msgr2=0x7f10980780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.031+0000 7f10a058e700 1 -- 172.21.15.60:0/2080767259 shutdown_connections 2024-03-04T12:24:06.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.031+0000 7f10a058e700 1 -- 172.21.15.60:0/2080767259 wait complete. 2024-03-04T12:24:06.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.031+0000 7f10a058e700 1 Processor -- start 2024-03-04T12:24:06.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.031+0000 7f10a058e700 1 -- start start 2024-03-04T12:24:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f10a058e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f109806a720 0x7f1098198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f10a058e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f109806b0d0 0x7f1098198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f10a058e700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1098107190 0x7f109819ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f10a058e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1098112120 con 0x7f109806b0d0 2024-03-04T12:24:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f10a058e700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1098111fa0 con 0x7f1098107190 2024-03-04T12:24:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f10a058e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10981122a0 con 0x7f109806a720 2024-03-04T12:24:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f109db29700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f109806b0d0 0x7f1098198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f109db29700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f109806b0d0 0x7f1098198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53862/0 (socket says 172.21.15.60:53862) 2024-03-04T12:24:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f109db29700 1 -- 172.21.15.60:0/1226420699 learned_addr learned my addr 172.21.15.60:0/1226420699 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f109e32a700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f109806a720 0x7f1098198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f109eb2b700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1098107190 0x7f109819ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:06.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f109db29700 1 -- 172.21.15.60:0/1226420699 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f109806a720 msgr2=0x7f1098198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f109db29700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f109806a720 0x7f1098198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f109db29700 1 -- 172.21.15.60:0/1226420699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1098107190 msgr2=0x7f109819ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f109db29700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1098107190 0x7f109819ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f109db29700 1 -- 172.21.15.60:0/1226420699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109819d540 con 0x7f109806b0d0 2024-03-04T12:24:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f109db29700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f109806b0d0 0x7f1098198a70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f108800c9b0 tx=0x7f108800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.032+0000 7f108f7fe700 1 -- 172.21.15.60:0/1226420699 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1088007aa0 con 0x7f109806b0d0 2024-03-04T12:24:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.033+0000 7f10a058e700 1 -- 172.21.15.60:0/1226420699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f109819d830 con 0x7f109806b0d0 2024-03-04T12:24:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.033+0000 7f108f7fe700 1 -- 172.21.15.60:0/1226420699 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f108800ce50 con 0x7f109806b0d0 2024-03-04T12:24:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.033+0000 7f108f7fe700 1 -- 172.21.15.60:0/1226420699 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1088018660 con 0x7f109806b0d0 2024-03-04T12:24:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.033+0000 7f10a058e700 1 -- 172.21.15.60:0/1226420699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10981a50b0 con 0x7f109806b0d0 2024-03-04T12:24:06.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.033+0000 7f10a058e700 1 -- 172.21.15.60:0/1226420699 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f10980625b0 con 0x7f109806b0d0 2024-03-04T12:24:06.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.035+0000 7f108f7fe700 1 -- 172.21.15.60:0/1226420699 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f108800f450 con 0x7f109806b0d0 2024-03-04T12:24:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.035+0000 7f108f7fe700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f10840a1ea0 0x7f10840a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.035+0000 7f109e32a700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f10840a1ea0 0x7f10840a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.035+0000 7f108f7fe700 1 -- 172.21.15.60:0/1226420699 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f10880ce510 con 0x7f109806b0d0 2024-03-04T12:24:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.035+0000 7f108f7fe700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944] conn(0x7f10840ad000 0x7f10840af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.036+0000 7f108f7fe700 1 -- 172.21.15.60:0/1226420699 --> [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10840afb70 con 0x7f10840ad000 2024-03-04T12:24:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.036+0000 7f108f7fe700 1 -- 172.21.15.60:0/1226420699 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10880ce960 con 0x7f109806b0d0 2024-03-04T12:24:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.036+0000 7f109e32a700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f10840a1ea0 0x7f10840a4360 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1094000c00 tx=0x7f1094005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.036+0000 7f109eb2b700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944] conn(0x7f10840ad000 0x7f10840af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.036+0000 7f109eb2b700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944] conn(0x7f10840ad000 0x7f10840af4a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:06.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.037+0000 7f108f7fe700 1 -- 172.21.15.60:0/1226420699 <== osd.8 v2:172.21.15.86:6816/1013701944 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f10840afb70 con 0x7f10840ad000 2024-03-04T12:24:06.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.053+0000 7f10a058e700 1 -- 172.21.15.60:0/1226420699 --> [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f10980629b0 con 0x7f10840ad000 2024-03-04T12:24:06.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.054+0000 7f108f7fe700 1 -- 172.21.15.60:0/1226420699 <== osd.8 v2:172.21.15.86:6816/1013701944 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f10980629b0 con 0x7f10840ad000 2024-03-04T12:24:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.055+0000 7f10a058e700 1 -- 172.21.15.60:0/1226420699 >> [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944] conn(0x7f10840ad000 msgr2=0x7f10840af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.055+0000 7f10a058e700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944] conn(0x7f10840ad000 0x7f10840af4a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.055+0000 7f10a058e700 1 -- 172.21.15.60:0/1226420699 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f10840a1ea0 msgr2=0x7f10840a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.055+0000 7f10a058e700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f10840a1ea0 0x7f10840a4360 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1094000c00 tx=0x7f1094005fb0 comp rx=0 tx=0).stop 2024-03-04T12:24:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.055+0000 7f10a058e700 1 -- 172.21.15.60:0/1226420699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f109806b0d0 msgr2=0x7f1098198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.055+0000 7f10a058e700 1 --2- 172.21.15.60:0/1226420699 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f109806b0d0 0x7f1098198a70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f108800c9b0 tx=0x7f108800ccc0 comp rx=0 tx=0).stop 2024-03-04T12:24:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.055+0000 7f109eb2b700 1 -- 172.21.15.60:0/1226420699 reap_dead start 2024-03-04T12:24:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.055+0000 7f10a058e700 1 -- 172.21.15.60:0/1226420699 shutdown_connections 2024-03-04T12:24:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.055+0000 7f10a058e700 1 -- 172.21.15.60:0/1226420699 >> 172.21.15.60:0/1226420699 conn(0x7f1098077c90 msgr2=0x7f10980fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.055+0000 7f10a058e700 1 -- 172.21.15.60:0/1226420699 shutdown_connections 2024-03-04T12:24:06.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:06.055+0000 7f10a058e700 1 -- 172.21.15.60:0/1226420699 wait complete. 2024-03-04T12:24:06.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:05 smithi060 ceph-mon[32745]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-04T12:24:06.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-03-04T12:24:06.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-04T12:24:06.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 7]}]': finished 2024-03-04T12:24:06.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-03-04T12:24:06.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-04T12:24:06.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-04T12:24:06.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-03-04T12:24:06.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-04T12:24:06.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-03-04T12:24:06.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-04T12:24:06.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:05 smithi060 ceph-mon[32745]: osdmap e90: 12 total, 12 up, 12 in 2024-03-04T12:24:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:05 smithi086 ceph-mon[36851]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-04T12:24:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-03-04T12:24:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-04T12:24:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 7]}]': finished 2024-03-04T12:24:06.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-03-04T12:24:06.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-04T12:24:06.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-04T12:24:06.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-03-04T12:24:06.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-04T12:24:06.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-03-04T12:24:06.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-04T12:24:06.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:05 smithi086 ceph-mon[36851]: osdmap e90: 12 total, 12 up, 12 in 2024-03-04T12:24:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:05 smithi063 ceph-mon[36717]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-04T12:24:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-03-04T12:24:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2024-03-04T12:24:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.27", "id": [4, 7]}]': finished 2024-03-04T12:24:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-03-04T12:24:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-03-04T12:24:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-03-04T12:24:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-03-04T12:24:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-04T12:24:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-03-04T12:24:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-03-04T12:24:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:05 smithi063 ceph-mon[36717]: osdmap e90: 12 total, 12 up, 12 in 2024-03-04T12:24:07.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.099+0000 7ff1463fc700 1 -- 172.21.15.60:0/3154173757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff14006af30 msgr2=0x7ff14010ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:07.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.099+0000 7ff1463fc700 1 --2- 172.21.15.60:0/3154173757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff14006af30 0x7ff14010ec70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff13800b0d0 tx=0x7ff13800b3e0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.099+0000 7ff1463fc700 1 -- 172.21.15.60:0/3154173757 shutdown_connections 2024-03-04T12:24:07.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.099+0000 7ff1463fc700 1 --2- 172.21.15.60:0/3154173757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff14006af30 0x7ff14010ec70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.099+0000 7ff1463fc700 1 --2- 172.21.15.60:0/3154173757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff140107e50 0x7ff140108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.099+0000 7ff1463fc700 1 --2- 172.21.15.60:0/3154173757 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff14006a570 0x7ff14006a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.099+0000 7ff1463fc700 1 -- 172.21.15.60:0/3154173757 >> 172.21.15.60:0/3154173757 conn(0x7ff140077d30 msgr2=0x7ff140078140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.099+0000 7ff1463fc700 1 -- 172.21.15.60:0/3154173757 shutdown_connections 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.099+0000 7ff1463fc700 1 -- 172.21.15.60:0/3154173757 wait complete. 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.099+0000 7ff1463fc700 1 Processor -- start 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.099+0000 7ff1463fc700 1 -- start start 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff1463fc700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff14006a570 0x7ff140100dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff1463fc700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff14006af30 0x7ff140101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff1463fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff140107e50 0x7ff140101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff1463fc700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1401120f0 con 0x7ff140107e50 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff1463fc700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff140111f70 con 0x7ff14006af30 2024-03-04T12:24:07.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff1463fc700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff140112270 con 0x7ff14006a570 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff144999700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff140107e50 0x7ff140101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff144999700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff140107e50 0x7ff140101850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53894/0 (socket says 172.21.15.60:53894) 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff144999700 1 -- 172.21.15.60:0/1348332381 learned_addr learned my addr 172.21.15.60:0/1348332381 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff13f7fe700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff14006af30 0x7ff140101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff13ffff700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff14006a570 0x7ff140100dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff144999700 1 -- 172.21.15.60:0/1348332381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff14006a570 msgr2=0x7ff140100dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff144999700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff14006a570 0x7ff140100dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff144999700 1 -- 172.21.15.60:0/1348332381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff14006af30 msgr2=0x7ff140101310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff144999700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff14006af30 0x7ff140101310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff144999700 1 -- 172.21.15.60:0/1348332381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1401a6a30 con 0x7ff140107e50 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff13ffff700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff14006a570 0x7ff140100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff144999700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff140107e50 0x7ff140101850 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff138006070 tx=0x7ff138004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff13d7fa700 1 -- 172.21.15.60:0/1348332381 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138021070 con 0x7ff140107e50 2024-03-04T12:24:07.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff13d7fa700 1 -- 172.21.15.60:0/1348332381 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff138007c80 con 0x7ff140107e50 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff13d7fa700 1 -- 172.21.15.60:0/1348332381 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1380053c0 con 0x7ff140107e50 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.100+0000 7ff1463fc700 1 -- 172.21.15.60:0/1348332381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1401a6cc0 con 0x7ff140107e50 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.101+0000 7ff1463fc700 1 -- 172.21.15.60:0/1348332381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1401a7120 con 0x7ff140107e50 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.101+0000 7ff1463fc700 1 -- 172.21.15.60:0/1348332381 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff1400624f0 con 0x7ff140107e50 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.103+0000 7ff13d7fa700 1 -- 172.21.15.60:0/1348332381 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7ff138024070 con 0x7ff140107e50 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.103+0000 7ff13d7fa700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff1280a1ea0 0x7ff1280a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.103+0000 7ff13d7fa700 1 -- 172.21.15.60:0/1348332381 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff13800f020 con 0x7ff140107e50 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.103+0000 7ff13ffff700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff1280a1ea0 0x7ff1280a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.103+0000 7ff13d7fa700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157] conn(0x7ff1280aced0 0x7ff1280af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.103+0000 7ff13d7fa700 1 -- 172.21.15.60:0/1348332381 --> [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1280afa40 con 0x7ff1280aced0 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.103+0000 7ff13d7fa700 1 -- 172.21.15.60:0/1348332381 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1380cfef0 con 0x7ff140107e50 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.104+0000 7ff13ffff700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff1280a1ea0 0x7ff1280a4360 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff130009b50 tx=0x7ff130009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:07.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.104+0000 7ff13f7fe700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157] conn(0x7ff1280aced0 0x7ff1280af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:07.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.104+0000 7ff13f7fe700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157] conn(0x7ff1280aced0 0x7ff1280af370 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:07.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.105+0000 7ff13d7fa700 1 -- 172.21.15.60:0/1348332381 <== osd.5 v2:172.21.15.86:6808/1744734157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff1280afa40 con 0x7ff1280aced0 2024-03-04T12:24:07.129 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.122+0000 7ff1463fc700 1 -- 172.21.15.60:0/1348332381 --> [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff140062970 con 0x7ff1280aced0 2024-03-04T12:24:07.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.123+0000 7ff13d7fa700 1 -- 172.21.15.60:0/1348332381 <== osd.5 v2:172.21.15.86:6808/1744734157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff140062970 con 0x7ff1280aced0 2024-03-04T12:24:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.124+0000 7ff1463fc700 1 -- 172.21.15.60:0/1348332381 >> [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157] conn(0x7ff1280aced0 msgr2=0x7ff1280af370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.124+0000 7ff1463fc700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157] conn(0x7ff1280aced0 0x7ff1280af370 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.124+0000 7ff1463fc700 1 -- 172.21.15.60:0/1348332381 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff1280a1ea0 msgr2=0x7ff1280a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.124+0000 7ff1463fc700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff1280a1ea0 0x7ff1280a4360 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff130009b50 tx=0x7ff130009550 comp rx=0 tx=0).stop 2024-03-04T12:24:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.124+0000 7ff1463fc700 1 -- 172.21.15.60:0/1348332381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff140107e50 msgr2=0x7ff140101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.124+0000 7ff1463fc700 1 --2- 172.21.15.60:0/1348332381 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff140107e50 0x7ff140101850 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff138006070 tx=0x7ff138004db0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.124+0000 7ff144999700 1 -- 172.21.15.60:0/1348332381 reap_dead start 2024-03-04T12:24:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.124+0000 7ff1463fc700 1 -- 172.21.15.60:0/1348332381 shutdown_connections 2024-03-04T12:24:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.124+0000 7ff1463fc700 1 -- 172.21.15.60:0/1348332381 >> 172.21.15.60:0/1348332381 conn(0x7ff140077d30 msgr2=0x7ff1400ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.124+0000 7ff1463fc700 1 -- 172.21.15.60:0/1348332381 shutdown_connections 2024-03-04T12:24:07.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.124+0000 7ff1463fc700 1 -- 172.21.15.60:0/1348332381 wait complete. 2024-03-04T12:24:07.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:06 smithi086 ceph-mon[36851]: osdmap e91: 12 total, 12 up, 12 in 2024-03-04T12:24:07.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:06 smithi063 ceph-mon[36717]: osdmap e91: 12 total, 12 up, 12 in 2024-03-04T12:24:07.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:06 smithi060 ceph-mon[32745]: osdmap e91: 12 total, 12 up, 12 in 2024-03-04T12:24:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.879+0000 7f979ef79700 1 -- 172.21.15.60:0/3107332088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9798101960 msgr2=0x7f9798105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.879+0000 7f979ef79700 1 --2- 172.21.15.60:0/3107332088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9798101960 0x7f9798105920 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f97840099c0 tx=0x7f9784009cd0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.879+0000 7f979ef79700 1 -- 172.21.15.60:0/3107332088 shutdown_connections 2024-03-04T12:24:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.879+0000 7f979ef79700 1 --2- 172.21.15.60:0/3107332088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9798106000 0x7f979810dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.879+0000 7f979ef79700 1 --2- 172.21.15.60:0/3107332088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9798100fb0 0x7f9798101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.879+0000 7f979ef79700 1 --2- 172.21.15.60:0/3107332088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9798101960 0x7f9798105920 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.879+0000 7f979ef79700 1 -- 172.21.15.60:0/3107332088 >> 172.21.15.60:0/3107332088 conn(0x7f9798076d70 msgr2=0x7f9798077180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.879+0000 7f979ef79700 1 -- 172.21.15.60:0/3107332088 shutdown_connections 2024-03-04T12:24:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.879+0000 7f979ef79700 1 -- 172.21.15.60:0/3107332088 wait complete. 2024-03-04T12:24:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.879+0000 7f979ef79700 1 Processor -- start 2024-03-04T12:24:07.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.879+0000 7f979ef79700 1 -- start start 2024-03-04T12:24:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f979ef79700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9798100fb0 0x7f97981a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f979ef79700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9798101960 0x7f97981a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f979ef79700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9798106000 0x7f979819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f979ef79700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9798111040 con 0x7f9798100fb0 2024-03-04T12:24:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f979ef79700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9798110ec0 con 0x7f9798101960 2024-03-04T12:24:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f979ef79700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97981111c0 con 0x7f9798106000 2024-03-04T12:24:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f979d516700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9798106000 0x7f979819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f979cd15700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9798100fb0 0x7f97981a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f9797fff700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9798101960 0x7f97981a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f9797fff700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9798101960 0x7f97981a0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:44092/0 (socket says 172.21.15.60:44092) 2024-03-04T12:24:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f979d516700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9798106000 0x7f979819a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:39852/0 (socket says 172.21.15.60:39852) 2024-03-04T12:24:07.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f9797fff700 1 -- 172.21.15.60:0/4217984567 learned_addr learned my addr 172.21.15.60:0/4217984567 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:07.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f9797fff700 1 -- 172.21.15.60:0/4217984567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9798106000 msgr2=0x7f979819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:07.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f9797fff700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9798106000 0x7f979819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f9797fff700 1 -- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9798100fb0 msgr2=0x7f97981a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:07.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f9797fff700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9798100fb0 0x7f97981a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f9797fff700 1 -- 172.21.15.60:0/4217984567 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979819ad50 con 0x7f9798101960 2024-03-04T12:24:07.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f979cd15700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9798100fb0 0x7f97981a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:24:07.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.880+0000 7f979d516700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9798106000 0x7f979819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:24:07.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.881+0000 7f9797fff700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9798101960 0x7f97981a0970 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f978400c010 tx=0x7f97840047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:07.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.881+0000 7f9795ffb700 1 -- 172.21.15.60:0/4217984567 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784004830 con 0x7f9798101960 2024-03-04T12:24:07.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.881+0000 7f979ef79700 1 -- 172.21.15.60:0/4217984567 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979819afe0 con 0x7f9798101960 2024-03-04T12:24:07.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.881+0000 7f979ef79700 1 -- 172.21.15.60:0/4217984567 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97981a7300 con 0x7f9798101960 2024-03-04T12:24:07.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.881+0000 7f9795ffb700 1 -- 172.21.15.60:0/4217984567 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9784005020 con 0x7f9798101960 2024-03-04T12:24:07.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.881+0000 7f9795ffb700 1 -- 172.21.15.60:0/4217984567 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784003c70 con 0x7f9798101960 2024-03-04T12:24:07.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.882+0000 7f979ef79700 1 -- 172.21.15.60:0/4217984567 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9798062570 con 0x7f9798101960 2024-03-04T12:24:07.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.883+0000 7f9795ffb700 1 -- 172.21.15.60:0/4217984567 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f9784005540 con 0x7f9798101960 2024-03-04T12:24:07.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.884+0000 7f9795ffb700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f97880a1ea0 0x7f97880a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:07.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.884+0000 7f979cd15700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f97880a1ea0 0x7f97880a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.884+0000 7f9795ffb700 1 -- 172.21.15.60:0/4217984567 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f97840cfc30 con 0x7f9798101960 2024-03-04T12:24:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.884+0000 7f9795ffb700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157] conn(0x7f97880ad000 0x7f97880af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.884+0000 7f9795ffb700 1 -- 172.21.15.60:0/4217984567 --> [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9784015ec0 con 0x7f97880ad000 2024-03-04T12:24:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.884+0000 7f9795ffb700 1 -- 172.21.15.60:0/4217984567 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97840cffe0 con 0x7f9798101960 2024-03-04T12:24:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.885+0000 7f979d516700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157] conn(0x7f97880ad000 0x7f97880af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.885+0000 7f979cd15700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f97880a1ea0 0x7f97880a4360 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f978c009de0 tx=0x7f978c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.885+0000 7f979d516700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157] conn(0x7f97880ad000 0x7f97880af4a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:07.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.886+0000 7f9795ffb700 1 -- 172.21.15.60:0/4217984567 <== osd.3 v2:172.21.15.60:6810/1173200157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9784015ec0 con 0x7f97880ad000 2024-03-04T12:24:07.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.904+0000 7f979ef79700 1 -- 172.21.15.60:0/4217984567 --> [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f97980629a0 con 0x7f97880ad000 2024-03-04T12:24:07.911 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.904+0000 7f9795ffb700 1 -- 172.21.15.60:0/4217984567 <== osd.3 v2:172.21.15.60:6810/1173200157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f97980629a0 con 0x7f97880ad000 2024-03-04T12:24:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.904+0000 7f979ef79700 1 -- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157] conn(0x7f97880ad000 msgr2=0x7f97880af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.904+0000 7f979ef79700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157] conn(0x7f97880ad000 0x7f97880af4a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.905+0000 7f979ef79700 1 -- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f97880a1ea0 msgr2=0x7f97880a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.905+0000 7f979ef79700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f97880a1ea0 0x7f97880a4360 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f978c009de0 tx=0x7f978c009190 comp rx=0 tx=0).stop 2024-03-04T12:24:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.905+0000 7f979ef79700 1 -- 172.21.15.60:0/4217984567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9798101960 msgr2=0x7f97981a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.905+0000 7f979ef79700 1 --2- 172.21.15.60:0/4217984567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9798101960 0x7f97981a0970 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f978400c010 tx=0x7f97840047b0 comp rx=0 tx=0).stop 2024-03-04T12:24:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.905+0000 7f979d516700 1 -- 172.21.15.60:0/4217984567 reap_dead start 2024-03-04T12:24:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.905+0000 7f979ef79700 1 -- 172.21.15.60:0/4217984567 shutdown_connections 2024-03-04T12:24:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.905+0000 7f979ef79700 1 -- 172.21.15.60:0/4217984567 >> 172.21.15.60:0/4217984567 conn(0x7f9798076d70 msgr2=0x7f979810c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:07.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.905+0000 7f979ef79700 1 -- 172.21.15.60:0/4217984567 shutdown_connections 2024-03-04T12:24:07.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:07.905+0000 7f979ef79700 1 -- 172.21.15.60:0/4217984567 wait complete. 2024-03-04T12:24:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.105+0000 7f2147bbd700 1 -- 172.21.15.60:0/4083962308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2140100f70 msgr2=0x7f2140101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.105+0000 7f2147bbd700 1 --2- 172.21.15.60:0/4083962308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2140100f70 0x7f2140101350 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f2138009990 tx=0x7f2138009ca0 comp rx=0 tx=0).stop 2024-03-04T12:24:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.105+0000 7f2147bbd700 1 -- 172.21.15.60:0/4083962308 shutdown_connections 2024-03-04T12:24:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.105+0000 7f2147bbd700 1 --2- 172.21.15.60:0/4083962308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2140106030 0x7f214010dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.105+0000 7f2147bbd700 1 --2- 172.21.15.60:0/4083962308 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2140100f70 0x7f2140101350 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.105+0000 7f2147bbd700 1 --2- 172.21.15.60:0/4083962308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2140101920 0x7f2140105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.105+0000 7f2147bbd700 1 -- 172.21.15.60:0/4083962308 >> 172.21.15.60:0/4083962308 conn(0x7f2140077c90 msgr2=0x7f21400780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:08.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.105+0000 7f2147bbd700 1 -- 172.21.15.60:0/4083962308 shutdown_connections 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.105+0000 7f2147bbd700 1 -- 172.21.15.60:0/4083962308 wait complete. 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.106+0000 7f2147bbd700 1 Processor -- start 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.106+0000 7f2147bbd700 1 -- start start 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.106+0000 7f2147bbd700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2140100f70 0x7f2140109ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.106+0000 7f2147bbd700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2140101920 0x7f214010a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.106+0000 7f2147bbd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2140106030 0x7f214010aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.106+0000 7f2147bbd700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214010ff00 con 0x7f2140106030 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.106+0000 7f2147bbd700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214010fd80 con 0x7f2140100f70 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.106+0000 7f2147bbd700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2140110080 con 0x7f2140101920 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.106+0000 7f214615a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2140106030 0x7f214010aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.106+0000 7f214615a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2140106030 0x7f214010aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53922/0 (socket says 172.21.15.60:53922) 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.106+0000 7f214615a700 1 -- 172.21.15.60:0/3272483656 learned_addr learned my addr 172.21.15.60:0/3272483656 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.106+0000 7f2145158700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2140101920 0x7f214010a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.107+0000 7f2145959700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2140100f70 0x7f2140109ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.107+0000 7f214615a700 1 -- 172.21.15.60:0/3272483656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2140101920 msgr2=0x7f214010a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:08.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.107+0000 7f214615a700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2140101920 0x7f214010a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.107+0000 7f214615a700 1 -- 172.21.15.60:0/3272483656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2140100f70 msgr2=0x7f2140109ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.107+0000 7f214615a700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2140100f70 0x7f2140109ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.107+0000 7f214615a700 1 -- 172.21.15.60:0/3272483656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21401aaeb0 con 0x7f2140106030 2024-03-04T12:24:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.107+0000 7f214615a700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2140106030 0x7f214010aa70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f213000ada0 tx=0x7f213000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.107+0000 7f213effd700 1 -- 172.21.15.60:0/3272483656 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2130017020 con 0x7f2140106030 2024-03-04T12:24:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.107+0000 7f2147bbd700 1 -- 172.21.15.60:0/3272483656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21401ab1a0 con 0x7f2140106030 2024-03-04T12:24:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.107+0000 7f213effd700 1 -- 172.21.15.60:0/3272483656 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f213000a0d0 con 0x7f2140106030 2024-03-04T12:24:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.107+0000 7f213effd700 1 -- 172.21.15.60:0/3272483656 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2130005480 con 0x7f2140106030 2024-03-04T12:24:08.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.107+0000 7f2147bbd700 1 -- 172.21.15.60:0/3272483656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21401ab690 con 0x7f2140106030 2024-03-04T12:24:08.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.108+0000 7f2147bbd700 1 -- 172.21.15.60:0/3272483656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21400625b0 con 0x7f2140106030 2024-03-04T12:24:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.109+0000 7f213effd700 1 -- 172.21.15.60:0/3272483656 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f2130005620 con 0x7f2140106030 2024-03-04T12:24:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.110+0000 7f213effd700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21280a1ea0 0x7f21280a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.110+0000 7f2145959700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21280a1ea0 0x7f21280a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.110+0000 7f213effd700 1 -- 172.21.15.60:0/3272483656 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f21300ce360 con 0x7f2140106030 2024-03-04T12:24:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.110+0000 7f213effd700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647] conn(0x7f21280ad000 0x7f21280af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.110+0000 7f213effd700 1 -- 172.21.15.60:0/3272483656 --> [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2130019e50 con 0x7f21280ad000 2024-03-04T12:24:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.110+0000 7f213effd700 1 -- 172.21.15.60:0/3272483656 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21300ce7b0 con 0x7f2140106030 2024-03-04T12:24:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.110+0000 7f2145158700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647] conn(0x7f21280ad000 0x7f21280af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.111+0000 7f2145959700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21280a1ea0 0x7f21280a4360 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2138006010 tx=0x7f213800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:08.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.111+0000 7f2145158700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647] conn(0x7f21280ad000 0x7f21280af4a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:08.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.111+0000 7f213effd700 1 -- 172.21.15.60:0/3272483656 <== osd.0 v2:172.21.15.60:6802/4258653647 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2130019e50 con 0x7f21280ad000 2024-03-04T12:24:08.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.128+0000 7f2147bbd700 1 -- 172.21.15.60:0/3272483656 --> [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2140062a30 con 0x7f21280ad000 2024-03-04T12:24:08.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.128+0000 7f213effd700 1 -- 172.21.15.60:0/3272483656 <== osd.0 v2:172.21.15.60:6802/4258653647 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2140062a30 con 0x7f21280ad000 2024-03-04T12:24:08.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.128+0000 7f2147bbd700 1 -- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647] conn(0x7f21280ad000 msgr2=0x7f21280af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.128+0000 7f2147bbd700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647] conn(0x7f21280ad000 0x7f21280af4a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.129+0000 7f2147bbd700 1 -- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21280a1ea0 msgr2=0x7f21280a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.129+0000 7f2147bbd700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21280a1ea0 0x7f21280a4360 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2138006010 tx=0x7f213800b540 comp rx=0 tx=0).stop 2024-03-04T12:24:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.129+0000 7f2147bbd700 1 -- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2140106030 msgr2=0x7f214010aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.129+0000 7f2147bbd700 1 --2- 172.21.15.60:0/3272483656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2140106030 0x7f214010aa70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f213000ada0 tx=0x7f213000c330 comp rx=0 tx=0).stop 2024-03-04T12:24:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.129+0000 7f214615a700 1 -- 172.21.15.60:0/3272483656 reap_dead start 2024-03-04T12:24:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.129+0000 7f2147bbd700 1 -- 172.21.15.60:0/3272483656 shutdown_connections 2024-03-04T12:24:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.129+0000 7f2147bbd700 1 -- 172.21.15.60:0/3272483656 >> 172.21.15.60:0/3272483656 conn(0x7f2140077c90 msgr2=0x7f21401071b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.129+0000 7f2147bbd700 1 -- 172.21.15.60:0/3272483656 shutdown_connections 2024-03-04T12:24:08.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:08.129+0000 7f2147bbd700 1 -- 172.21.15.60:0/3272483656 wait complete. 2024-03-04T12:24:08.237 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:07 smithi060 ceph-mon[32745]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-04T12:24:08.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:07 smithi086 ceph-mon[36851]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-04T12:24:08.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:07 smithi063 ceph-mon[36717]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-04T12:24:10.086 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:09 smithi060 ceph-mon[32745]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:10.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:09 smithi086 ceph-mon[36851]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:10.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:09 smithi063 ceph-mon[36717]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:10.540 INFO:teuthology.orchestra.run.smithi060.stdout:253403070505 2024-03-04T12:24:10.540 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-04T12:24:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.471+0000 7f0a5f916700 1 -- 172.21.15.60:0/3033111674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a58107190 msgr2=0x7f0a5810ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.471+0000 7f0a5f916700 1 --2- 172.21.15.60:0/3033111674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a58107190 0x7f0a5810ed10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0a5000b0d0 tx=0x7f0a5000b3e0 comp rx=0 tx=0).stop 2024-03-04T12:24:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.471+0000 7f0a5f916700 1 -- 172.21.15.60:0/3033111674 shutdown_connections 2024-03-04T12:24:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.471+0000 7f0a5f916700 1 --2- 172.21.15.60:0/3033111674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a58107190 0x7f0a5810ed10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.471+0000 7f0a5f916700 1 --2- 172.21.15.60:0/3033111674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a5806a720 0x7f0a5806ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.471+0000 7f0a5f916700 1 --2- 172.21.15.60:0/3033111674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a5806b0d0 0x7f0a58106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.471+0000 7f0a5f916700 1 -- 172.21.15.60:0/3033111674 >> 172.21.15.60:0/3033111674 conn(0x7f0a58077c90 msgr2=0x7f0a580780a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:11.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.471+0000 7f0a5f916700 1 -- 172.21.15.60:0/3033111674 shutdown_connections 2024-03-04T12:24:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.471+0000 7f0a5f916700 1 -- 172.21.15.60:0/3033111674 wait complete. 2024-03-04T12:24:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5f916700 1 Processor -- start 2024-03-04T12:24:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5f916700 1 -- start start 2024-03-04T12:24:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5f916700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a5806a720 0x7f0a58100de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5f916700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a5806b0d0 0x7f0a58101320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5f916700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a58107190 0x7f0a58101860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5f916700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58112150 con 0x7f0a5806a720 2024-03-04T12:24:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5f916700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58111fd0 con 0x7f0a58107190 2024-03-04T12:24:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5f916700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a581122d0 con 0x7f0a5806b0d0 2024-03-04T12:24:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5d6b2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a5806a720 0x7f0a58100de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5d6b2700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a5806a720 0x7f0a58100de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53976/0 (socket says 172.21.15.60:53976) 2024-03-04T12:24:11.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5d6b2700 1 -- 172.21.15.60:0/1377490522 learned_addr learned my addr 172.21.15.60:0/1377490522 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5ceb1700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a5806b0d0 0x7f0a58101320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5deb3700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a58107190 0x7f0a58101860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5d6b2700 1 -- 172.21.15.60:0/1377490522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a5806b0d0 msgr2=0x7f0a58101320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5d6b2700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a5806b0d0 0x7f0a58101320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5d6b2700 1 -- 172.21.15.60:0/1377490522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a58107190 msgr2=0x7f0a58101860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5d6b2700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a58107190 0x7f0a58101860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.472+0000 7f0a5d6b2700 1 -- 172.21.15.60:0/1377490522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a581a6b50 con 0x7f0a5806a720 2024-03-04T12:24:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.473+0000 7f0a5d6b2700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a5806a720 0x7f0a58100de0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400e9c0 tx=0x7f0a5400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.473+0000 7f0a4e7fc700 1 -- 172.21.15.60:0/1377490522 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5400cbd0 con 0x7f0a5806a720 2024-03-04T12:24:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.473+0000 7f0a5f916700 1 -- 172.21.15.60:0/1377490522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a581a6e40 con 0x7f0a5806a720 2024-03-04T12:24:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.473+0000 7f0a4e7fc700 1 -- 172.21.15.60:0/1377490522 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a5400ee60 con 0x7f0a5806a720 2024-03-04T12:24:11.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.473+0000 7f0a5f916700 1 -- 172.21.15.60:0/1377490522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a581a72f0 con 0x7f0a5806a720 2024-03-04T12:24:11.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.473+0000 7f0a4e7fc700 1 -- 172.21.15.60:0/1377490522 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a54005630 con 0x7f0a5806a720 2024-03-04T12:24:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.474+0000 7f0a5f916700 1 -- 172.21.15.60:0/1377490522 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0a3c001030 con 0x7f0a5806a720 2024-03-04T12:24:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.476+0000 7f0a4e7fc700 1 -- 172.21.15.60:0/1377490522 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f0a54004750 con 0x7f0a5806a720 2024-03-04T12:24:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.476+0000 7f0a4e7fc700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0a440a1ea0 0x7f0a440a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.477+0000 7f0a5ceb1700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0a440a1ea0 0x7f0a440a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.477+0000 7f0a4e7fc700 1 -- 172.21.15.60:0/1377490522 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f0a540ce7e0 con 0x7f0a5806a720 2024-03-04T12:24:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.477+0000 7f0a4e7fc700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459] conn(0x7f0a440ad000 0x7f0a440af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.477+0000 7f0a4e7fc700 1 -- 172.21.15.60:0/1377490522 --> [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0a440afb70 con 0x7f0a440ad000 2024-03-04T12:24:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.477+0000 7f0a4e7fc700 1 -- 172.21.15.60:0/1377490522 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a540cec30 con 0x7f0a5806a720 2024-03-04T12:24:11.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.477+0000 7f0a5deb3700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459] conn(0x7f0a440ad000 0x7f0a440af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:11.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.477+0000 7f0a5ceb1700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0a440a1ea0 0x7f0a440a4360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0a58100950 tx=0x7f0a48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:11.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.477+0000 7f0a5deb3700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459] conn(0x7f0a440ad000 0x7f0a440af4a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:11.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.479+0000 7f0a4e7fc700 1 -- 172.21.15.60:0/1377490522 <== osd.6 v2:172.21.15.60:6818/285200459 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0a440afb70 con 0x7f0a440ad000 2024-03-04T12:24:11.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.495+0000 7f0a5f916700 1 -- 172.21.15.60:0/1377490522 --> [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0a3c002e60 con 0x7f0a440ad000 2024-03-04T12:24:11.502 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.497+0000 7f0a4e7fc700 1 -- 172.21.15.60:0/1377490522 <== osd.6 v2:172.21.15.60:6818/285200459 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0a3c002e60 con 0x7f0a440ad000 2024-03-04T12:24:11.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.497+0000 7f0a5f916700 1 -- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459] conn(0x7f0a440ad000 msgr2=0x7f0a440af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:11.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.497+0000 7f0a5f916700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459] conn(0x7f0a440ad000 0x7f0a440af4a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:11.503 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.497+0000 7f0a5f916700 1 -- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0a440a1ea0 msgr2=0x7f0a440a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:11.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.497+0000 7f0a5f916700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0a440a1ea0 0x7f0a440a4360 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0a58100950 tx=0x7f0a48009260 comp rx=0 tx=0).stop 2024-03-04T12:24:11.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.497+0000 7f0a5f916700 1 -- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a5806a720 msgr2=0x7f0a58100de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:11.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.497+0000 7f0a5f916700 1 --2- 172.21.15.60:0/1377490522 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a5806a720 0x7f0a58100de0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400e9c0 tx=0x7f0a5400ecd0 comp rx=0 tx=0).stop 2024-03-04T12:24:11.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.497+0000 7f0a5deb3700 1 -- 172.21.15.60:0/1377490522 reap_dead start 2024-03-04T12:24:11.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.497+0000 7f0a5f916700 1 -- 172.21.15.60:0/1377490522 shutdown_connections 2024-03-04T12:24:11.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.497+0000 7f0a5f916700 1 -- 172.21.15.60:0/1377490522 >> 172.21.15.60:0/1377490522 conn(0x7f0a58077c90 msgr2=0x7f0a580ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:11.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.497+0000 7f0a5f916700 1 -- 172.21.15.60:0/1377490522 shutdown_connections 2024-03-04T12:24:11.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.497+0000 7f0a5f916700 1 -- 172.21.15.60:0/1377490522 wait complete. 2024-03-04T12:24:12.117 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:11 smithi060 ceph-mon[32745]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:12.117 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:11 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:24:12.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.114+0000 7f52aaf7a700 1 -- 172.21.15.60:0/22209514 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52a4100500 msgr2=0x7f52a4100980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:12.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.114+0000 7f52aaf7a700 1 --2- 172.21.15.60:0/22209514 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52a4100500 0x7f52a4100980 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f529000aad0 tx=0x7f529000ade0 comp rx=0 tx=0).stop 2024-03-04T12:24:12.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.114+0000 7f52aaf7a700 1 -- 172.21.15.60:0/22209514 shutdown_connections 2024-03-04T12:24:12.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.114+0000 7f52aaf7a700 1 --2- 172.21.15.60:0/22209514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52a4100ec0 0x7f52a410d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.114+0000 7f52aaf7a700 1 --2- 172.21.15.60:0/22209514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52a41065b0 0x7f52a4106990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.114+0000 7f52aaf7a700 1 --2- 172.21.15.60:0/22209514 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52a4100500 0x7f52a4100980 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.114+0000 7f52aaf7a700 1 -- 172.21.15.60:0/22209514 >> 172.21.15.60:0/22209514 conn(0x7f52a4077320 msgr2=0x7f52a40fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.114+0000 7f52aaf7a700 1 -- 172.21.15.60:0/22209514 shutdown_connections 2024-03-04T12:24:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.114+0000 7f52aaf7a700 1 -- 172.21.15.60:0/22209514 wait complete. 2024-03-04T12:24:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52aaf7a700 1 Processor -- start 2024-03-04T12:24:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52aaf7a700 1 -- start start 2024-03-04T12:24:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52aaf7a700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52a4100500 0x7f52a41985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52aaf7a700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52a4100ec0 0x7f52a4198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52aaf7a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52a41065b0 0x7f52a419ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52aaf7a700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a410ffd0 con 0x7f52a41065b0 2024-03-04T12:24:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52aaf7a700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a410fe50 con 0x7f52a4100500 2024-03-04T12:24:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52aaf7a700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a4110150 con 0x7f52a4100ec0 2024-03-04T12:24:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52a9517700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52a41065b0 0x7f52a419ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52a8d16700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52a4100500 0x7f52a41985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52a9517700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52a41065b0 0x7f52a419ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53998/0 (socket says 172.21.15.60:53998) 2024-03-04T12:24:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52a9517700 1 -- 172.21.15.60:0/485674441 learned_addr learned my addr 172.21.15.60:0/485674441 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.115+0000 7f52a3fff700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52a4100ec0 0x7f52a4198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.116+0000 7f52a8d16700 1 -- 172.21.15.60:0/485674441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52a4100ec0 msgr2=0x7f52a4198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.116+0000 7f52a8d16700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52a4100ec0 0x7f52a4198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.116+0000 7f52a8d16700 1 -- 172.21.15.60:0/485674441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52a41065b0 msgr2=0x7f52a419ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.116+0000 7f52a8d16700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52a41065b0 0x7f52a419ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.116+0000 7f52a8d16700 1 -- 172.21.15.60:0/485674441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a419d570 con 0x7f52a4100500 2024-03-04T12:24:12.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.116+0000 7f52a8d16700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52a4100500 0x7f52a41985b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f529800e9f0 tx=0x7f529800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.116+0000 7f52a1ffb700 1 -- 172.21.15.60:0/485674441 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529800c9c0 con 0x7f52a4100500 2024-03-04T12:24:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.116+0000 7f52aaf7a700 1 -- 172.21.15.60:0/485674441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a419d860 con 0x7f52a4100500 2024-03-04T12:24:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.116+0000 7f52a1ffb700 1 -- 172.21.15.60:0/485674441 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f529800cb60 con 0x7f52a4100500 2024-03-04T12:24:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.116+0000 7f52a1ffb700 1 -- 172.21.15.60:0/485674441 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298018680 con 0x7f52a4100500 2024-03-04T12:24:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.116+0000 7f52aaf7a700 1 -- 172.21.15.60:0/485674441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a41a50e0 con 0x7f52a4100500 2024-03-04T12:24:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.117+0000 7f52aaf7a700 1 -- 172.21.15.60:0/485674441 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f52a4062530 con 0x7f52a4100500 2024-03-04T12:24:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.118+0000 7f52a1ffb700 1 -- 172.21.15.60:0/485674441 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f5298010430 con 0x7f52a4100500 2024-03-04T12:24:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.119+0000 7f52a1ffb700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f52940a1ea0 0x7f52940a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.119+0000 7f52a3fff700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f52940a1ea0 0x7f52940a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.119+0000 7f52a1ffb700 1 -- 172.21.15.60:0/485674441 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f52980ce300 con 0x7f52a4100500 2024-03-04T12:24:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.119+0000 7f52a1ffb700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955] conn(0x7f52940ad000 0x7f52940af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.119+0000 7f52a1ffb700 1 -- 172.21.15.60:0/485674441 --> [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f529800be50 con 0x7f52940ad000 2024-03-04T12:24:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.119+0000 7f52a1ffb700 1 -- 172.21.15.60:0/485674441 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52980ce6b0 con 0x7f52a4100500 2024-03-04T12:24:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.119+0000 7f52a3fff700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f52940a1ea0 0x7f52940a4360 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f529000afd0 tx=0x7f5290005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.119+0000 7f52a9517700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955] conn(0x7f52940ad000 0x7f52940af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.119+0000 7f52a9517700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955] conn(0x7f52940ad000 0x7f52940af4a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.120+0000 7f52a1ffb700 1 -- 172.21.15.60:0/485674441 <== osd.4 v2:172.21.15.63:6808/793955 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f529800be50 con 0x7f52940ad000 2024-03-04T12:24:12.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.137+0000 7f52aaf7a700 1 -- 172.21.15.60:0/485674441 --> [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f52a4062930 con 0x7f52940ad000 2024-03-04T12:24:12.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.138+0000 7f52a1ffb700 1 -- 172.21.15.60:0/485674441 <== osd.4 v2:172.21.15.63:6808/793955 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f52a4062930 con 0x7f52940ad000 2024-03-04T12:24:12.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.138+0000 7f52aaf7a700 1 -- 172.21.15.60:0/485674441 >> [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955] conn(0x7f52940ad000 msgr2=0x7f52940af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.138+0000 7f52aaf7a700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955] conn(0x7f52940ad000 0x7f52940af4a0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.138+0000 7f52aaf7a700 1 -- 172.21.15.60:0/485674441 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f52940a1ea0 msgr2=0x7f52940a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.138+0000 7f52aaf7a700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f52940a1ea0 0x7f52940a4360 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f529000afd0 tx=0x7f5290005fb0 comp rx=0 tx=0).stop 2024-03-04T12:24:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.138+0000 7f52aaf7a700 1 -- 172.21.15.60:0/485674441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52a4100500 msgr2=0x7f52a41985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.138+0000 7f52aaf7a700 1 --2- 172.21.15.60:0/485674441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52a4100500 0x7f52a41985b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f529800e9f0 tx=0x7f529800ed00 comp rx=0 tx=0).stop 2024-03-04T12:24:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.138+0000 7f52a9517700 1 -- 172.21.15.60:0/485674441 reap_dead start 2024-03-04T12:24:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.138+0000 7f52aaf7a700 1 -- 172.21.15.60:0/485674441 shutdown_connections 2024-03-04T12:24:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.138+0000 7f52aaf7a700 1 -- 172.21.15.60:0/485674441 >> 172.21.15.60:0/485674441 conn(0x7f52a4077320 msgr2=0x7f52a40fe9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.138+0000 7f52aaf7a700 1 -- 172.21.15.60:0/485674441 shutdown_connections 2024-03-04T12:24:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:12.138+0000 7f52aaf7a700 1 -- 172.21.15.60:0/485674441 wait complete. 2024-03-04T12:24:12.316 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:11 smithi086 ceph-mon[36851]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:12.316 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:11 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:24:12.336 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:11 smithi063 ceph-mon[36717]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:12.336 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:11 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:24:13.028 INFO:teuthology.orchestra.run.smithi060.stdout:34359738448 2024-03-04T12:24:13.028 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-04T12:24:13.468 INFO:teuthology.orchestra.run.smithi060.stdout:107374182467 2024-03-04T12:24:13.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-04T12:24:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:14 smithi060 ceph-mon[32745]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:24:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:14 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:14 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:14.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:14 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:14.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:14 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:14 smithi086 ceph-mon[36851]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:24:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:14 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:14 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:14 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:14 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:14 smithi063 ceph-mon[36717]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:24:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:14 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:14 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:14 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:14 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:14.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.234+0000 7f1d69831700 1 -- 172.21.15.60:0/1355358393 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64100ec0 msgr2=0x7f1d6410d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.234+0000 7f1d69831700 1 --2- 172.21.15.60:0/1355358393 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64100ec0 0x7f1d6410d380 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00b0d0 tx=0x7f1d5c00b3e0 comp rx=0 tx=0).stop 2024-03-04T12:24:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.234+0000 7f1d69831700 1 -- 172.21.15.60:0/1355358393 shutdown_connections 2024-03-04T12:24:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.234+0000 7f1d69831700 1 --2- 172.21.15.60:0/1355358393 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64100ec0 0x7f1d6410d380 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.234+0000 7f1d69831700 1 --2- 172.21.15.60:0/1355358393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d641065b0 0x7f1d64106990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.234+0000 7f1d69831700 1 --2- 172.21.15.60:0/1355358393 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d64100500 0x7f1d64100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.234+0000 7f1d69831700 1 -- 172.21.15.60:0/1355358393 >> 172.21.15.60:0/1355358393 conn(0x7f1d64077320 msgr2=0x7f1d640fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.234+0000 7f1d69831700 1 -- 172.21.15.60:0/1355358393 shutdown_connections 2024-03-04T12:24:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.234+0000 7f1d69831700 1 -- 172.21.15.60:0/1355358393 wait complete. 2024-03-04T12:24:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.235+0000 7f1d69831700 1 Processor -- start 2024-03-04T12:24:14.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.235+0000 7f1d69831700 1 -- start start 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.235+0000 7f1d69831700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64100500 0x7f1d64102170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.235+0000 7f1d69831700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d64100ec0 0x7f1d641026b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.235+0000 7f1d69831700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d641065b0 0x7f1d64102bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.235+0000 7f1d69831700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d64111080 con 0x7f1d64100500 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.235+0000 7f1d69831700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d64110f00 con 0x7f1d641065b0 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.235+0000 7f1d69831700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d64111200 con 0x7f1d64100ec0 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.235+0000 7f1d62ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64100500 0x7f1d64102170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.235+0000 7f1d62ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64100500 0x7f1d64102170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53948/0 (socket says 172.21.15.60:53948) 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.235+0000 7f1d62ffd700 1 -- 172.21.15.60:0/3123473865 learned_addr learned my addr 172.21.15.60:0/3123473865 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d637fe700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d641065b0 0x7f1d64102bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d627fc700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d64100ec0 0x7f1d641026b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d62ffd700 1 -- 172.21.15.60:0/3123473865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d64100ec0 msgr2=0x7f1d641026b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:14.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d62ffd700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d64100ec0 0x7f1d641026b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:14.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d62ffd700 1 -- 172.21.15.60:0/3123473865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d641065b0 msgr2=0x7f1d64102bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:14.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d62ffd700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d641065b0 0x7f1d64102bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:14.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d62ffd700 1 -- 172.21.15.60:0/3123473865 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d641a6a40 con 0x7f1d64100500 2024-03-04T12:24:14.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d627fc700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d64100ec0 0x7f1d641026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:24:14.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d62ffd700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64100500 0x7f1d64102170 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1d5400d980 tx=0x7f1d5400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:14.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d6882f700 1 -- 172.21.15.60:0/3123473865 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d54017070 con 0x7f1d64100500 2024-03-04T12:24:14.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d69831700 1 -- 172.21.15.60:0/3123473865 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d641a6d30 con 0x7f1d64100500 2024-03-04T12:24:14.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d6882f700 1 -- 172.21.15.60:0/3123473865 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d54005020 con 0x7f1d64100500 2024-03-04T12:24:14.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d6882f700 1 -- 172.21.15.60:0/3123473865 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d54009bc0 con 0x7f1d64100500 2024-03-04T12:24:14.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.236+0000 7f1d69831700 1 -- 172.21.15.60:0/3123473865 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d641a7210 con 0x7f1d64100500 2024-03-04T12:24:14.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.237+0000 7f1d69831700 1 -- 172.21.15.60:0/3123473865 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1d64104520 con 0x7f1d64100500 2024-03-04T12:24:14.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.238+0000 7f1d6882f700 1 -- 172.21.15.60:0/3123473865 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f1d540051d0 con 0x7f1d64100500 2024-03-04T12:24:14.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.239+0000 7f1d6882f700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1d4c0a1ea0 0x7f1d4c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:14.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.239+0000 7f1d627fc700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1d4c0a1ea0 0x7f1d4c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:14.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.239+0000 7f1d6882f700 1 -- 172.21.15.60:0/3123473865 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f1d540ce6f0 con 0x7f1d64100500 2024-03-04T12:24:14.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.239+0000 7f1d6882f700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290] conn(0x7f1d4c0ad000 0x7f1d4c0af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:14.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.239+0000 7f1d6882f700 1 -- 172.21.15.60:0/3123473865 --> [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d4c0afb70 con 0x7f1d4c0ad000 2024-03-04T12:24:14.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.239+0000 7f1d6882f700 1 -- 172.21.15.60:0/3123473865 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d540ceb40 con 0x7f1d64100500 2024-03-04T12:24:14.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.240+0000 7f1d627fc700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1d4c0a1ea0 0x7f1d4c0a4360 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1d58005950 tx=0x7f1d580058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:14.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.240+0000 7f1d637fe700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290] conn(0x7f1d4c0ad000 0x7f1d4c0af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:14.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.240+0000 7f1d637fe700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290] conn(0x7f1d4c0ad000 0x7f1d4c0af4a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:14.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.241+0000 7f1d6882f700 1 -- 172.21.15.60:0/3123473865 <== osd.7 v2:172.21.15.63:6816/942963290 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1d4c0afb70 con 0x7f1d4c0ad000 2024-03-04T12:24:14.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.260+0000 7f1d69831700 1 -- 172.21.15.60:0/3123473865 --> [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1d64101c20 con 0x7f1d4c0ad000 2024-03-04T12:24:14.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.260+0000 7f1d6882f700 1 -- 172.21.15.60:0/3123473865 <== osd.7 v2:172.21.15.63:6816/942963290 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1d64101c20 con 0x7f1d4c0ad000 2024-03-04T12:24:14.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.261+0000 7f1d69831700 1 -- 172.21.15.60:0/3123473865 >> [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290] conn(0x7f1d4c0ad000 msgr2=0x7f1d4c0af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.261+0000 7f1d69831700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290] conn(0x7f1d4c0ad000 0x7f1d4c0af4a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.261+0000 7f1d69831700 1 -- 172.21.15.60:0/3123473865 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1d4c0a1ea0 msgr2=0x7f1d4c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.261+0000 7f1d69831700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f1d4c0a1ea0 0x7f1d4c0a4360 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1d58005950 tx=0x7f1d580058e0 comp rx=0 tx=0).stop 2024-03-04T12:24:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.261+0000 7f1d69831700 1 -- 172.21.15.60:0/3123473865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64100500 msgr2=0x7f1d64102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.261+0000 7f1d69831700 1 --2- 172.21.15.60:0/3123473865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1d64100500 0x7f1d64102170 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1d5400d980 tx=0x7f1d5400dc90 comp rx=0 tx=0).stop 2024-03-04T12:24:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.261+0000 7f1d637fe700 1 -- 172.21.15.60:0/3123473865 reap_dead start 2024-03-04T12:24:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.261+0000 7f1d69831700 1 -- 172.21.15.60:0/3123473865 shutdown_connections 2024-03-04T12:24:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.261+0000 7f1d69831700 1 -- 172.21.15.60:0/3123473865 >> 172.21.15.60:0/3123473865 conn(0x7f1d64077320 msgr2=0x7f1d640ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.261+0000 7f1d69831700 1 -- 172.21.15.60:0/3123473865 shutdown_connections 2024-03-04T12:24:14.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:11.261+0000 7f1d69831700 1 -- 172.21.15.60:0/3123473865 wait complete. 2024-03-04T12:24:15.034 INFO:teuthology.orchestra.run.smithi060.stdout:219043332142 2024-03-04T12:24:15.034 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-04T12:24:16.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:16 smithi086 ceph-mon[36851]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-04T12:24:16.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:16 smithi063 ceph-mon[36717]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-04T12:24:16.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:16 smithi060 ceph-mon[32745]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-04T12:24:17.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.087+0000 7f7c46d15700 1 -- 172.21.15.60:0/3003995906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c401009c0 msgr2=0x7f7c40100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:17.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.087+0000 7f7c46d15700 1 --2- 172.21.15.60:0/3003995906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c401009c0 0x7f7c40100da0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7c340099c0 tx=0x7f7c34009cd0 comp rx=0 tx=0).stop 2024-03-04T12:24:17.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.088+0000 7f7c46d15700 1 -- 172.21.15.60:0/3003995906 shutdown_connections 2024-03-04T12:24:17.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.088+0000 7f7c46d15700 1 --2- 172.21.15.60:0/3003995906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c401071e0 0x7f7c4010ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:17.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.088+0000 7f7c46d15700 1 --2- 172.21.15.60:0/3003995906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c401009c0 0x7f7c40100da0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:17.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.088+0000 7f7c46d15700 1 --2- 172.21.15.60:0/3003995906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c40101370 0x7f7c40106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:17.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.088+0000 7f7c46d15700 1 -- 172.21.15.60:0/3003995906 >> 172.21.15.60:0/3003995906 conn(0x7f7c40076060 msgr2=0x7f7c40076470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:17.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.088+0000 7f7c46d15700 1 -- 172.21.15.60:0/3003995906 shutdown_connections 2024-03-04T12:24:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.088+0000 7f7c46d15700 1 -- 172.21.15.60:0/3003995906 wait complete. 2024-03-04T12:24:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.088+0000 7f7c46d15700 1 Processor -- start 2024-03-04T12:24:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.088+0000 7f7c46d15700 1 -- start start 2024-03-04T12:24:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.088+0000 7f7c46d15700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c40101370 0x7f7c4019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c46d15700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c401071e0 0x7f7c4019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c46d15700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c40198630 0x7f7c40198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c46d15700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c40112170 con 0x7f7c40101370 2024-03-04T12:24:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c46d15700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c40111ff0 con 0x7f7c40198630 2024-03-04T12:24:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c46d15700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c401122f0 con 0x7f7c401071e0 2024-03-04T12:24:17.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c44ab1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c40101370 0x7f7c4019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c44ab1700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c40101370 0x7f7c4019e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35966/0 (socket says 172.21.15.60:35966) 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c44ab1700 1 -- 172.21.15.60:0/379346045 learned_addr learned my addr 172.21.15.60:0/379346045 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c3ffff700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c401071e0 0x7f7c4019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c452b2700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c40198630 0x7f7c40198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c44ab1700 1 -- 172.21.15.60:0/379346045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c401071e0 msgr2=0x7f7c4019eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c44ab1700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c401071e0 0x7f7c4019eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c44ab1700 1 -- 172.21.15.60:0/379346045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c40198630 msgr2=0x7f7c40198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c44ab1700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c40198630 0x7f7c40198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c44ab1700 1 -- 172.21.15.60:0/379346045 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c40199250 con 0x7f7c40101370 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c452b2700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c40198630 0x7f7c40198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c44ab1700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c40101370 0x7f7c4019e560 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7c340098e0 tx=0x7f7c340047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c3dffb700 1 -- 172.21.15.60:0/379346045 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34026070 con 0x7f7c40101370 2024-03-04T12:24:17.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.089+0000 7f7c46d15700 1 -- 172.21.15.60:0/379346045 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c401a5310 con 0x7f7c40101370 2024-03-04T12:24:17.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.090+0000 7f7c3dffb700 1 -- 172.21.15.60:0/379346045 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c34003e40 con 0x7f7c40101370 2024-03-04T12:24:17.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.090+0000 7f7c3dffb700 1 -- 172.21.15.60:0/379346045 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3400fbb0 con 0x7f7c40101370 2024-03-04T12:24:17.099 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.090+0000 7f7c46d15700 1 -- 172.21.15.60:0/379346045 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c401a56b0 con 0x7f7c40101370 2024-03-04T12:24:17.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.091+0000 7f7c46d15700 1 -- 172.21.15.60:0/379346045 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7c400625b0 con 0x7f7c40101370 2024-03-04T12:24:17.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.093+0000 7f7c3dffb700 1 -- 172.21.15.60:0/379346045 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f7c34029070 con 0x7f7c40101370 2024-03-04T12:24:17.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.093+0000 7f7c3dffb700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7c300a1ea0 0x7f7c300a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:17.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.093+0000 7f7c3ffff700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7c300a1ea0 0x7f7c300a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:17.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.093+0000 7f7c3dffb700 1 -- 172.21.15.60:0/379346045 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f7c340cf200 con 0x7f7c40101370 2024-03-04T12:24:17.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.094+0000 7f7c3dffb700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384] conn(0x7f7c300ad000 0x7f7c300af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:17.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.094+0000 7f7c3dffb700 1 -- 172.21.15.60:0/379346045 --> [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c340043d0 con 0x7f7c300ad000 2024-03-04T12:24:17.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.094+0000 7f7c3dffb700 1 -- 172.21.15.60:0/379346045 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c340cf650 con 0x7f7c40101370 2024-03-04T12:24:17.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.094+0000 7f7c452b2700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384] conn(0x7f7c300ad000 0x7f7c300af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:17.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.094+0000 7f7c3ffff700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7c300a1ea0 0x7f7c300a4360 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7c40199df0 tx=0x7f7c2c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:17.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.094+0000 7f7c452b2700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384] conn(0x7f7c300ad000 0x7f7c300af4a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:17.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.095+0000 7f7c3dffb700 1 -- 172.21.15.60:0/379346045 <== osd.9 v2:172.21.15.60:6826/1552052384 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7c340043d0 con 0x7f7c300ad000 2024-03-04T12:24:17.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.111+0000 7f7c46d15700 1 -- 172.21.15.60:0/379346045 --> [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7c40062a30 con 0x7f7c300ad000 2024-03-04T12:24:17.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.112+0000 7f7c3dffb700 1 -- 172.21.15.60:0/379346045 <== osd.9 v2:172.21.15.60:6826/1552052384 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7c40062a30 con 0x7f7c300ad000 2024-03-04T12:24:17.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.112+0000 7f7c46d15700 1 -- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384] conn(0x7f7c300ad000 msgr2=0x7f7c300af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.112+0000 7f7c46d15700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384] conn(0x7f7c300ad000 0x7f7c300af4a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.112+0000 7f7c46d15700 1 -- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7c300a1ea0 msgr2=0x7f7c300a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.112+0000 7f7c46d15700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7c300a1ea0 0x7f7c300a4360 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7c40199df0 tx=0x7f7c2c00b410 comp rx=0 tx=0).stop 2024-03-04T12:24:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.112+0000 7f7c46d15700 1 -- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c40101370 msgr2=0x7f7c4019e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.112+0000 7f7c46d15700 1 --2- 172.21.15.60:0/379346045 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7c40101370 0x7f7c4019e560 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7c340098e0 tx=0x7f7c340047b0 comp rx=0 tx=0).stop 2024-03-04T12:24:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.112+0000 7f7c452b2700 1 -- 172.21.15.60:0/379346045 reap_dead start 2024-03-04T12:24:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.112+0000 7f7c46d15700 1 -- 172.21.15.60:0/379346045 shutdown_connections 2024-03-04T12:24:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.112+0000 7f7c46d15700 1 -- 172.21.15.60:0/379346045 >> 172.21.15.60:0/379346045 conn(0x7f7c40076060 msgr2=0x7f7c40108380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.112+0000 7f7c46d15700 1 -- 172.21.15.60:0/379346045 shutdown_connections 2024-03-04T12:24:17.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:17.112+0000 7f7c46d15700 1 -- 172.21.15.60:0/379346045 wait complete. 2024-03-04T12:24:17.559 INFO:teuthology.orchestra.run.smithi060.stdout:163208757304 2024-03-04T12:24:17.560 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-04T12:24:18.104 INFO:teuthology.orchestra.run.smithi060.stdout:141733920830 2024-03-04T12:24:18.104 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-04T12:24:18.311 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:18 smithi060 ceph-mon[32745]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-04T12:24:18.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:18 smithi086 ceph-mon[36851]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-04T12:24:18.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:18 smithi063 ceph-mon[36717]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-04T12:24:18.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.561+0000 7efec8137700 1 -- 172.21.15.60:0/3933223259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec006aec0 msgr2=0x7efec010ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:18.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.561+0000 7efec8137700 1 --2- 172.21.15.60:0/3933223259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec006aec0 0x7efec010ece0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efeb800b0d0 tx=0x7efeb800b3e0 comp rx=0 tx=0).stop 2024-03-04T12:24:18.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.561+0000 7efec8137700 1 -- 172.21.15.60:0/3933223259 shutdown_connections 2024-03-04T12:24:18.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.561+0000 7efec8137700 1 --2- 172.21.15.60:0/3933223259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec006aec0 0x7efec010ece0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:18.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.561+0000 7efec8137700 1 --2- 172.21.15.60:0/3933223259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efec0107ea0 0x7efec0108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:18.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.561+0000 7efec8137700 1 --2- 172.21.15.60:0/3933223259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec006a520 0x7efec006a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:18.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.561+0000 7efec8137700 1 -- 172.21.15.60:0/3933223259 >> 172.21.15.60:0/3933223259 conn(0x7efec0076d40 msgr2=0x7efec0077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:18.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.561+0000 7efec8137700 1 -- 172.21.15.60:0/3933223259 shutdown_connections 2024-03-04T12:24:18.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.561+0000 7efec8137700 1 -- 172.21.15.60:0/3933223259 wait complete. 2024-03-04T12:24:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.562+0000 7efec8137700 1 Processor -- start 2024-03-04T12:24:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.562+0000 7efec8137700 1 -- start start 2024-03-04T12:24:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.562+0000 7efec8137700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec006a520 0x7efec01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.562+0000 7efec8137700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efec006aec0 0x7efec0198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.562+0000 7efec8137700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec0107ea0 0x7efec019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.562+0000 7efec8137700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec0112120 con 0x7efec0107ea0 2024-03-04T12:24:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.562+0000 7efec8137700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec0111fa0 con 0x7efec006a520 2024-03-04T12:24:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.562+0000 7efec8137700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec01122a0 con 0x7efec006aec0 2024-03-04T12:24:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.562+0000 7efec66d4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec0107ea0 0x7efec019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec66d4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec0107ea0 0x7efec019ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35978/0 (socket says 172.21.15.60:35978) 2024-03-04T12:24:18.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec66d4700 1 -- 172.21.15.60:0/3323475309 learned_addr learned my addr 172.21.15.60:0/3323475309 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec5ed3700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec006a520 0x7efec01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec56d2700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efec006aec0 0x7efec0198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec66d4700 1 -- 172.21.15.60:0/3323475309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efec006aec0 msgr2=0x7efec0198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec66d4700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efec006aec0 0x7efec0198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec66d4700 1 -- 172.21.15.60:0/3323475309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec006a520 msgr2=0x7efec01985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec66d4700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec006a520 0x7efec01985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec66d4700 1 -- 172.21.15.60:0/3323475309 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec019d570 con 0x7efec0107ea0 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec5ed3700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efec006a520 0x7efec01985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec66d4700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec0107ea0 0x7efec019ce30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7efeb8006070 tx=0x7efeb8004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efeb2ffd700 1 -- 172.21.15.60:0/3323475309 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb8021070 con 0x7efec0107ea0 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efeb2ffd700 1 -- 172.21.15.60:0/3323475309 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efeb8007c80 con 0x7efec0107ea0 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec8137700 1 -- 172.21.15.60:0/3323475309 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec019d800 con 0x7efec0107ea0 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efeb2ffd700 1 -- 172.21.15.60:0/3323475309 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb80053c0 con 0x7efec0107ea0 2024-03-04T12:24:18.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.563+0000 7efec8137700 1 -- 172.21.15.60:0/3323475309 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec01a50e0 con 0x7efec0107ea0 2024-03-04T12:24:18.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.564+0000 7efec8137700 1 -- 172.21.15.60:0/3323475309 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efec00625b0 con 0x7efec0107ea0 2024-03-04T12:24:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.565+0000 7efeb2ffd700 1 -- 172.21.15.60:0/3323475309 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7efeb8024070 con 0x7efec0107ea0 2024-03-04T12:24:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.566+0000 7efeb2ffd700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efeac0a1ea0 0x7efeac0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.566+0000 7efec5ed3700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efeac0a1ea0 0x7efeac0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.566+0000 7efeb2ffd700 1 -- 172.21.15.60:0/3323475309 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7efeb80cf850 con 0x7efec0107ea0 2024-03-04T12:24:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.566+0000 7efeb2ffd700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227] conn(0x7efeac0ad000 0x7efeac0af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.566+0000 7efeb2ffd700 1 -- 172.21.15.60:0/3323475309 --> [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efeb801fe80 con 0x7efeac0ad000 2024-03-04T12:24:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.566+0000 7efeb2ffd700 1 -- 172.21.15.60:0/3323475309 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7efeb80cfca0 con 0x7efec0107ea0 2024-03-04T12:24:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.566+0000 7efec5ed3700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efeac0a1ea0 0x7efeac0a4360 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efeb4005e10 tx=0x7efeb4005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.566+0000 7efec56d2700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227] conn(0x7efeac0ad000 0x7efeac0af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:18.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.567+0000 7efec56d2700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227] conn(0x7efeac0ad000 0x7efeac0af4a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:18.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.568+0000 7efeb2ffd700 1 -- 172.21.15.60:0/3323475309 <== osd.10 v2:172.21.15.63:6824/4239140227 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7efeb801fe80 con 0x7efeac0ad000 2024-03-04T12:24:18.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.585+0000 7efec8137700 1 -- 172.21.15.60:0/3323475309 --> [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efec0062a30 con 0x7efeac0ad000 2024-03-04T12:24:18.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.585+0000 7efeb2ffd700 1 -- 172.21.15.60:0/3323475309 <== osd.10 v2:172.21.15.63:6824/4239140227 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7efec0062a30 con 0x7efeac0ad000 2024-03-04T12:24:18.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.586+0000 7efec8137700 1 -- 172.21.15.60:0/3323475309 >> [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227] conn(0x7efeac0ad000 msgr2=0x7efeac0af4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:18.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.586+0000 7efec8137700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227] conn(0x7efeac0ad000 0x7efeac0af4a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:18.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.586+0000 7efec8137700 1 -- 172.21.15.60:0/3323475309 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efeac0a1ea0 msgr2=0x7efeac0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:18.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.586+0000 7efec8137700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efeac0a1ea0 0x7efeac0a4360 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efeb4005e10 tx=0x7efeb4005d80 comp rx=0 tx=0).stop 2024-03-04T12:24:18.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.586+0000 7efec8137700 1 -- 172.21.15.60:0/3323475309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec0107ea0 msgr2=0x7efec019ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:18.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.586+0000 7efec8137700 1 --2- 172.21.15.60:0/3323475309 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efec0107ea0 0x7efec019ce30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7efeb8006070 tx=0x7efeb8004db0 comp rx=0 tx=0).stop 2024-03-04T12:24:18.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.586+0000 7efec66d4700 1 -- 172.21.15.60:0/3323475309 reap_dead start 2024-03-04T12:24:18.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.586+0000 7efec8137700 1 -- 172.21.15.60:0/3323475309 shutdown_connections 2024-03-04T12:24:18.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.586+0000 7efec8137700 1 -- 172.21.15.60:0/3323475309 >> 172.21.15.60:0/3323475309 conn(0x7efec0076d40 msgr2=0x7efec00ff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:18.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.586+0000 7efec8137700 1 -- 172.21.15.60:0/3323475309 shutdown_connections 2024-03-04T12:24:18.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:18.586+0000 7efec8137700 1 -- 172.21.15.60:0/3323475309 wait complete. 2024-03-04T12:24:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.081+0000 7f002af97700 1 -- 172.21.15.60:0/544123899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0024107e70 msgr2=0x7f0024108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.081+0000 7f002af97700 1 --2- 172.21.15.60:0/544123899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0024107e70 0x7f0024108250 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f00140099c0 tx=0x7f0014009cd0 comp rx=0 tx=0).stop 2024-03-04T12:24:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.081+0000 7f002af97700 1 -- 172.21.15.60:0/544123899 shutdown_connections 2024-03-04T12:24:19.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.081+0000 7f002af97700 1 --2- 172.21.15.60:0/544123899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f002406a470 0x7f002410ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.081+0000 7f002af97700 1 --2- 172.21.15.60:0/544123899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0024107e70 0x7f0024108250 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.081+0000 7f002af97700 1 --2- 172.21.15.60:0/544123899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0024069ad0 0x7f0024069f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.081+0000 7f002af97700 1 -- 172.21.15.60:0/544123899 >> 172.21.15.60:0/544123899 conn(0x7f0024077c50 msgr2=0x7f0024078060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.081+0000 7f002af97700 1 -- 172.21.15.60:0/544123899 shutdown_connections 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.081+0000 7f002af97700 1 -- 172.21.15.60:0/544123899 wait complete. 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f002af97700 1 Processor -- start 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f002af97700 1 -- start start 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f002af97700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0024069ad0 0x7f00241985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f002af97700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002406a470 0x7f0024198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f002af97700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0024107e70 0x7f002419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f002af97700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00241120d0 con 0x7f002406a470 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f002af97700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0024111f50 con 0x7f0024107e70 2024-03-04T12:24:19.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f002af97700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0024112250 con 0x7f0024069ad0 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f0023fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002406a470 0x7f0024198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f0023fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002406a470 0x7f0024198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:35984/0 (socket says 172.21.15.60:35984) 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f0023fff700 1 -- 172.21.15.60:0/3139319374 learned_addr learned my addr 172.21.15.60:0/3139319374 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f0029534700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0024107e70 0x7f002419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f0028d33700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0024069ad0 0x7f00241985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f0023fff700 1 -- 172.21.15.60:0/3139319374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0024069ad0 msgr2=0x7f00241985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f0023fff700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0024069ad0 0x7f00241985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f0023fff700 1 -- 172.21.15.60:0/3139319374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0024107e70 msgr2=0x7f002419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f0023fff700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0024107e70 0x7f002419ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f0023fff700 1 -- 172.21.15.60:0/3139319374 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002419d5a0 con 0x7f002406a470 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f0028d33700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0024069ad0 0x7f00241985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.082+0000 7f0029534700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0024107e70 0x7f002419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.083+0000 7f0023fff700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002406a470 0x7f0024198b20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f001800e9a0 tx=0x7f001800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.083+0000 7f0021ffb700 1 -- 172.21.15.60:0/3139319374 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001800cbd0 con 0x7f002406a470 2024-03-04T12:24:19.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.083+0000 7f002af97700 1 -- 172.21.15.60:0/3139319374 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002419d890 con 0x7f002406a470 2024-03-04T12:24:19.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.083+0000 7f0021ffb700 1 -- 172.21.15.60:0/3139319374 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f001800cd70 con 0x7f002406a470 2024-03-04T12:24:19.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.083+0000 7f0021ffb700 1 -- 172.21.15.60:0/3139319374 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00180187c0 con 0x7f002406a470 2024-03-04T12:24:19.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.083+0000 7f002af97700 1 -- 172.21.15.60:0/3139319374 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00241a5110 con 0x7f002406a470 2024-03-04T12:24:19.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.084+0000 7f002af97700 1 -- 172.21.15.60:0/3139319374 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f00240625b0 con 0x7f002406a470 2024-03-04T12:24:19.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.086+0000 7f0021ffb700 1 -- 172.21.15.60:0/3139319374 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f0018010430 con 0x7f002406a470 2024-03-04T12:24:19.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.086+0000 7f0021ffb700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f000c0a1f20 0x7f000c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:19.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.086+0000 7f0028d33700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f000c0a1f20 0x7f000c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:19.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.086+0000 7f0021ffb700 1 -- 172.21.15.60:0/3139319374 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f00180cece0 con 0x7f002406a470 2024-03-04T12:24:19.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.086+0000 7f0021ffb700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222] conn(0x7f000c0ad080 0x7f000c0af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:19.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.086+0000 7f0021ffb700 1 -- 172.21.15.60:0/3139319374 --> [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f001801bec0 con 0x7f000c0ad080 2024-03-04T12:24:19.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.086+0000 7f0021ffb700 1 -- 172.21.15.60:0/3139319374 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00180cf130 con 0x7f002406a470 2024-03-04T12:24:19.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.087+0000 7f0028d33700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f000c0a1f20 0x7f000c0a43e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0014006010 tx=0x7f0014005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:19.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.087+0000 7f0029534700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222] conn(0x7f000c0ad080 0x7f000c0af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:19.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.087+0000 7f0029534700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222] conn(0x7f000c0ad080 0x7f000c0af520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:19.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.088+0000 7f0021ffb700 1 -- 172.21.15.60:0/3139319374 <== osd.2 v2:172.21.15.86:6800/3773043222 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f001801bec0 con 0x7f000c0ad080 2024-03-04T12:24:19.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.105+0000 7f002af97700 1 -- 172.21.15.60:0/3139319374 --> [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f00240629b0 con 0x7f000c0ad080 2024-03-04T12:24:19.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.105+0000 7f0021ffb700 1 -- 172.21.15.60:0/3139319374 <== osd.2 v2:172.21.15.86:6800/3773043222 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f00240629b0 con 0x7f000c0ad080 2024-03-04T12:24:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.105+0000 7f002af97700 1 -- 172.21.15.60:0/3139319374 >> [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222] conn(0x7f000c0ad080 msgr2=0x7f000c0af520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.106+0000 7f002af97700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222] conn(0x7f000c0ad080 0x7f000c0af520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.106+0000 7f002af97700 1 -- 172.21.15.60:0/3139319374 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f000c0a1f20 msgr2=0x7f000c0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.106+0000 7f002af97700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f000c0a1f20 0x7f000c0a43e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0014006010 tx=0x7f0014005e80 comp rx=0 tx=0).stop 2024-03-04T12:24:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.106+0000 7f002af97700 1 -- 172.21.15.60:0/3139319374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002406a470 msgr2=0x7f0024198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.106+0000 7f002af97700 1 --2- 172.21.15.60:0/3139319374 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f002406a470 0x7f0024198b20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f001800e9a0 tx=0x7f001800ed60 comp rx=0 tx=0).stop 2024-03-04T12:24:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.106+0000 7f0029534700 1 -- 172.21.15.60:0/3139319374 reap_dead start 2024-03-04T12:24:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.106+0000 7f002af97700 1 -- 172.21.15.60:0/3139319374 shutdown_connections 2024-03-04T12:24:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.106+0000 7f002af97700 1 -- 172.21.15.60:0/3139319374 >> 172.21.15.60:0/3139319374 conn(0x7f0024077c50 msgr2=0x7f00240ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.106+0000 7f002af97700 1 -- 172.21.15.60:0/3139319374 shutdown_connections 2024-03-04T12:24:19.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:19.106+0000 7f002af97700 1 -- 172.21.15.60:0/3139319374 wait complete. 2024-03-04T12:24:19.751 INFO:teuthology.orchestra.run.smithi060.stdout:77309411402 2024-03-04T12:24:19.752 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-04T12:24:20.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:20 smithi086 ceph-mon[36851]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:20.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:20 smithi063 ceph-mon[36717]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:20.371 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:20 smithi060 ceph-mon[32745]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:20.401 INFO:teuthology.orchestra.run.smithi060.stdout:309237645345 2024-03-04T12:24:20.402 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-04T12:24:21.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.956+0000 7fa604a47700 1 -- 172.21.15.60:0/1007755006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa60006aec0 msgr2=0x7fa60010ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:21.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.956+0000 7fa604a47700 1 --2- 172.21.15.60:0/1007755006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa60006aec0 0x7fa60010ece0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800b0d0 tx=0x7fa5f800b3e0 comp rx=0 tx=0).stop 2024-03-04T12:24:21.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.956+0000 7fa604a47700 1 -- 172.21.15.60:0/1007755006 shutdown_connections 2024-03-04T12:24:21.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.956+0000 7fa604a47700 1 --2- 172.21.15.60:0/1007755006 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa60006aec0 0x7fa60010ece0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:21.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.956+0000 7fa604a47700 1 --2- 172.21.15.60:0/1007755006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa600107ea0 0x7fa600108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:21.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.956+0000 7fa604a47700 1 --2- 172.21.15.60:0/1007755006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa60006a520 0x7fa60006a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.956+0000 7fa604a47700 1 -- 172.21.15.60:0/1007755006 >> 172.21.15.60:0/1007755006 conn(0x7fa600076d40 msgr2=0x7fa600077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.956+0000 7fa604a47700 1 -- 172.21.15.60:0/1007755006 shutdown_connections 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.956+0000 7fa604a47700 1 -- 172.21.15.60:0/1007755006 wait complete. 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa604a47700 1 Processor -- start 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa604a47700 1 -- start start 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa604a47700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa60006a520 0x7fa600198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa604a47700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa60006aec0 0x7fa600198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa604a47700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa600107ea0 0x7fa60019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa604a47700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6001120f0 con 0x7fa60006a520 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa604a47700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa600111f70 con 0x7fa600107ea0 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa604a47700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa600112270 con 0x7fa60006aec0 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa5fe59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa60006a520 0x7fa600198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa5fe59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa60006a520 0x7fa600198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36010/0 (socket says 172.21.15.60:36010) 2024-03-04T12:24:21.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa5fe59c700 1 -- 172.21.15.60:0/1006430483 learned_addr learned my addr 172.21.15.60:0/1006430483 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:21.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa5fed9d700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa600107ea0 0x7fa60019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:21.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa5fdd9b700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa60006aec0 0x7fa600198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:21.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa5fe59c700 1 -- 172.21.15.60:0/1006430483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa60006aec0 msgr2=0x7fa600198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:21.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa5fe59c700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa60006aec0 0x7fa600198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:21.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa5fe59c700 1 -- 172.21.15.60:0/1006430483 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa600107ea0 msgr2=0x7fa60019ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:21.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa5fe59c700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa600107ea0 0x7fa60019ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:21.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.957+0000 7fa5fe59c700 1 -- 172.21.15.60:0/1006430483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60019d570 con 0x7fa60006a520 2024-03-04T12:24:21.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.958+0000 7fa5fe59c700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa60006a520 0x7fa600198560 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000e9a0 tx=0x7fa5f000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:21.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.958+0000 7fa5ef7fe700 1 -- 172.21.15.60:0/1006430483 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f000cbd0 con 0x7fa60006a520 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.958+0000 7fa5ef7fe700 1 -- 172.21.15.60:0/1006430483 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5f000cd70 con 0x7fa60006a520 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.958+0000 7fa604a47700 1 -- 172.21.15.60:0/1006430483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60019d860 con 0x7fa60006a520 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.958+0000 7fa5ef7fe700 1 -- 172.21.15.60:0/1006430483 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0018750 con 0x7fa60006a520 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.958+0000 7fa604a47700 1 -- 172.21.15.60:0/1006430483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6001a50e0 con 0x7fa60006a520 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.958+0000 7fa604a47700 1 -- 172.21.15.60:0/1006430483 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa600062570 con 0x7fa60006a520 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.960+0000 7fa5ef7fe700 1 -- 172.21.15.60:0/1006430483 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fa5f0010430 con 0x7fa60006a520 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.960+0000 7fa5ef7fe700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa5e80a1f20 0x7fa5e80a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.961+0000 7fa5fdd9b700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa5e80a1f20 0x7fa5e80a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.961+0000 7fa5ef7fe700 1 -- 172.21.15.60:0/1006430483 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa5f00cecb0 con 0x7fa60006a520 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.961+0000 7fa5ef7fe700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201] conn(0x7fa5e80ad080 0x7fa5e80af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.961+0000 7fa5ef7fe700 1 -- 172.21.15.60:0/1006430483 --> [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5f001ae50 con 0x7fa5e80ad080 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.961+0000 7fa5ef7fe700 1 -- 172.21.15.60:0/1006430483 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5f00cf060 con 0x7fa60006a520 2024-03-04T12:24:21.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.961+0000 7fa5fdd9b700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa5e80a1f20 0x7fa5e80a43e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa600077340 tx=0x7fa5f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.961+0000 7fa5fed9d700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201] conn(0x7fa5e80ad080 0x7fa5e80af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.962+0000 7fa5fed9d700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201] conn(0x7fa5e80ad080 0x7fa5e80af520 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.962+0000 7fa5ef7fe700 1 -- 172.21.15.60:0/1006430483 <== osd.11 v2:172.21.15.86:6824/2397477201 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa5f001ae50 con 0x7fa5e80ad080 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.979+0000 7fa604a47700 1 -- 172.21.15.60:0/1006430483 --> [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa6000629a0 con 0x7fa5e80ad080 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.979+0000 7fa5ef7fe700 1 -- 172.21.15.60:0/1006430483 <== osd.11 v2:172.21.15.86:6824/2397477201 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa6000629a0 con 0x7fa5e80ad080 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.980+0000 7fa604a47700 1 -- 172.21.15.60:0/1006430483 >> [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201] conn(0x7fa5e80ad080 msgr2=0x7fa5e80af520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.980+0000 7fa604a47700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201] conn(0x7fa5e80ad080 0x7fa5e80af520 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.980+0000 7fa604a47700 1 -- 172.21.15.60:0/1006430483 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa5e80a1f20 msgr2=0x7fa5e80a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.980+0000 7fa604a47700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa5e80a1f20 0x7fa5e80a43e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa600077340 tx=0x7fa5f4009260 comp rx=0 tx=0).stop 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.980+0000 7fa604a47700 1 -- 172.21.15.60:0/1006430483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa60006a520 msgr2=0x7fa600198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.980+0000 7fa604a47700 1 --2- 172.21.15.60:0/1006430483 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa60006a520 0x7fa600198560 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000e9a0 tx=0x7fa5f000ecb0 comp rx=0 tx=0).stop 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.980+0000 7fa5fed9d700 1 -- 172.21.15.60:0/1006430483 reap_dead start 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.980+0000 7fa604a47700 1 -- 172.21.15.60:0/1006430483 shutdown_connections 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.980+0000 7fa604a47700 1 -- 172.21.15.60:0/1006430483 >> 172.21.15.60:0/1006430483 conn(0x7fa600076d40 msgr2=0x7fa6000ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:21.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.980+0000 7fa604a47700 1 -- 172.21.15.60:0/1006430483 shutdown_connections 2024-03-04T12:24:21.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:20.980+0000 7fa604a47700 1 -- 172.21.15.60:0/1006430483 wait complete. 2024-03-04T12:24:21.668 INFO:teuthology.orchestra.run.smithi060.stdout:343597383708 2024-03-04T12:24:21.669 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-04T12:24:22.318 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:22 smithi060 ceph-mon[32745]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-04T12:24:22.345 INFO:teuthology.orchestra.run.smithi060.stdout:287762808870 2024-03-04T12:24:22.346 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-04T12:24:22.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:22 smithi086 ceph-mon[36851]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-04T12:24:22.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:22 smithi063 ceph-mon[36717]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-04T12:24:22.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.868+0000 7f805f59e700 1 -- 172.21.15.60:0/140742737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8060105f40 msgr2=0x7f806010db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:22.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.868+0000 7f805f59e700 1 --2- 172.21.15.60:0/140742737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8060105f40 0x7f806010db10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f805800b0d0 tx=0x7f805800b3e0 comp rx=0 tx=0).stop 2024-03-04T12:24:22.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.868+0000 7f805f59e700 1 -- 172.21.15.60:0/140742737 shutdown_connections 2024-03-04T12:24:22.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.868+0000 7f805f59e700 1 --2- 172.21.15.60:0/140742737 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8060105f40 0x7f806010db10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:22.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.868+0000 7f805f59e700 1 --2- 172.21.15.60:0/140742737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8060100e60 0x7f8060101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:22.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.868+0000 7f805f59e700 1 --2- 172.21.15.60:0/140742737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8060101810 0x7f8060105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:22.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.868+0000 7f805f59e700 1 -- 172.21.15.60:0/140742737 >> 172.21.15.60:0/140742737 conn(0x7f80600fc6f0 msgr2=0x7f80600feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:22.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.868+0000 7f805f59e700 1 -- 172.21.15.60:0/140742737 shutdown_connections 2024-03-04T12:24:22.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.868+0000 7f805f59e700 1 -- 172.21.15.60:0/140742737 wait complete. 2024-03-04T12:24:22.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805f59e700 1 Processor -- start 2024-03-04T12:24:22.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805f59e700 1 -- start start 2024-03-04T12:24:22.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805f59e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8060100e60 0x7f80601104c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:22.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805f59e700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8060101810 0x7f8060110a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:22.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805f59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8060105f40 0x7f8060110f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:22.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805f59e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806006a600 con 0x7f8060105f40 2024-03-04T12:24:22.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805f59e700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806006a480 con 0x7f8060101810 2024-03-04T12:24:22.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805f59e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806006a780 con 0x7f8060100e60 2024-03-04T12:24:22.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8060105f40 0x7f8060110f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805ed9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8060105f40 0x7f8060110f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:36026/0 (socket says 172.21.15.60:36026) 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805ed9d700 1 -- 172.21.15.60:0/1200489139 learned_addr learned my addr 172.21.15.60:0/1200489139 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805e59c700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8060100e60 0x7f80601104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.869+0000 7f805dd9b700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8060101810 0x7f8060110a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.870+0000 7f805ed9d700 1 -- 172.21.15.60:0/1200489139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8060100e60 msgr2=0x7f80601104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.870+0000 7f805ed9d700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8060100e60 0x7f80601104c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.870+0000 7f805ed9d700 1 -- 172.21.15.60:0/1200489139 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8060101810 msgr2=0x7f8060110a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.870+0000 7f805ed9d700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8060101810 0x7f8060110a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.870+0000 7f805ed9d700 1 -- 172.21.15.60:0/1200489139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80601a68c0 con 0x7f8060105f40 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.870+0000 7f805dd9b700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8060101810 0x7f8060110a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.870+0000 7f805ed9d700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8060105f40 0x7f8060110f40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8058006070 tx=0x7f8058004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.870+0000 7f804f7fe700 1 -- 172.21.15.60:0/1200489139 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058021070 con 0x7f8060105f40 2024-03-04T12:24:22.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.870+0000 7f804f7fe700 1 -- 172.21.15.60:0/1200489139 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8058007df0 con 0x7f8060105f40 2024-03-04T12:24:22.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.870+0000 7f805f59e700 1 -- 172.21.15.60:0/1200489139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80601a6b50 con 0x7f8060105f40 2024-03-04T12:24:22.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.870+0000 7f804f7fe700 1 -- 172.21.15.60:0/1200489139 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058005470 con 0x7f8060105f40 2024-03-04T12:24:22.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.870+0000 7f805f59e700 1 -- 172.21.15.60:0/1200489139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80601a7050 con 0x7f8060105f40 2024-03-04T12:24:22.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.871+0000 7f805f59e700 1 -- 172.21.15.60:0/1200489139 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f80600fd9a0 con 0x7f8060105f40 2024-03-04T12:24:22.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.872+0000 7f804f7fe700 1 -- 172.21.15.60:0/1200489139 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f8058024070 con 0x7f8060105f40 2024-03-04T12:24:22.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.872+0000 7f804f7fe700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f80480a1f20 0x7f80480a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:22.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.873+0000 7f805e59c700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f80480a1f20 0x7f80480a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:22.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.873+0000 7f804f7fe700 1 -- 172.21.15.60:0/1200489139 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f805800f020 con 0x7f8060105f40 2024-03-04T12:24:22.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.873+0000 7f804f7fe700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350] conn(0x7f80480ad080 0x7f80480af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:22.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.873+0000 7f804f7fe700 1 -- 172.21.15.60:0/1200489139 --> [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f80480afbf0 con 0x7f80480ad080 2024-03-04T12:24:22.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.873+0000 7f804f7fe700 1 -- 172.21.15.60:0/1200489139 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80580cffe0 con 0x7f8060105f40 2024-03-04T12:24:22.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.873+0000 7f805e59c700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f80480a1f20 0x7f80480a43e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8054005fd0 tx=0x7f805400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:22.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.873+0000 7f805dd9b700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350] conn(0x7f80480ad080 0x7f80480af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:22.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.874+0000 7f805dd9b700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350] conn(0x7f80480ad080 0x7f80480af520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:22.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.874+0000 7f804f7fe700 1 -- 172.21.15.60:0/1200489139 <== osd.1 v2:172.21.15.63:6800/1936533350 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f80480afbf0 con 0x7f80480ad080 2024-03-04T12:24:22.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.891+0000 7f805f59e700 1 -- 172.21.15.60:0/1200489139 --> [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8060102340 con 0x7f80480ad080 2024-03-04T12:24:22.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.891+0000 7f804f7fe700 1 -- 172.21.15.60:0/1200489139 <== osd.1 v2:172.21.15.63:6800/1936533350 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8060102340 con 0x7f80480ad080 2024-03-04T12:24:22.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.892+0000 7f805f59e700 1 -- 172.21.15.60:0/1200489139 >> [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350] conn(0x7f80480ad080 msgr2=0x7f80480af520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:22.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.892+0000 7f805f59e700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350] conn(0x7f80480ad080 0x7f80480af520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:22.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.892+0000 7f805f59e700 1 -- 172.21.15.60:0/1200489139 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f80480a1f20 msgr2=0x7f80480a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:22.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.892+0000 7f805f59e700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f80480a1f20 0x7f80480a43e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8054005fd0 tx=0x7f805400a560 comp rx=0 tx=0).stop 2024-03-04T12:24:22.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.892+0000 7f805f59e700 1 -- 172.21.15.60:0/1200489139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8060105f40 msgr2=0x7f8060110f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:22.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.892+0000 7f805f59e700 1 --2- 172.21.15.60:0/1200489139 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8060105f40 0x7f8060110f40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8058006070 tx=0x7f8058004ef0 comp rx=0 tx=0).stop 2024-03-04T12:24:22.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.892+0000 7f805ed9d700 1 -- 172.21.15.60:0/1200489139 reap_dead start 2024-03-04T12:24:22.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.892+0000 7f805f59e700 1 -- 172.21.15.60:0/1200489139 shutdown_connections 2024-03-04T12:24:22.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.892+0000 7f805f59e700 1 -- 172.21.15.60:0/1200489139 >> 172.21.15.60:0/1200489139 conn(0x7f80600fc6f0 msgr2=0x7f80600fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:22.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.892+0000 7f805f59e700 1 -- 172.21.15.60:0/1200489139 shutdown_connections 2024-03-04T12:24:22.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:22.892+0000 7f805f59e700 1 -- 172.21.15.60:0/1200489139 wait complete. 2024-03-04T12:24:23.562 INFO:teuthology.orchestra.run.smithi060.stdout:197568495666 2024-03-04T12:24:23.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-04T12:24:24.003 INFO:teuthology.orchestra.run.smithi060.stdout:55834574927 2024-03-04T12:24:24.004 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-04T12:24:24.038 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:24.038 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:24.060 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:24.061 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:24.070 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:24.091 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:24.165 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:24 smithi060 ceph-mon[32745]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-04T12:24:24.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:24 smithi086 ceph-mon[36851]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-04T12:24:24.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:24 smithi063 ceph-mon[36717]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-04T12:24:25.591 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:25.592 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:26.036 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:26.038 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:26.060 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:26.063 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:26 smithi060 ceph-mon[32745]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:26.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:26 smithi086 ceph-mon[36851]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:26.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:26 smithi063 ceph-mon[36717]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:28.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:28 smithi086 ceph-mon[36851]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:28.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:28 smithi063 ceph-mon[36717]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:28.352 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:28 smithi060 ceph-mon[32745]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.961+0000 7f4691b9f700 1 -- 172.21.15.60:0/1655872016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c101f50 msgr2=0x7f468c102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.961+0000 7f4691b9f700 1 --2- 172.21.15.60:0/1655872016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c101f50 0x7f468c102330 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f46800099c0 tx=0x7f4680009cd0 comp rx=0 tx=0).stop 2024-03-04T12:24:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.961+0000 7f4691b9f700 1 -- 172.21.15.60:0/1655872016 shutdown_connections 2024-03-04T12:24:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.961+0000 7f4691b9f700 1 --2- 172.21.15.60:0/1655872016 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f468c107030 0x7f468c10ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.961+0000 7f4691b9f700 1 --2- 172.21.15.60:0/1655872016 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c101f50 0x7f468c102330 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.961+0000 7f4691b9f700 1 --2- 172.21.15.60:0/1655872016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f468c102900 0x7f468c106950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.961+0000 7f4691b9f700 1 -- 172.21.15.60:0/1655872016 >> 172.21.15.60:0/1655872016 conn(0x7f468c0fd840 msgr2=0x7f468c0ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.961+0000 7f4691b9f700 1 -- 172.21.15.60:0/1655872016 shutdown_connections 2024-03-04T12:24:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.961+0000 7f4691b9f700 1 -- 172.21.15.60:0/1655872016 wait complete. 2024-03-04T12:24:29.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f4691b9f700 1 Processor -- start 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f4691b9f700 1 -- start start 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f4691b9f700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f468c101f50 0x7f468c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f4691b9f700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f468c102900 0x7f468c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f4691b9f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c107030 0x7f468c19cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f4691b9f700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c112020 con 0x7f468c107030 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f4691b9f700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c111ea0 con 0x7f468c101f50 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f4691b9f700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c1121a0 con 0x7f468c102900 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f469139e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c107030 0x7f468c19cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f469139e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c107030 0x7f468c19cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37232/0 (socket says 172.21.15.60:37232) 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f469139e700 1 -- 172.21.15.60:0/452603088 learned_addr learned my addr 172.21.15.60:0/452603088 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f468bfff700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f468c102900 0x7f468c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f4690b9d700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f468c101f50 0x7f468c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f469139e700 1 -- 172.21.15.60:0/452603088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f468c102900 msgr2=0x7f468c1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:29.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f469139e700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f468c102900 0x7f468c1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f469139e700 1 -- 172.21.15.60:0/452603088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f468c101f50 msgr2=0x7f468c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f469139e700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f468c101f50 0x7f468c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f469139e700 1 -- 172.21.15.60:0/452603088 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c19d450 con 0x7f468c107030 2024-03-04T12:24:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.962+0000 7f4690b9d700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f468c101f50 0x7f468c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:24:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.963+0000 7f469139e700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c107030 0x7f468c19cd10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f468400ada0 tx=0x7f468400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.963+0000 7f4689ffb700 1 -- 172.21.15.60:0/452603088 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684017020 con 0x7f468c107030 2024-03-04T12:24:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.963+0000 7f4691b9f700 1 -- 172.21.15.60:0/452603088 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468c19d740 con 0x7f468c107030 2024-03-04T12:24:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.963+0000 7f4689ffb700 1 -- 172.21.15.60:0/452603088 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f468400a0d0 con 0x7f468c107030 2024-03-04T12:24:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.963+0000 7f4689ffb700 1 -- 172.21.15.60:0/452603088 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684005480 con 0x7f468c107030 2024-03-04T12:24:29.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.963+0000 7f4691b9f700 1 -- 172.21.15.60:0/452603088 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468c1a4fc0 con 0x7f468c107030 2024-03-04T12:24:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.964+0000 7f4691b9f700 1 -- 172.21.15.60:0/452603088 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468c068ae0 con 0x7f468c107030 2024-03-04T12:24:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.965+0000 7f4689ffb700 1 -- 172.21.15.60:0/452603088 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f4684005620 con 0x7f468c107030 2024-03-04T12:24:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.966+0000 7f4689ffb700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f467c0a1ea0 0x7f467c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.966+0000 7f4690b9d700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f467c0a1ea0 0x7f467c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.966+0000 7f4689ffb700 1 -- 172.21.15.60:0/452603088 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f46840cd970 con 0x7f468c107030 2024-03-04T12:24:29.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.966+0000 7f4690b9d700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f467c0a1ea0 0x7f467c0a4360 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4680005b40 tx=0x7f4680005ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:29.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:29.970+0000 7f4689ffb700 1 -- 172.21.15.60:0/452603088 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4684021020 con 0x7f468c107030 2024-03-04T12:24:30.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.182+0000 7f4691b9f700 1 -- 172.21.15.60:0/452603088 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f468c02d870 con 0x7f468c107030 2024-03-04T12:24:30.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.183+0000 7f4689ffb700 1 -- 172.21.15.60:0/452603088 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f468409a970 con 0x7f468c107030 2024-03-04T12:24:30.189 INFO:teuthology.orchestra.run.smithi060.stdout:107374182471 2024-03-04T12:24:30.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.185+0000 7f4691b9f700 1 -- 172.21.15.60:0/452603088 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f467c0a1ea0 msgr2=0x7f467c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:30.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.185+0000 7f4691b9f700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f467c0a1ea0 0x7f467c0a4360 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4680005b40 tx=0x7f4680005ab0 comp rx=0 tx=0).stop 2024-03-04T12:24:30.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.185+0000 7f4691b9f700 1 -- 172.21.15.60:0/452603088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c107030 msgr2=0x7f468c19cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:30.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.185+0000 7f4691b9f700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c107030 0x7f468c19cd10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f468400ada0 tx=0x7f468400c330 comp rx=0 tx=0).stop 2024-03-04T12:24:30.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.185+0000 7f4691b9f700 1 -- 172.21.15.60:0/452603088 shutdown_connections 2024-03-04T12:24:30.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.185+0000 7f4691b9f700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f467c0a1ea0 0x7f467c0a4360 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:30.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.185+0000 7f4691b9f700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f468c107030 0x7f468c19cd10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:30.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.185+0000 7f4691b9f700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f468c101f50 0x7f468c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:30.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.185+0000 7f4691b9f700 1 --2- 172.21.15.60:0/452603088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f468c102900 0x7f468c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:30.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.185+0000 7f4691b9f700 1 -- 172.21.15.60:0/452603088 >> 172.21.15.60:0/452603088 conn(0x7f468c0fd840 msgr2=0x7f468c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:30.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.185+0000 7f4691b9f700 1 -- 172.21.15.60:0/452603088 shutdown_connections 2024-03-04T12:24:30.192 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:30.185+0000 7f4691b9f700 1 -- 172.21.15.60:0/452603088 wait complete. 2024-03-04T12:24:30.318 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:30 smithi060 ceph-mon[32745]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:24:30.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:30 smithi086 ceph-mon[36851]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:24:30.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:30 smithi063 ceph-mon[36717]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:24:31.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:31 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/452603088' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T12:24:31.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:31 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/452603088' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T12:24:31.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:31 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/452603088' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T12:24:31.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.594+0000 7f6a89f05700 1 -- 172.21.15.60:0/3653327875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a84101960 msgr2=0x7f6a84105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:31.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.594+0000 7f6a89f05700 1 --2- 172.21.15.60:0/3653327875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a84101960 0x7f6a84105960 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009990 tx=0x7f6a78009ca0 comp rx=0 tx=0).stop 2024-03-04T12:24:31.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.594+0000 7f6a89f05700 1 -- 172.21.15.60:0/3653327875 shutdown_connections 2024-03-04T12:24:31.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.594+0000 7f6a89f05700 1 --2- 172.21.15.60:0/3653327875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a84106040 0x7f6a8410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:31.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.594+0000 7f6a89f05700 1 --2- 172.21.15.60:0/3653327875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a84100fb0 0x7f6a84101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:31.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.594+0000 7f6a89f05700 1 --2- 172.21.15.60:0/3653327875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a84101960 0x7f6a84105960 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:31.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.594+0000 7f6a89f05700 1 -- 172.21.15.60:0/3653327875 >> 172.21.15.60:0/3653327875 conn(0x7f6a84076d40 msgr2=0x7f6a84077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:31.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.594+0000 7f6a89f05700 1 -- 172.21.15.60:0/3653327875 shutdown_connections 2024-03-04T12:24:31.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.594+0000 7f6a89f05700 1 -- 172.21.15.60:0/3653327875 wait complete. 2024-03-04T12:24:31.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a89f05700 1 Processor -- start 2024-03-04T12:24:31.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a89f05700 1 -- start start 2024-03-04T12:24:31.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a89f05700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a84100fb0 0x7f6a84199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:31.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a89f05700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a84101960 0x7f6a841999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:31.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a89f05700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a84106040 0x7f6a841a44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:31.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a89f05700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a84110fe0 con 0x7f6a84101960 2024-03-04T12:24:31.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a89f05700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a84110e60 con 0x7f6a84100fb0 2024-03-04T12:24:31.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a89f05700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a84111160 con 0x7f6a84106040 2024-03-04T12:24:31.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a82ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a84101960 0x7f6a841999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:31.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a82ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a84101960 0x7f6a841999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:37246/0 (socket says 172.21.15.60:37246) 2024-03-04T12:24:31.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a82ffd700 1 -- 172.21.15.60:0/996821260 learned_addr learned my addr 172.21.15.60:0/996821260 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:31.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a837fe700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a84100fb0 0x7f6a84199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:31.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.595+0000 7f6a83fff700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a84106040 0x7f6a841a44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:31.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.596+0000 7f6a82ffd700 1 -- 172.21.15.60:0/996821260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a84106040 msgr2=0x7f6a841a44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:31.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.596+0000 7f6a82ffd700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a84106040 0x7f6a841a44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:31.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.596+0000 7f6a82ffd700 1 -- 172.21.15.60:0/996821260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a84100fb0 msgr2=0x7f6a84199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:31.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.596+0000 7f6a82ffd700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a84100fb0 0x7f6a84199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:31.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.596+0000 7f6a82ffd700 1 -- 172.21.15.60:0/996821260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a841a4a20 con 0x7f6a84101960 2024-03-04T12:24:31.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.596+0000 7f6a82ffd700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a84101960 0x7f6a841999a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6a78000c00 tx=0x7f6a7800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:31.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.596+0000 7f6a80ff9700 1 -- 172.21.15.60:0/996821260 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a78027070 con 0x7f6a84101960 2024-03-04T12:24:31.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.596+0000 7f6a80ff9700 1 -- 172.21.15.60:0/996821260 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a78003d70 con 0x7f6a84101960 2024-03-04T12:24:31.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.596+0000 7f6a89f05700 1 -- 172.21.15.60:0/996821260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a841a4bf0 con 0x7f6a84101960 2024-03-04T12:24:31.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.596+0000 7f6a89f05700 1 -- 172.21.15.60:0/996821260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a841a5040 con 0x7f6a84101960 2024-03-04T12:24:31.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.597+0000 7f6a80ff9700 1 -- 172.21.15.60:0/996821260 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7800f6c0 con 0x7f6a84101960 2024-03-04T12:24:31.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.597+0000 7f6a89f05700 1 -- 172.21.15.60:0/996821260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a84102440 con 0x7f6a84101960 2024-03-04T12:24:31.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.598+0000 7f6a80ff9700 1 -- 172.21.15.60:0/996821260 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f6a7802a070 con 0x7f6a84101960 2024-03-04T12:24:31.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.599+0000 7f6a80ff9700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6a6c0a1ea0 0x7f6a6c0a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:31.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.599+0000 7f6a837fe700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6a6c0a1ea0 0x7f6a6c0a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:31.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.599+0000 7f6a80ff9700 1 -- 172.21.15.60:0/996821260 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f6a780cf2c0 con 0x7f6a84101960 2024-03-04T12:24:31.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.599+0000 7f6a837fe700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6a6c0a1ea0 0x7f6a6c0a4360 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009c60 tx=0x7f6a74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:31.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.603+0000 7f6a80ff9700 1 -- 172.21.15.60:0/996821260 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6a7805dd20 con 0x7f6a84101960 2024-03-04T12:24:31.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.809+0000 7f6a89f05700 1 -- 172.21.15.60:0/996821260 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f6a8402d870 con 0x7f6a84101960 2024-03-04T12:24:31.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.809+0000 7f6a80ff9700 1 -- 172.21.15.60:0/996821260 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f6a78013070 con 0x7f6a84101960 2024-03-04T12:24:31.816 INFO:teuthology.orchestra.run.smithi060.stdout:309237645347 2024-03-04T12:24:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.812+0000 7f6a89f05700 1 -- 172.21.15.60:0/996821260 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6a6c0a1ea0 msgr2=0x7f6a6c0a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.812+0000 7f6a89f05700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6a6c0a1ea0 0x7f6a6c0a4360 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009c60 tx=0x7f6a74009260 comp rx=0 tx=0).stop 2024-03-04T12:24:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.812+0000 7f6a89f05700 1 -- 172.21.15.60:0/996821260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a84101960 msgr2=0x7f6a841999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.812+0000 7f6a89f05700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a84101960 0x7f6a841999a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6a78000c00 tx=0x7f6a7800bab0 comp rx=0 tx=0).stop 2024-03-04T12:24:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.812+0000 7f6a89f05700 1 -- 172.21.15.60:0/996821260 shutdown_connections 2024-03-04T12:24:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.812+0000 7f6a89f05700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a84106040 0x7f6a841a44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.812+0000 7f6a89f05700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a84100fb0 0x7f6a84199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.812+0000 7f6a89f05700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6a6c0a1ea0 0x7f6a6c0a4360 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.812+0000 7f6a89f05700 1 --2- 172.21.15.60:0/996821260 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6a84101960 0x7f6a841999a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.812+0000 7f6a89f05700 1 -- 172.21.15.60:0/996821260 >> 172.21.15.60:0/996821260 conn(0x7f6a84076d40 msgr2=0x7f6a841071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:31.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.812+0000 7f6a89f05700 1 -- 172.21.15.60:0/996821260 shutdown_connections 2024-03-04T12:24:31.819 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:31.812+0000 7f6a89f05700 1 -- 172.21.15.60:0/996821260 wait complete. 2024-03-04T12:24:32.237 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:32 smithi060 ceph-mon[32745]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-04T12:24:32.237 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:32 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/996821260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-04T12:24:32.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:32 smithi086 ceph-mon[36851]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-04T12:24:32.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:32 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/996821260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-04T12:24:32.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:32 smithi063 ceph-mon[36717]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-04T12:24:32.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:32 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/996821260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-04T12:24:34.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:34 smithi086 ceph-mon[36851]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:24:34.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:34 smithi063 ceph-mon[36717]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:24:34.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:34 smithi060 ceph-mon[32745]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:24:35.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.510+0000 7f99438f4700 1 -- 172.21.15.60:0/1130228706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f993c102830 msgr2=0x7f993c10ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:35.516 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.510+0000 7f99438f4700 1 --2- 172.21.15.60:0/1130228706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f993c102830 0x7f993c10ec70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f992c00b0d0 tx=0x7f992c00b3e0 comp rx=0 tx=0).stop 2024-03-04T12:24:35.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.510+0000 7f99438f4700 1 -- 172.21.15.60:0/1130228706 shutdown_connections 2024-03-04T12:24:35.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.510+0000 7f99438f4700 1 --2- 172.21.15.60:0/1130228706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f993c102830 0x7f993c10ec70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:35.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.510+0000 7f99438f4700 1 --2- 172.21.15.60:0/1130228706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f993c107e50 0x7f993c108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:35.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.510+0000 7f99438f4700 1 --2- 172.21.15.60:0/1130228706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f993c101e70 0x7f993c1022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:35.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.510+0000 7f99438f4700 1 -- 172.21.15.60:0/1130228706 >> 172.21.15.60:0/1130228706 conn(0x7f993c078f50 msgr2=0x7f993c079360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:35.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.510+0000 7f99438f4700 1 -- 172.21.15.60:0/1130228706 shutdown_connections 2024-03-04T12:24:35.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.510+0000 7f99438f4700 1 -- 172.21.15.60:0/1130228706 wait complete. 2024-03-04T12:24:35.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.511+0000 7f99438f4700 1 Processor -- start 2024-03-04T12:24:35.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.511+0000 7f99438f4700 1 -- start start 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.511+0000 7f99438f4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f993c101e70 0x7f993c198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.511+0000 7f99438f4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f993c102830 0x7f993c198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.511+0000 7f99438f4700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f993c107e50 0x7f993c19cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.511+0000 7f99438f4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993c112150 con 0x7f993c102830 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.511+0000 7f99438f4700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993c111fd0 con 0x7f993c107e50 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.511+0000 7f99438f4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993c1122d0 con 0x7f993c101e70 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.511+0000 7f9940e8f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f993c102830 0x7f993c198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.511+0000 7f9940e8f700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f993c102830 0x7f993c198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47196/0 (socket says 172.21.15.60:47196) 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.511+0000 7f9940e8f700 1 -- 172.21.15.60:0/1564090294 learned_addr learned my addr 172.21.15.60:0/1564090294 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.511+0000 7f9941e91700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f993c107e50 0x7f993c19cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.512+0000 7f9941690700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f993c101e70 0x7f993c198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.512+0000 7f9940e8f700 1 -- 172.21.15.60:0/1564090294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f993c101e70 msgr2=0x7f993c198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.512+0000 7f9940e8f700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f993c101e70 0x7f993c198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.512+0000 7f9940e8f700 1 -- 172.21.15.60:0/1564090294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f993c107e50 msgr2=0x7f993c19cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.512+0000 7f9940e8f700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f993c107e50 0x7f993c19cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:35.519 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.512+0000 7f9940e8f700 1 -- 172.21.15.60:0/1564090294 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c19d510 con 0x7f993c102830 2024-03-04T12:24:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.512+0000 7f9940e8f700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f993c102830 0x7f993c198a40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f993000d950 tx=0x7f993000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.512+0000 7f993a7fc700 1 -- 172.21.15.60:0/1564090294 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9930016070 con 0x7f993c102830 2024-03-04T12:24:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.512+0000 7f99438f4700 1 -- 172.21.15.60:0/1564090294 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f993c19d800 con 0x7f993c102830 2024-03-04T12:24:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.512+0000 7f993a7fc700 1 -- 172.21.15.60:0/1564090294 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9930009e20 con 0x7f993c102830 2024-03-04T12:24:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.512+0000 7f993a7fc700 1 -- 172.21.15.60:0/1564090294 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993000f620 con 0x7f993c102830 2024-03-04T12:24:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.512+0000 7f99438f4700 1 -- 172.21.15.60:0/1564090294 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f993c1a5080 con 0x7f993c102830 2024-03-04T12:24:35.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.513+0000 7f99438f4700 1 -- 172.21.15.60:0/1564090294 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993c068ae0 con 0x7f993c102830 2024-03-04T12:24:35.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.514+0000 7f993a7fc700 1 -- 172.21.15.60:0/1564090294 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f9930019070 con 0x7f993c102830 2024-03-04T12:24:35.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.515+0000 7f993a7fc700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99240a1ea0 0x7f99240a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:35.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.515+0000 7f9941690700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99240a1ea0 0x7f99240a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:35.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.515+0000 7f993a7fc700 1 -- 172.21.15.60:0/1564090294 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f99300cded0 con 0x7f993c102830 2024-03-04T12:24:35.521 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.515+0000 7f9941690700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99240a1ea0 0x7f99240a4360 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9934006fd0 tx=0x7f9934008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:35.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.518+0000 7f993a7fc700 1 -- 172.21.15.60:0/1564090294 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f993009ade0 con 0x7f993c102830 2024-03-04T12:24:35.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.724+0000 7f99438f4700 1 -- 172.21.15.60:0/1564090294 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f993c02d870 con 0x7f993c102830 2024-03-04T12:24:35.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.725+0000 7f993a7fc700 1 -- 172.21.15.60:0/1564090294 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f993009a6e0 con 0x7f993c102830 2024-03-04T12:24:35.731 INFO:teuthology.orchestra.run.smithi060.stdout:55834574930 2024-03-04T12:24:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.727+0000 7f99438f4700 1 -- 172.21.15.60:0/1564090294 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99240a1ea0 msgr2=0x7f99240a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.727+0000 7f99438f4700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99240a1ea0 0x7f99240a4360 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9934006fd0 tx=0x7f9934008040 comp rx=0 tx=0).stop 2024-03-04T12:24:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.727+0000 7f99438f4700 1 -- 172.21.15.60:0/1564090294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f993c102830 msgr2=0x7f993c198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.727+0000 7f99438f4700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f993c102830 0x7f993c198a40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f993000d950 tx=0x7f993000dc60 comp rx=0 tx=0).stop 2024-03-04T12:24:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.727+0000 7f99438f4700 1 -- 172.21.15.60:0/1564090294 shutdown_connections 2024-03-04T12:24:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.727+0000 7f99438f4700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f993c107e50 0x7f993c19cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.727+0000 7f99438f4700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f993c101e70 0x7f993c198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.727+0000 7f99438f4700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f99240a1ea0 0x7f99240a4360 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:35.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.727+0000 7f99438f4700 1 --2- 172.21.15.60:0/1564090294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f993c102830 0x7f993c198a40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:35.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.727+0000 7f99438f4700 1 -- 172.21.15.60:0/1564090294 >> 172.21.15.60:0/1564090294 conn(0x7f993c078f50 msgr2=0x7f993c1009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:35.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.727+0000 7f99438f4700 1 -- 172.21.15.60:0/1564090294 shutdown_connections 2024-03-04T12:24:35.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:35.727+0000 7f99438f4700 1 -- 172.21.15.60:0/1564090294 wait complete. 2024-03-04T12:24:36.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:36 smithi086 ceph-mon[36851]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:36.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:36 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1564090294' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T12:24:36.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:36 smithi063 ceph-mon[36717]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:36.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:36 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1564090294' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T12:24:36.379 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:36 smithi060 ceph-mon[32745]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-04T12:24:36.379 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:36 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1564090294' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T12:24:36.919 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182467 got 107374182471 for osd.3 2024-03-04T12:24:36.919 DEBUG:teuthology.parallel:result is None 2024-03-04T12:24:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.553+0000 7f81abbca700 1 -- 172.21.15.60:0/1328567311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81a4107ef0 msgr2=0x7f81a41082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.553+0000 7f81abbca700 1 --2- 172.21.15.60:0/1328567311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81a4107ef0 0x7f81a41082d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f81a0009990 tx=0x7f81a0009ca0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.553+0000 7f81abbca700 1 -- 172.21.15.60:0/1328567311 shutdown_connections 2024-03-04T12:24:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.553+0000 7f81abbca700 1 --2- 172.21.15.60:0/1328567311 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81a4102860 0x7f81a410ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.553+0000 7f81abbca700 1 --2- 172.21.15.60:0/1328567311 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81a4107ef0 0x7f81a41082d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.553+0000 7f81abbca700 1 --2- 172.21.15.60:0/1328567311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a4101ea0 0x7f81a4102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.553+0000 7f81abbca700 1 -- 172.21.15.60:0/1328567311 >> 172.21.15.60:0/1328567311 conn(0x7f81a4078f70 msgr2=0x7f81a4079380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.553+0000 7f81abbca700 1 -- 172.21.15.60:0/1328567311 shutdown_connections 2024-03-04T12:24:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.553+0000 7f81abbca700 1 -- 172.21.15.60:0/1328567311 wait complete. 2024-03-04T12:24:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.554+0000 7f81abbca700 1 Processor -- start 2024-03-04T12:24:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.554+0000 7f81abbca700 1 -- start start 2024-03-04T12:24:37.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.554+0000 7f81abbca700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81a4101ea0 0x7f81a4198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.554+0000 7f81abbca700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a4102860 0x7f81a4198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.554+0000 7f81abbca700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81a4107ef0 0x7f81a419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.554+0000 7f81abbca700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a4112190 con 0x7f81a4101ea0 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.554+0000 7f81abbca700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a4112010 con 0x7f81a4107ef0 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.554+0000 7f81abbca700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a4112310 con 0x7f81a4102860 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.554+0000 7f81a9966700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81a4101ea0 0x7f81a4198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.554+0000 7f81a9966700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81a4101ea0 0x7f81a4198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47204/0 (socket says 172.21.15.60:47204) 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.554+0000 7f81a9966700 1 -- 172.21.15.60:0/1234230303 learned_addr learned my addr 172.21.15.60:0/1234230303 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f81a9966700 1 -- 172.21.15.60:0/1234230303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a4102860 msgr2=0x7f81a4198b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f81aa167700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81a4107ef0 0x7f81a419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f81a9165700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a4102860 0x7f81a4198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f81a9966700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a4102860 0x7f81a4198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f81a9966700 1 -- 172.21.15.60:0/1234230303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81a4107ef0 msgr2=0x7f81a419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:37.283 INFO:teuthology.orchestra.run.smithi060.stdout:141733920835 2024-03-04T12:24:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f81a9966700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81a4107ef0 0x7f81a419ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f81a9966700 1 -- 172.21.15.60:0/1234230303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a419d610 con 0x7f81a4101ea0 2024-03-04T12:24:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f81a9966700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81a4101ea0 0x7f81a4198600 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f81a0000c00 tx=0x7f81a0003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f819affd700 1 -- 172.21.15.60:0/1234230303 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a0027070 con 0x7f81a4101ea0 2024-03-04T12:24:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f81abbca700 1 -- 172.21.15.60:0/1234230303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a419d8a0 con 0x7f81a4101ea0 2024-03-04T12:24:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f819affd700 1 -- 172.21.15.60:0/1234230303 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81a0004190 con 0x7f81a4101ea0 2024-03-04T12:24:37.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f819affd700 1 -- 172.21.15.60:0/1234230303 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a000f460 con 0x7f81a4101ea0 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.555+0000 7f81abbca700 1 -- 172.21.15.60:0/1234230303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a41a5180 con 0x7f81a4101ea0 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.556+0000 7f81abbca700 1 -- 172.21.15.60:0/1234230303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a410a3b0 con 0x7f81a4101ea0 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.557+0000 7f819affd700 1 -- 172.21.15.60:0/1234230303 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f81a002a070 con 0x7f81a4101ea0 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.558+0000 7f819affd700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f81900a1ea0 0x7f81900a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.558+0000 7f81a9165700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f81900a1ea0 0x7f81900a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.558+0000 7f819affd700 1 -- 172.21.15.60:0/1234230303 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f81a00cf150 con 0x7f81a4101ea0 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.558+0000 7f81a9165700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f81900a1ea0 0x7f81900a4360 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8194006fd0 tx=0x7f8194008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.562+0000 7f819affd700 1 -- 172.21.15.60:0/1234230303 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f81a0013070 con 0x7f81a4101ea0 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.768+0000 7f81abbca700 1 -- 172.21.15.60:0/1234230303 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f81a4199960 con 0x7f81a4101ea0 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.768+0000 7f819affd700 1 -- 172.21.15.60:0/1234230303 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f81a009c0f0 con 0x7f81a4101ea0 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.770+0000 7f81abbca700 1 -- 172.21.15.60:0/1234230303 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f81900a1ea0 msgr2=0x7f81900a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.770+0000 7f81abbca700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f81900a1ea0 0x7f81900a4360 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8194006fd0 tx=0x7f8194008040 comp rx=0 tx=0).stop 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.770+0000 7f81abbca700 1 -- 172.21.15.60:0/1234230303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81a4101ea0 msgr2=0x7f81a4198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:37.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.770+0000 7f81abbca700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81a4101ea0 0x7f81a4198600 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f81a0000c00 tx=0x7f81a0003730 comp rx=0 tx=0).stop 2024-03-04T12:24:37.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.770+0000 7f81abbca700 1 -- 172.21.15.60:0/1234230303 shutdown_connections 2024-03-04T12:24:37.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.770+0000 7f81abbca700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f81a4107ef0 0x7f81a419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.770+0000 7f81abbca700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f81900a1ea0 0x7f81900a4360 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.770+0000 7f81abbca700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f81a4101ea0 0x7f81a4198600 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.770+0000 7f81abbca700 1 --2- 172.21.15.60:0/1234230303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a4102860 0x7f81a4198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.770+0000 7f81abbca700 1 -- 172.21.15.60:0/1234230303 >> 172.21.15.60:0/1234230303 conn(0x7f81a4078f70 msgr2=0x7f81a4100b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:37.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.770+0000 7f81abbca700 1 -- 172.21.15.60:0/1234230303 shutdown_connections 2024-03-04T12:24:37.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:36.770+0000 7f81abbca700 1 -- 172.21.15.60:0/1234230303 wait complete. 2024-03-04T12:24:37.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:37 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1234230303' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T12:24:37.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:37 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1234230303' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T12:24:37.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:37 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1234230303' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T12:24:37.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.075+0000 7fe517743700 1 -- 172.21.15.60:0/1910036608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe510102840 msgr2=0x7fe51010ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:37.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.075+0000 7fe517743700 1 --2- 172.21.15.60:0/1910036608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe510102840 0x7fe51010ecf0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe50800b0a0 tx=0x7fe50800b3b0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.075+0000 7fe517743700 1 -- 172.21.15.60:0/1910036608 shutdown_connections 2024-03-04T12:24:37.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.075+0000 7fe517743700 1 --2- 172.21.15.60:0/1910036608 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe510102840 0x7fe51010ecf0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.075+0000 7fe517743700 1 --2- 172.21.15.60:0/1910036608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe510107eb0 0x7fe510108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.075+0000 7fe517743700 1 --2- 172.21.15.60:0/1910036608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe510101ea0 0x7fe510102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.075+0000 7fe517743700 1 -- 172.21.15.60:0/1910036608 >> 172.21.15.60:0/1910036608 conn(0x7fe510078ef0 msgr2=0x7fe510079300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.075+0000 7fe517743700 1 -- 172.21.15.60:0/1910036608 shutdown_connections 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.075+0000 7fe517743700 1 -- 172.21.15.60:0/1910036608 wait complete. 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.076+0000 7fe517743700 1 Processor -- start 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.076+0000 7fe517743700 1 -- start start 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.076+0000 7fe517743700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe510101ea0 0x7fe510198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.076+0000 7fe517743700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe510102840 0x7fe510198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.076+0000 7fe517743700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe510107eb0 0x7fe51019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe517743700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe510112100 con 0x7fe510101ea0 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe517743700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe510111f80 con 0x7fe510102840 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe517743700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe510112280 con 0x7fe510107eb0 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe5154df700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe510101ea0 0x7fe510198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe514cde700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe510102840 0x7fe510198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe514cde700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe510102840 0x7fe510198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:47888/0 (socket says 172.21.15.60:47888) 2024-03-04T12:24:37.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe514cde700 1 -- 172.21.15.60:0/2797980347 learned_addr learned my addr 172.21.15.60:0/2797980347 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:37.610 INFO:teuthology.orchestra.run.smithi060.stdout:197568495671 2024-03-04T12:24:37.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe5154df700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe510101ea0 0x7fe510198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47226/0 (socket says 172.21.15.60:47226) 2024-03-04T12:24:37.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe515ce0700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe510107eb0 0x7fe51019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:37.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe5154df700 1 -- 172.21.15.60:0/2797980347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe510107eb0 msgr2=0x7fe51019ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:37.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe5154df700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe510107eb0 0x7fe51019ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe5154df700 1 -- 172.21.15.60:0/2797980347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe510102840 msgr2=0x7fe510198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe5154df700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe510102840 0x7fe510198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe5154df700 1 -- 172.21.15.60:0/2797980347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51019d540 con 0x7fe510101ea0 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe514cde700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe510102840 0x7fe510198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.077+0000 7fe5154df700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe510101ea0 0x7fe510198530 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe50c007ae0 tx=0x7fe50c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.079+0000 7fe5067fc700 1 -- 172.21.15.60:0/2797980347 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c01b020 con 0x7fe510101ea0 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.079+0000 7fe5067fc700 1 -- 172.21.15.60:0/2797980347 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe50c004d10 con 0x7fe510101ea0 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.079+0000 7fe5067fc700 1 -- 172.21.15.60:0/2797980347 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c0056e0 con 0x7fe510101ea0 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.079+0000 7fe517743700 1 -- 172.21.15.60:0/2797980347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51019d830 con 0x7fe510101ea0 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.079+0000 7fe517743700 1 -- 172.21.15.60:0/2797980347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe510074050 con 0x7fe510101ea0 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.080+0000 7fe517743700 1 -- 172.21.15.60:0/2797980347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe510068ae0 con 0x7fe510101ea0 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.081+0000 7fe5067fc700 1 -- 172.21.15.60:0/2797980347 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fe50c004750 con 0x7fe510101ea0 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.082+0000 7fe5067fc700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe4fc0a1f20 0x7fe4fc0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.082+0000 7fe514cde700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe4fc0a1f20 0x7fe4fc0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:37.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.082+0000 7fe5067fc700 1 -- 172.21.15.60:0/2797980347 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe50c0d16a0 con 0x7fe510101ea0 2024-03-04T12:24:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.083+0000 7fe514cde700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe4fc0a1f20 0x7fe4fc0a43e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe5100794f0 tx=0x7fe500007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.088+0000 7fe5067fc700 1 -- 172.21.15.60:0/2797980347 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe50c025020 con 0x7fe510101ea0 2024-03-04T12:24:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.294+0000 7fe517743700 1 -- 172.21.15.60:0/2797980347 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe51002d870 con 0x7fe510101ea0 2024-03-04T12:24:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.294+0000 7fe5067fc700 1 -- 172.21.15.60:0/2797980347 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe50c09e6a0 con 0x7fe510101ea0 2024-03-04T12:24:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.296+0000 7fe517743700 1 -- 172.21.15.60:0/2797980347 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe4fc0a1f20 msgr2=0x7fe4fc0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.296+0000 7fe517743700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe4fc0a1f20 0x7fe4fc0a43e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe5100794f0 tx=0x7fe500007480 comp rx=0 tx=0).stop 2024-03-04T12:24:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.296+0000 7fe517743700 1 -- 172.21.15.60:0/2797980347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe510101ea0 msgr2=0x7fe510198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.296+0000 7fe517743700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe510101ea0 0x7fe510198530 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe50c007ae0 tx=0x7fe50c007df0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.296+0000 7fe517743700 1 -- 172.21.15.60:0/2797980347 shutdown_connections 2024-03-04T12:24:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.296+0000 7fe517743700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe510107eb0 0x7fe51019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.296+0000 7fe517743700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe4fc0a1f20 0x7fe4fc0a43e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.296+0000 7fe517743700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe510101ea0 0x7fe510198530 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.296+0000 7fe517743700 1 --2- 172.21.15.60:0/2797980347 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe510102840 0x7fe510198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:37.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.296+0000 7fe517743700 1 -- 172.21.15.60:0/2797980347 >> 172.21.15.60:0/2797980347 conn(0x7fe510078ef0 msgr2=0x7fe510100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:37.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.296+0000 7fe517743700 1 -- 172.21.15.60:0/2797980347 shutdown_connections 2024-03-04T12:24:37.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:37.297+0000 7fe517743700 1 -- 172.21.15.60:0/2797980347 wait complete. 2024-03-04T12:24:38.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:38 smithi060 ceph-mon[32745]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:38.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:38 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2797980347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T12:24:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.218+0000 7fbefbcd0700 1 -- 172.21.15.60:0/248984147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef4069ad0 msgr2=0x7fbef4069f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.218+0000 7fbefbcd0700 1 --2- 172.21.15.60:0/248984147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef4069ad0 0x7fbef4069f30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbee40099c0 tx=0x7fbee4009cd0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.218+0000 7fbefbcd0700 1 -- 172.21.15.60:0/248984147 shutdown_connections 2024-03-04T12:24:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.218+0000 7fbefbcd0700 1 --2- 172.21.15.60:0/248984147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbef406a470 0x7fbef410ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.218+0000 7fbefbcd0700 1 --2- 172.21.15.60:0/248984147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbef4107e70 0x7fbef4108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.218+0000 7fbefbcd0700 1 --2- 172.21.15.60:0/248984147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef4069ad0 0x7fbef4069f30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.218+0000 7fbefbcd0700 1 -- 172.21.15.60:0/248984147 >> 172.21.15.60:0/248984147 conn(0x7fbef4077c50 msgr2=0x7fbef4078060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:38.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.218+0000 7fbefbcd0700 1 -- 172.21.15.60:0/248984147 shutdown_connections 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.218+0000 7fbefbcd0700 1 -- 172.21.15.60:0/248984147 wait complete. 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbefbcd0700 1 Processor -- start 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbefbcd0700 1 -- start start 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbefbcd0700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbef4069ad0 0x7fbef4198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbefbcd0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef406a470 0x7fbef4198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbefbcd0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbef4107e70 0x7fbef419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbefbcd0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef41120f0 con 0x7fbef406a470 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbefbcd0700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef4111f70 con 0x7fbef4069ad0 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbefbcd0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef4112270 con 0x7fbef4107e70 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbef926b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef406a470 0x7fbef4198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbef926b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef406a470 0x7fbef4198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47254/0 (socket says 172.21.15.60:47254) 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbef926b700 1 -- 172.21.15.60:0/79276528 learned_addr learned my addr 172.21.15.60:0/79276528 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbef9a6c700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbef4069ad0 0x7fbef4198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbefa26d700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbef4107e70 0x7fbef419ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:38.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbef926b700 1 -- 172.21.15.60:0/79276528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbef4107e70 msgr2=0x7fbef419ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbef926b700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbef4107e70 0x7fbef419ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbef926b700 1 -- 172.21.15.60:0/79276528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbef4069ad0 msgr2=0x7fbef4198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbef926b700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbef4069ad0 0x7fbef4198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbef926b700 1 -- 172.21.15.60:0/79276528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef419d550 con 0x7fbef406a470 2024-03-04T12:24:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.219+0000 7fbef9a6c700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbef4069ad0 0x7fbef4198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:24:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.220+0000 7fbef926b700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef406a470 0x7fbef4198a80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbee4006010 tx=0x7fbee4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.220+0000 7fbeeaffd700 1 -- 172.21.15.60:0/79276528 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4026070 con 0x7fbef406a470 2024-03-04T12:24:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.220+0000 7fbeeaffd700 1 -- 172.21.15.60:0/79276528 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbee4004cd0 con 0x7fbef406a470 2024-03-04T12:24:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.220+0000 7fbefbcd0700 1 -- 172.21.15.60:0/79276528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef419d7e0 con 0x7fbef406a470 2024-03-04T12:24:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.220+0000 7fbeeaffd700 1 -- 172.21.15.60:0/79276528 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee4003ba0 con 0x7fbef406a470 2024-03-04T12:24:38.228 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.220+0000 7fbefbcd0700 1 -- 172.21.15.60:0/79276528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef41a50c0 con 0x7fbef406a470 2024-03-04T12:24:38.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.221+0000 7fbefbcd0700 1 -- 172.21.15.60:0/79276528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef4068ae0 con 0x7fbef406a470 2024-03-04T12:24:38.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.222+0000 7fbeeaffd700 1 -- 172.21.15.60:0/79276528 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fbee4029070 con 0x7fbef406a470 2024-03-04T12:24:38.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.223+0000 7fbeeaffd700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbee00a1f20 0x7fbee00a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:38.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.223+0000 7fbef9a6c700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbee00a1f20 0x7fbee00a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:38.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.223+0000 7fbeeaffd700 1 -- 172.21.15.60:0/79276528 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fbee40cef40 con 0x7fbef406a470 2024-03-04T12:24:38.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.223+0000 7fbef9a6c700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbee00a1f20 0x7fbee00a43e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbef000afa0 tx=0x7fbef000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:38.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.226+0000 7fbeeaffd700 1 -- 172.21.15.60:0/79276528 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbee4030020 con 0x7fbef406a470 2024-03-04T12:24:38.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.431+0000 7fbefbcd0700 1 -- 172.21.15.60:0/79276528 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fbef402d870 con 0x7fbef406a470 2024-03-04T12:24:38.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.432+0000 7fbeeaffd700 1 -- 172.21.15.60:0/79276528 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fbee4013070 con 0x7fbef406a470 2024-03-04T12:24:38.438 INFO:teuthology.orchestra.run.smithi060.stdout:287762808874 2024-03-04T12:24:38.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.434+0000 7fbefbcd0700 1 -- 172.21.15.60:0/79276528 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbee00a1f20 msgr2=0x7fbee00a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:38.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.434+0000 7fbefbcd0700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbee00a1f20 0x7fbee00a43e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbef000afa0 tx=0x7fbef000a380 comp rx=0 tx=0).stop 2024-03-04T12:24:38.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.434+0000 7fbefbcd0700 1 -- 172.21.15.60:0/79276528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef406a470 msgr2=0x7fbef4198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:38.440 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.434+0000 7fbefbcd0700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef406a470 0x7fbef4198a80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbee4006010 tx=0x7fbee4009f80 comp rx=0 tx=0).stop 2024-03-04T12:24:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.434+0000 7fbefbcd0700 1 -- 172.21.15.60:0/79276528 shutdown_connections 2024-03-04T12:24:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.434+0000 7fbefbcd0700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbef4107e70 0x7fbef419ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.434+0000 7fbefbcd0700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbef4069ad0 0x7fbef4198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.434+0000 7fbefbcd0700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbee00a1f20 0x7fbee00a43e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.434+0000 7fbefbcd0700 1 --2- 172.21.15.60:0/79276528 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbef406a470 0x7fbef4198a80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.434+0000 7fbefbcd0700 1 -- 172.21.15.60:0/79276528 >> 172.21.15.60:0/79276528 conn(0x7fbef4077c50 msgr2=0x7fbef40ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.434+0000 7fbefbcd0700 1 -- 172.21.15.60:0/79276528 shutdown_connections 2024-03-04T12:24:38.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.434+0000 7fbefbcd0700 1 -- 172.21.15.60:0/79276528 wait complete. 2024-03-04T12:24:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.474+0000 7fc6575df700 1 -- 172.21.15.60:0/4031025546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc650106000 msgr2=0x7fc65010dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.474+0000 7fc6575df700 1 --2- 172.21.15.60:0/4031025546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc650106000 0x7fc65010dbd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc64800b0d0 tx=0x7fc64800b3e0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.474+0000 7fc6575df700 1 -- 172.21.15.60:0/4031025546 shutdown_connections 2024-03-04T12:24:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.474+0000 7fc6575df700 1 --2- 172.21.15.60:0/4031025546 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc650106000 0x7fc65010dbd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.474+0000 7fc6575df700 1 --2- 172.21.15.60:0/4031025546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc650100fb0 0x7fc650101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.475+0000 7fc6575df700 1 --2- 172.21.15.60:0/4031025546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc650101960 0x7fc650105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.475+0000 7fc6575df700 1 -- 172.21.15.60:0/4031025546 >> 172.21.15.60:0/4031025546 conn(0x7fc650076d70 msgr2=0x7fc650077180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:38.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.475+0000 7fc6575df700 1 -- 172.21.15.60:0/4031025546 shutdown_connections 2024-03-04T12:24:38.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.475+0000 7fc6575df700 1 -- 172.21.15.60:0/4031025546 wait complete. 2024-03-04T12:24:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.476+0000 7fc6575df700 1 Processor -- start 2024-03-04T12:24:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.476+0000 7fc6575df700 1 -- start start 2024-03-04T12:24:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.476+0000 7fc6575df700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc650100fb0 0x7fc650199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.476+0000 7fc6575df700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc650101960 0x7fc650199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.476+0000 7fc6575df700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc650106000 0x7fc6501a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.476+0000 7fc6575df700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc650111050 con 0x7fc650101960 2024-03-04T12:24:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.476+0000 7fc6575df700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc650110ed0 con 0x7fc650100fb0 2024-03-04T12:24:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.476+0000 7fc6575df700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6501111d0 con 0x7fc650106000 2024-03-04T12:24:38.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.476+0000 7fc654b7a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc650101960 0x7fc650199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.476+0000 7fc654b7a700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc650101960 0x7fc650199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47282/0 (socket says 172.21.15.60:47282) 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.476+0000 7fc654b7a700 1 -- 172.21.15.60:0/798981131 learned_addr learned my addr 172.21.15.60:0/798981131 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.477+0000 7fc655b7c700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc650106000 0x7fc6501a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.477+0000 7fc65537b700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc650100fb0 0x7fc650199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.477+0000 7fc654b7a700 1 -- 172.21.15.60:0/798981131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc650106000 msgr2=0x7fc6501a44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.477+0000 7fc654b7a700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc650106000 0x7fc6501a44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.477+0000 7fc654b7a700 1 -- 172.21.15.60:0/798981131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc650100fb0 msgr2=0x7fc650199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.477+0000 7fc654b7a700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc650100fb0 0x7fc650199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.477+0000 7fc654b7a700 1 -- 172.21.15.60:0/798981131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6501a49f0 con 0x7fc650101960 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.477+0000 7fc654b7a700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc650101960 0x7fc650199970 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc64000e9f0 tx=0x7fc64000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.478+0000 7fc6467fc700 1 -- 172.21.15.60:0/798981131 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc640004a30 con 0x7fc650101960 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.478+0000 7fc6467fc700 1 -- 172.21.15.60:0/798981131 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc64000cca0 con 0x7fc650101960 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.478+0000 7fc6467fc700 1 -- 172.21.15.60:0/798981131 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6400056c0 con 0x7fc650101960 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.478+0000 7fc6575df700 1 -- 172.21.15.60:0/798981131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc650197220 con 0x7fc650101960 2024-03-04T12:24:38.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.478+0000 7fc6575df700 1 -- 172.21.15.60:0/798981131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc650197710 con 0x7fc650101960 2024-03-04T12:24:38.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.479+0000 7fc6575df700 1 -- 172.21.15.60:0/798981131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc634005330 con 0x7fc650101960 2024-03-04T12:24:38.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.480+0000 7fc6467fc700 1 -- 172.21.15.60:0/798981131 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fc64000c7c0 con 0x7fc650101960 2024-03-04T12:24:38.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.481+0000 7fc6467fc700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fc63c0a1f20 0x7fc63c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:38.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.481+0000 7fc65537b700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fc63c0a1f20 0x7fc63c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:38.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.481+0000 7fc6467fc700 1 -- 172.21.15.60:0/798981131 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc6400ce2c0 con 0x7fc650101960 2024-03-04T12:24:38.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.482+0000 7fc65537b700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fc63c0a1f20 0x7fc63c0a43e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009c60 tx=0x7fc64c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:38.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.485+0000 7fc6467fc700 1 -- 172.21.15.60:0/798981131 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc64009b230 con 0x7fc650101960 2024-03-04T12:24:38.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:38 smithi086 ceph-mon[36851]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:38.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:38 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2797980347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T12:24:38.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:38 smithi063 ceph-mon[36717]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:38.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:38 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2797980347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T12:24:38.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.688+0000 7fc6575df700 1 -- 172.21.15.60:0/798981131 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc6340058f0 con 0x7fc650101960 2024-03-04T12:24:38.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.688+0000 7fc6467fc700 1 -- 172.21.15.60:0/798981131 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc64009ab30 con 0x7fc650101960 2024-03-04T12:24:38.695 INFO:teuthology.orchestra.run.smithi060.stdout:163208757310 2024-03-04T12:24:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.690+0000 7fc6575df700 1 -- 172.21.15.60:0/798981131 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fc63c0a1f20 msgr2=0x7fc63c0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.690+0000 7fc6575df700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fc63c0a1f20 0x7fc63c0a43e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009c60 tx=0x7fc64c009260 comp rx=0 tx=0).stop 2024-03-04T12:24:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.690+0000 7fc6575df700 1 -- 172.21.15.60:0/798981131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc650101960 msgr2=0x7fc650199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.690+0000 7fc6575df700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc650101960 0x7fc650199970 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc64000e9f0 tx=0x7fc64000ed00 comp rx=0 tx=0).stop 2024-03-04T12:24:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.691+0000 7fc6575df700 1 -- 172.21.15.60:0/798981131 shutdown_connections 2024-03-04T12:24:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.691+0000 7fc6575df700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc650106000 0x7fc6501a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.691+0000 7fc6575df700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc650100fb0 0x7fc650199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.691+0000 7fc6575df700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fc63c0a1f20 0x7fc63c0a43e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.691+0000 7fc6575df700 1 --2- 172.21.15.60:0/798981131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc650101960 0x7fc650199970 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:38.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.691+0000 7fc6575df700 1 -- 172.21.15.60:0/798981131 >> 172.21.15.60:0/798981131 conn(0x7fc650076d70 msgr2=0x7fc6500feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:38.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.691+0000 7fc6575df700 1 -- 172.21.15.60:0/798981131 shutdown_connections 2024-03-04T12:24:38.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:38.691+0000 7fc6575df700 1 -- 172.21.15.60:0/798981131 wait complete. 2024-03-04T12:24:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:39 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/79276528' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-04T12:24:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:39 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/798981131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T12:24:39.459 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808870 got 287762808874 for osd.9 2024-03-04T12:24:39.459 DEBUG:teuthology.parallel:result is None 2024-03-04T12:24:39.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:39 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/79276528' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-04T12:24:39.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:39 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/798981131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T12:24:39.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:39 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/79276528' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-04T12:24:39.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:39 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/798981131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T12:24:40.375 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:40 smithi060 ceph-mon[32745]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:24:40.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:40 smithi086 ceph-mon[36851]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:24:40.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:40 smithi063 ceph-mon[36717]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:24:40.728 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920830 got 141733920835 for osd.4 2024-03-04T12:24:40.729 DEBUG:teuthology.parallel:result is None 2024-03-04T12:24:41.576 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574927 got 55834574930 for osd.1 2024-03-04T12:24:41.576 DEBUG:teuthology.parallel:result is None 2024-03-04T12:24:42.003 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757304 got 163208757310 for osd.5 2024-03-04T12:24:42.004 DEBUG:teuthology.parallel:result is None 2024-03-04T12:24:42.123 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:42 smithi060 ceph-mon[32745]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:24:42.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:42 smithi086 ceph-mon[36851]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:24:42.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:42 smithi063 ceph-mon[36717]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:24:42.900 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:43.474 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495666 got 197568495671 for osd.6 2024-03-04T12:24:43.474 DEBUG:teuthology.parallel:result is None 2024-03-04T12:24:44.180 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645345 got 309237645347 for osd.10 2024-03-04T12:24:44.180 DEBUG:teuthology.parallel:result is None 2024-03-04T12:24:44.368 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:44 smithi060 ceph-mon[32745]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:44.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:44 smithi086 ceph-mon[36851]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:44.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:44 smithi063 ceph-mon[36717]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:45.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.125+0000 7eff1cf77700 1 -- 172.21.15.60:0/4177291303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff18100d20 msgr2=0x7eff18101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:45.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.125+0000 7eff1cf77700 1 --2- 172.21.15.60:0/4177291303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff18100d20 0x7eff18101180 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7eff00009990 tx=0x7eff00009ca0 comp rx=0 tx=0).stop 2024-03-04T12:24:45.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.125+0000 7eff1cf77700 1 -- 172.21.15.60:0/4177291303 shutdown_connections 2024-03-04T12:24:45.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.125+0000 7eff1cf77700 1 --2- 172.21.15.60:0/4177291303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff181016c0 0x7eff1810dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:45.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.125+0000 7eff1cf77700 1 --2- 172.21.15.60:0/4177291303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff18106d60 0x7eff18107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:45.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.125+0000 7eff1cf77700 1 --2- 172.21.15.60:0/4177291303 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff18100d20 0x7eff18101180 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:45.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.125+0000 7eff1cf77700 1 -- 172.21.15.60:0/4177291303 >> 172.21.15.60:0/4177291303 conn(0x7eff18077c50 msgr2=0x7eff18078060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:45.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.125+0000 7eff1cf77700 1 -- 172.21.15.60:0/4177291303 shutdown_connections 2024-03-04T12:24:45.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.125+0000 7eff1cf77700 1 -- 172.21.15.60:0/4177291303 wait complete. 2024-03-04T12:24:45.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.126+0000 7eff1cf77700 1 Processor -- start 2024-03-04T12:24:45.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.126+0000 7eff1cf77700 1 -- start start 2024-03-04T12:24:45.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.126+0000 7eff1cf77700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff18100d20 0x7eff1819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:45.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.126+0000 7eff1cf77700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff181016c0 0x7eff1819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:45.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.126+0000 7eff1cf77700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff18106d60 0x7eff1819f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:45.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.126+0000 7eff1cf77700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff18110fc0 con 0x7eff18106d60 2024-03-04T12:24:45.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.126+0000 7eff1cf77700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff18110e40 con 0x7eff18100d20 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.126+0000 7eff1cf77700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff18111140 con 0x7eff181016c0 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.126+0000 7eff16d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff18106d60 0x7eff1819f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.126+0000 7eff16d9d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff18106d60 0x7eff1819f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:47308/0 (socket says 172.21.15.60:47308) 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.126+0000 7eff16d9d700 1 -- 172.21.15.60:0/3423446766 learned_addr learned my addr 172.21.15.60:0/3423446766 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff15d9b700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff181016c0 0x7eff1819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff1659c700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff18100d20 0x7eff1819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff16d9d700 1 -- 172.21.15.60:0/3423446766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff181016c0 msgr2=0x7eff1819ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff16d9d700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff181016c0 0x7eff1819ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff16d9d700 1 -- 172.21.15.60:0/3423446766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff18100d20 msgr2=0x7eff1819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff16d9d700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff18100d20 0x7eff1819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff16d9d700 1 -- 172.21.15.60:0/3423446766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1819f740 con 0x7eff18106d60 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff16d9d700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff18106d60 0x7eff1819f000 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7eff1000d980 tx=0x7eff1000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff0f7fe700 1 -- 172.21.15.60:0/3423446766 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff10017070 con 0x7eff18106d60 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff0f7fe700 1 -- 172.21.15.60:0/3423446766 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff10005020 con 0x7eff18106d60 2024-03-04T12:24:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff1cf77700 1 -- 172.21.15.60:0/3423446766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1819fa30 con 0x7eff18106d60 2024-03-04T12:24:45.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff0f7fe700 1 -- 172.21.15.60:0/3423446766 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1000fbf0 con 0x7eff18106d60 2024-03-04T12:24:45.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.127+0000 7eff1cf77700 1 -- 172.21.15.60:0/3423446766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff181a72b0 con 0x7eff18106d60 2024-03-04T12:24:45.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.128+0000 7eff1cf77700 1 -- 172.21.15.60:0/3423446766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff18068ae0 con 0x7eff18106d60 2024-03-04T12:24:45.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.130+0000 7eff0f7fe700 1 -- 172.21.15.60:0/3423446766 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7eff100051d0 con 0x7eff18106d60 2024-03-04T12:24:45.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.130+0000 7eff0f7fe700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7eff040a1ea0 0x7eff040a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:45.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.130+0000 7eff1659c700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7eff040a1ea0 0x7eff040a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:45.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.130+0000 7eff0f7fe700 1 -- 172.21.15.60:0/3423446766 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7eff100ce450 con 0x7eff18106d60 2024-03-04T12:24:45.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.131+0000 7eff1659c700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7eff040a1ea0 0x7eff040a4360 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7eff08007fd0 tx=0x7eff080079e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:45.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.134+0000 7eff0f7fe700 1 -- 172.21.15.60:0/3423446766 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7eff10020020 con 0x7eff18106d60 2024-03-04T12:24:45.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.338+0000 7eff1cf77700 1 -- 172.21.15.60:0/3423446766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7eff181a7480 con 0x7eff18106d60 2024-03-04T12:24:45.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.338+0000 7eff0f7fe700 1 -- 172.21.15.60:0/3423446766 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7eff1009b3c0 con 0x7eff18106d60 2024-03-04T12:24:45.345 INFO:teuthology.orchestra.run.smithi060.stdout:34359738455 2024-03-04T12:24:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.341+0000 7eff1cf77700 1 -- 172.21.15.60:0/3423446766 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7eff040a1ea0 msgr2=0x7eff040a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.341+0000 7eff1cf77700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7eff040a1ea0 0x7eff040a4360 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7eff08007fd0 tx=0x7eff080079e0 comp rx=0 tx=0).stop 2024-03-04T12:24:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.341+0000 7eff1cf77700 1 -- 172.21.15.60:0/3423446766 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff18106d60 msgr2=0x7eff1819f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.341+0000 7eff1cf77700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff18106d60 0x7eff1819f000 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7eff1000d980 tx=0x7eff1000dc90 comp rx=0 tx=0).stop 2024-03-04T12:24:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.341+0000 7eff1cf77700 1 -- 172.21.15.60:0/3423446766 shutdown_connections 2024-03-04T12:24:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.341+0000 7eff1cf77700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7eff040a1ea0 0x7eff040a4360 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.341+0000 7eff1cf77700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7eff18106d60 0x7eff1819f000 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.341+0000 7eff1cf77700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7eff18100d20 0x7eff1819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:45.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.341+0000 7eff1cf77700 1 --2- 172.21.15.60:0/3423446766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff181016c0 0x7eff1819ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.341+0000 7eff1cf77700 1 -- 172.21.15.60:0/3423446766 >> 172.21.15.60:0/3423446766 conn(0x7eff18077c50 msgr2=0x7eff18069af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.341+0000 7eff1cf77700 1 -- 172.21.15.60:0/3423446766 shutdown_connections 2024-03-04T12:24:45.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:45.341+0000 7eff1cf77700 1 -- 172.21.15.60:0/3423446766 wait complete. 2024-03-04T12:24:46.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:46 smithi060 ceph-mon[32745]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:24:46.213 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:46 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3423446766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T12:24:46.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.342+0000 7f72e2d84700 1 -- 172.21.15.60:0/3323026026 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72dc106d60 msgr2=0x7f72dc107140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.342+0000 7f72e2d84700 1 --2- 172.21.15.60:0/3323026026 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72dc106d60 0x7f72dc107140 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009990 tx=0x7f72cc009ca0 comp rx=0 tx=0).stop 2024-03-04T12:24:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.342+0000 7f72e2d84700 1 -- 172.21.15.60:0/3323026026 shutdown_connections 2024-03-04T12:24:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.342+0000 7f72e2d84700 1 --2- 172.21.15.60:0/3323026026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72dc1016c0 0x7f72dc10dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.342+0000 7f72e2d84700 1 --2- 172.21.15.60:0/3323026026 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72dc106d60 0x7f72dc107140 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.342+0000 7f72e2d84700 1 --2- 172.21.15.60:0/3323026026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72dc100d20 0x7f72dc101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.342+0000 7f72e2d84700 1 -- 172.21.15.60:0/3323026026 >> 172.21.15.60:0/3323026026 conn(0x7f72dc077c50 msgr2=0x7f72dc078060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.342+0000 7f72e2d84700 1 -- 172.21.15.60:0/3323026026 shutdown_connections 2024-03-04T12:24:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.342+0000 7f72e2d84700 1 -- 172.21.15.60:0/3323026026 wait complete. 2024-03-04T12:24:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.342+0000 7f72e2d84700 1 Processor -- start 2024-03-04T12:24:46.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e2d84700 1 -- start start 2024-03-04T12:24:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e2d84700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72dc100d20 0x7f72dc1029d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e2d84700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72dc1016c0 0x7f72dc102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:46.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e2d84700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72dc106d60 0x7f72dc103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e2d84700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc111020 con 0x7f72dc106d60 2024-03-04T12:24:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e2d84700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc110ea0 con 0x7f72dc1016c0 2024-03-04T12:24:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e2d84700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72dc1111a0 con 0x7f72dc100d20 2024-03-04T12:24:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e1321700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72dc106d60 0x7f72dc103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e1321700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72dc106d60 0x7f72dc103450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57590/0 (socket says 172.21.15.60:57590) 2024-03-04T12:24:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e1321700 1 -- 172.21.15.60:0/2085092797 learned_addr learned my addr 172.21.15.60:0/2085092797 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72dbfff700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72dc1016c0 0x7f72dc102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e0b20700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72dc100d20 0x7f72dc1029d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e1321700 1 -- 172.21.15.60:0/2085092797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72dc100d20 msgr2=0x7f72dc1029d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e1321700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72dc100d20 0x7f72dc1029d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:46.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e1321700 1 -- 172.21.15.60:0/2085092797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72dc1016c0 msgr2=0x7f72dc102f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e1321700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72dc1016c0 0x7f72dc102f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e1321700 1 -- 172.21.15.60:0/2085092797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc1a6b70 con 0x7f72dc106d60 2024-03-04T12:24:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e0b20700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72dc100d20 0x7f72dc1029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:24:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.343+0000 7f72e1321700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72dc106d60 0x7f72dc103450 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f72d4007d70 tx=0x7f72d400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.344+0000 7f72d9ffb700 1 -- 172.21.15.60:0/2085092797 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d4016020 con 0x7f72dc106d60 2024-03-04T12:24:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.344+0000 7f72d9ffb700 1 -- 172.21.15.60:0/2085092797 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72d400ad40 con 0x7f72dc106d60 2024-03-04T12:24:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.344+0000 7f72d9ffb700 1 -- 172.21.15.60:0/2085092797 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d4011600 con 0x7f72dc106d60 2024-03-04T12:24:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.344+0000 7f72e2d84700 1 -- 172.21.15.60:0/2085092797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72dc1a6e60 con 0x7f72dc106d60 2024-03-04T12:24:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.344+0000 7f72e2d84700 1 -- 172.21.15.60:0/2085092797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72dc1a73c0 con 0x7f72dc106d60 2024-03-04T12:24:46.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.345+0000 7f72e2d84700 1 -- 172.21.15.60:0/2085092797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72dc068ae0 con 0x7f72dc106d60 2024-03-04T12:24:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.346+0000 7f72d9ffb700 1 -- 172.21.15.60:0/2085092797 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f72d4019020 con 0x7f72dc106d60 2024-03-04T12:24:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.346+0000 7f72d9ffb700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72c40a1f20 0x7f72c40a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.346+0000 7f72e0b20700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72c40a1f20 0x7f72c40a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.347+0000 7f72d9ffb700 1 -- 172.21.15.60:0/2085092797 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f72d40ce080 con 0x7f72dc106d60 2024-03-04T12:24:46.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.347+0000 7f72e0b20700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72c40a1f20 0x7f72c40a43e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f72cc006010 tx=0x7f72cc00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:46.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.351+0000 7f72d9ffb700 1 -- 172.21.15.60:0/2085092797 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f72d4020020 con 0x7f72dc106d60 2024-03-04T12:24:46.401 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:46 smithi086 ceph-mon[36851]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:24:46.401 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:46 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3423446766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T12:24:46.576 INFO:teuthology.orchestra.run.smithi060.stdout:343597383712 2024-03-04T12:24:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.569+0000 7f72e2d84700 1 -- 172.21.15.60:0/2085092797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f72dc1a7650 con 0x7f72dc106d60 2024-03-04T12:24:46.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.569+0000 7f72d9ffb700 1 -- 172.21.15.60:0/2085092797 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f72d409b0a0 con 0x7f72dc106d60 2024-03-04T12:24:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.572+0000 7f72e2d84700 1 -- 172.21.15.60:0/2085092797 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72c40a1f20 msgr2=0x7f72c40a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.572+0000 7f72e2d84700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72c40a1f20 0x7f72c40a43e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f72cc006010 tx=0x7f72cc00b540 comp rx=0 tx=0).stop 2024-03-04T12:24:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.572+0000 7f72e2d84700 1 -- 172.21.15.60:0/2085092797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72dc106d60 msgr2=0x7f72dc103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.572+0000 7f72e2d84700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72dc106d60 0x7f72dc103450 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f72d4007d70 tx=0x7f72d400a340 comp rx=0 tx=0).stop 2024-03-04T12:24:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.572+0000 7f72e2d84700 1 -- 172.21.15.60:0/2085092797 shutdown_connections 2024-03-04T12:24:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.572+0000 7f72e2d84700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f72c40a1f20 0x7f72c40a43e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.572+0000 7f72e2d84700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f72dc106d60 0x7f72dc103450 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:46.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.572+0000 7f72e2d84700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72dc100d20 0x7f72dc1029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.572+0000 7f72e2d84700 1 --2- 172.21.15.60:0/2085092797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72dc1016c0 0x7f72dc102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.572+0000 7f72e2d84700 1 -- 172.21.15.60:0/2085092797 >> 172.21.15.60:0/2085092797 conn(0x7f72dc077c50 msgr2=0x7f72dc0ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.572+0000 7f72e2d84700 1 -- 172.21.15.60:0/2085092797 shutdown_connections 2024-03-04T12:24:46.579 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:46.572+0000 7f72e2d84700 1 -- 172.21.15.60:0/2085092797 wait complete. 2024-03-04T12:24:46.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:46 smithi063 ceph-mon[36717]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:24:46.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:46 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3423446766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T12:24:47.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:47 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2085092797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-04T12:24:47.468 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383708 got 343597383712 for osd.11 2024-03-04T12:24:47.468 DEBUG:teuthology.parallel:result is None 2024-03-04T12:24:47.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:47 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2085092797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-04T12:24:47.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:47 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2085092797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-04T12:24:48.142 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:48 smithi060 ceph-mon[32745]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.136+0000 7f2c7dda4700 1 -- 172.21.15.60:0/1888816020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c7806b130 msgr2=0x7f2c78106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.136+0000 7f2c7dda4700 1 --2- 172.21.15.60:0/1888816020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c7806b130 0x7f2c78106aa0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0099c0 tx=0x7f2c6c009cd0 comp rx=0 tx=0).stop 2024-03-04T12:24:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.136+0000 7f2c7dda4700 1 -- 172.21.15.60:0/1888816020 shutdown_connections 2024-03-04T12:24:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.136+0000 7f2c7dda4700 1 --2- 172.21.15.60:0/1888816020 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c781071d0 0x7f2c7810ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.136+0000 7f2c7dda4700 1 --2- 172.21.15.60:0/1888816020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c7806a780 0x7f2c7806ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.136+0000 7f2c7dda4700 1 --2- 172.21.15.60:0/1888816020 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c7806b130 0x7f2c78106aa0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.136+0000 7f2c7dda4700 1 -- 172.21.15.60:0/1888816020 >> 172.21.15.60:0/1888816020 conn(0x7f2c78076d70 msgr2=0x7f2c78077180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.136+0000 7f2c7dda4700 1 -- 172.21.15.60:0/1888816020 shutdown_connections 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.136+0000 7f2c7dda4700 1 -- 172.21.15.60:0/1888816020 wait complete. 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.136+0000 7f2c7dda4700 1 Processor -- start 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c7dda4700 1 -- start start 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c7dda4700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c7806a780 0x7f2c781985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c7dda4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c7806b130 0x7f2c78198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c7dda4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c781071d0 0x7f2c7819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c7dda4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78112120 con 0x7f2c7806b130 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c7dda4700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78111fa0 con 0x7f2c7806a780 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c7dda4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c781122a0 con 0x7f2c781071d0 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c76ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c7806b130 0x7f2c78198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c76ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c7806b130 0x7f2c78198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57608/0 (socket says 172.21.15.60:57608) 2024-03-04T12:24:48.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c76ffd700 1 -- 172.21.15.60:0/2297814095 learned_addr learned my addr 172.21.15.60:0/2297814095 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c777fe700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c7806a780 0x7f2c781985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c77fff700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c781071d0 0x7f2c7819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c76ffd700 1 -- 172.21.15.60:0/2297814095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c781071d0 msgr2=0x7f2c7819ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c76ffd700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c781071d0 0x7f2c7819ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c76ffd700 1 -- 172.21.15.60:0/2297814095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c7806a780 msgr2=0x7f2c781985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c76ffd700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c7806a780 0x7f2c781985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.137+0000 7f2c76ffd700 1 -- 172.21.15.60:0/2297814095 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7819d5c0 con 0x7f2c7806b130 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.138+0000 7f2c76ffd700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c7806b130 0x7f2c78198af0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0098e0 tx=0x7f2c6c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.138+0000 7f2c74ff9700 1 -- 172.21.15.60:0/2297814095 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c027070 con 0x7f2c7806b130 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.138+0000 7f2c7dda4700 1 -- 172.21.15.60:0/2297814095 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7819d850 con 0x7f2c7806b130 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.138+0000 7f2c74ff9700 1 -- 172.21.15.60:0/2297814095 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c6c003e40 con 0x7f2c7806b130 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.138+0000 7f2c7dda4700 1 -- 172.21.15.60:0/2297814095 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c781a5130 con 0x7f2c7806b130 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.138+0000 7f2c74ff9700 1 -- 172.21.15.60:0/2297814095 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c022c00 con 0x7f2c7806b130 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.139+0000 7f2c7dda4700 1 -- 172.21.15.60:0/2297814095 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c78068ae0 con 0x7f2c7806b130 2024-03-04T12:24:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.140+0000 7f2c74ff9700 1 -- 172.21.15.60:0/2297814095 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f2c6c02a070 con 0x7f2c7806b130 2024-03-04T12:24:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.140+0000 7f2c74ff9700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2c600a1ea0 0x7f2c600a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.141+0000 7f2c777fe700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2c600a1ea0 0x7f2c600a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.141+0000 7f2c74ff9700 1 -- 172.21.15.60:0/2297814095 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2c6c0cf180 con 0x7f2c7806b130 2024-03-04T12:24:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.141+0000 7f2c777fe700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2c600a1ea0 0x7f2c600a4360 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2c68005950 tx=0x7f2c6800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.145+0000 7f2c74ff9700 1 -- 172.21.15.60:0/2297814095 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2c6c05dd20 con 0x7f2c7806b130 2024-03-04T12:24:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.351+0000 7f2c7dda4700 1 -- 172.21.15.60:0/2297814095 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f2c7802d870 con 0x7f2c7806b130 2024-03-04T12:24:48.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.351+0000 7f2c74ff9700 1 -- 172.21.15.60:0/2297814095 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2c6c013070 con 0x7f2c7806b130 2024-03-04T12:24:48.357 INFO:teuthology.orchestra.run.smithi060.stdout:253403070513 2024-03-04T12:24:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.353+0000 7f2c7dda4700 1 -- 172.21.15.60:0/2297814095 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2c600a1ea0 msgr2=0x7f2c600a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.353+0000 7f2c7dda4700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2c600a1ea0 0x7f2c600a4360 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2c68005950 tx=0x7f2c6800b410 comp rx=0 tx=0).stop 2024-03-04T12:24:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.353+0000 7f2c7dda4700 1 -- 172.21.15.60:0/2297814095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c7806b130 msgr2=0x7f2c78198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.353+0000 7f2c7dda4700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c7806b130 0x7f2c78198af0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0098e0 tx=0x7f2c6c0047b0 comp rx=0 tx=0).stop 2024-03-04T12:24:48.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.353+0000 7f2c7dda4700 1 -- 172.21.15.60:0/2297814095 shutdown_connections 2024-03-04T12:24:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.353+0000 7f2c7dda4700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c781071d0 0x7f2c7819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.353+0000 7f2c7dda4700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c7806a780 0x7f2c781985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.353+0000 7f2c7dda4700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2c600a1ea0 0x7f2c600a4360 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.353+0000 7f2c7dda4700 1 --2- 172.21.15.60:0/2297814095 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c7806b130 0x7f2c78198af0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.353+0000 7f2c7dda4700 1 -- 172.21.15.60:0/2297814095 >> 172.21.15.60:0/2297814095 conn(0x7f2c78076d70 msgr2=0x7f2c780fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.353+0000 7f2c7dda4700 1 -- 172.21.15.60:0/2297814095 shutdown_connections 2024-03-04T12:24:48.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:48.353+0000 7f2c7dda4700 1 -- 172.21.15.60:0/2297814095 wait complete. 2024-03-04T12:24:48.503 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:48 smithi063 ceph-mon[36717]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:48.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:48 smithi086 ceph-mon[36851]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:49.045 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070505 got 253403070513 for osd.8 2024-03-04T12:24:49.045 DEBUG:teuthology.parallel:result is None 2024-03-04T12:24:49.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:49 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2297814095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-04T12:24:49.474 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738448 got 34359738455 for osd.0 2024-03-04T12:24:49.474 DEBUG:teuthology.parallel:result is None 2024-03-04T12:24:49.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:49 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2297814095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-04T12:24:49.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:49 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2297814095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-04T12:24:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.671+0000 7fb551112700 1 -- 172.21.15.60:0/3983512414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c102100 msgr2=0x7fb54c1024e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.671+0000 7fb551112700 1 --2- 172.21.15.60:0/3983512414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c102100 0x7fb54c1024e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb53c0099c0 tx=0x7fb53c009cd0 comp rx=0 tx=0).stop 2024-03-04T12:24:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.671+0000 7fb551112700 1 -- 172.21.15.60:0/3983512414 shutdown_connections 2024-03-04T12:24:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.671+0000 7fb551112700 1 --2- 172.21.15.60:0/3983512414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c1071b0 0x7fb54c10ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.671+0000 7fb551112700 1 --2- 172.21.15.60:0/3983512414 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c102100 0x7fb54c1024e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.671+0000 7fb551112700 1 --2- 172.21.15.60:0/3983512414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb54c102ab0 0x7fb54c106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.671+0000 7fb551112700 1 -- 172.21.15.60:0/3983512414 >> 172.21.15.60:0/3983512414 conn(0x7fb54c078f40 msgr2=0x7fb54c079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.671+0000 7fb551112700 1 -- 172.21.15.60:0/3983512414 shutdown_connections 2024-03-04T12:24:49.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.671+0000 7fb551112700 1 -- 172.21.15.60:0/3983512414 wait complete. 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb551112700 1 Processor -- start 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb551112700 1 -- start start 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb551112700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb54c102100 0x7fb54c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb551112700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c102ab0 0x7fb54c198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb551112700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c1071b0 0x7fb54c19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb551112700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c112170 con 0x7fb54c1071b0 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb551112700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c111ff0 con 0x7fb54c102100 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb551112700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c1122f0 con 0x7fb54c102ab0 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb54b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c1071b0 0x7fb54c19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb54b59e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c1071b0 0x7fb54c19ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57618/0 (socket says 172.21.15.60:57618) 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb54b59e700 1 -- 172.21.15.60:0/1678405852 learned_addr learned my addr 172.21.15.60:0/1678405852 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb54ad9d700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb54c102100 0x7fb54c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.672+0000 7fb54a59c700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c102ab0 0x7fb54c198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:49.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.673+0000 7fb54b59e700 1 -- 172.21.15.60:0/1678405852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c102ab0 msgr2=0x7fb54c198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.673+0000 7fb54b59e700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c102ab0 0x7fb54c198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.673+0000 7fb54b59e700 1 -- 172.21.15.60:0/1678405852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb54c102100 msgr2=0x7fb54c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.673+0000 7fb54b59e700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb54c102100 0x7fb54c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.673+0000 7fb54b59e700 1 -- 172.21.15.60:0/1678405852 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c19d5b0 con 0x7fb54c1071b0 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.673+0000 7fb54b59e700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c1071b0 0x7fb54c19ce70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb544007ee0 tx=0x7fb54400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.673+0000 7fb543fff700 1 -- 172.21.15.60:0/1678405852 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544004bb0 con 0x7fb54c1071b0 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.673+0000 7fb543fff700 1 -- 172.21.15.60:0/1678405852 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb544004d50 con 0x7fb54c1071b0 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.673+0000 7fb551112700 1 -- 172.21.15.60:0/1678405852 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54c19d8a0 con 0x7fb54c1071b0 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.673+0000 7fb543fff700 1 -- 172.21.15.60:0/1678405852 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544005960 con 0x7fb54c1071b0 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.673+0000 7fb551112700 1 -- 172.21.15.60:0/1678405852 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54c1a5120 con 0x7fb54c1071b0 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.674+0000 7fb551112700 1 -- 172.21.15.60:0/1678405852 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54c10a380 con 0x7fb54c1071b0 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.675+0000 7fb543fff700 1 -- 172.21.15.60:0/1678405852 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fb54401a020 con 0x7fb54c1071b0 2024-03-04T12:24:49.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.676+0000 7fb543fff700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb5380a1ea0 0x7fb5380a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:49.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.676+0000 7fb54ad9d700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb5380a1ea0 0x7fb5380a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:49.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.676+0000 7fb543fff700 1 -- 172.21.15.60:0/1678405852 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb5440ce070 con 0x7fb54c1071b0 2024-03-04T12:24:49.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.676+0000 7fb54ad9d700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb5380a1ea0 0x7fb5380a4360 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb53c000c00 tx=0x7fb53c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:49.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.679+0000 7fb543fff700 1 -- 172.21.15.60:0/1678405852 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb544010070 con 0x7fb54c1071b0 2024-03-04T12:24:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.885+0000 7fb551112700 1 -- 172.21.15.60:0/1678405852 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fb54c109d60 con 0x7fb54c1071b0 2024-03-04T12:24:49.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.885+0000 7fb543fff700 1 -- 172.21.15.60:0/1678405852 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb54409af80 con 0x7fb54c1071b0 2024-03-04T12:24:49.892 INFO:teuthology.orchestra.run.smithi060.stdout:219043332149 2024-03-04T12:24:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.887+0000 7fb551112700 1 -- 172.21.15.60:0/1678405852 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb5380a1ea0 msgr2=0x7fb5380a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:49.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.887+0000 7fb551112700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb5380a1ea0 0x7fb5380a4360 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb53c000c00 tx=0x7fb53c005fb0 comp rx=0 tx=0).stop 2024-03-04T12:24:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.887+0000 7fb551112700 1 -- 172.21.15.60:0/1678405852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c1071b0 msgr2=0x7fb54c19ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.887+0000 7fb551112700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c1071b0 0x7fb54c19ce70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fb544007ee0 tx=0x7fb54400d3b0 comp rx=0 tx=0).stop 2024-03-04T12:24:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.887+0000 7fb551112700 1 -- 172.21.15.60:0/1678405852 shutdown_connections 2024-03-04T12:24:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.887+0000 7fb551112700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb5380a1ea0 0x7fb5380a4360 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.887+0000 7fb551112700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb54c1071b0 0x7fb54c19ce70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.887+0000 7fb551112700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb54c102100 0x7fb54c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.887+0000 7fb551112700 1 --2- 172.21.15.60:0/1678405852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c102ab0 0x7fb54c198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.887+0000 7fb551112700 1 -- 172.21.15.60:0/1678405852 >> 172.21.15.60:0/1678405852 conn(0x7fb54c078f40 msgr2=0x7fb54c101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.888+0000 7fb551112700 1 -- 172.21.15.60:0/1678405852 shutdown_connections 2024-03-04T12:24:49.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:49.888+0000 7fb551112700 1 -- 172.21.15.60:0/1678405852 wait complete. 2024-03-04T12:24:50.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:50 smithi060 ceph-mon[32745]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:24:50.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:50 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1678405852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T12:24:50.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:50 smithi086 ceph-mon[36851]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:24:50.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:50 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1678405852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T12:24:50.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:50 smithi063 ceph-mon[36717]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:24:50.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:50 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1678405852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T12:24:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.702+0000 7f761174e700 1 -- 172.21.15.60:0/297165188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760c107260 msgr2=0x7f760c10ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.702+0000 7f761174e700 1 --2- 172.21.15.60:0/297165188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760c107260 0x7f760c10ede0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f760400b0a0 tx=0x7f760400b3b0 comp rx=0 tx=0).stop 2024-03-04T12:24:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.702+0000 7f761174e700 1 -- 172.21.15.60:0/297165188 shutdown_connections 2024-03-04T12:24:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.702+0000 7f761174e700 1 --2- 172.21.15.60:0/297165188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760c107260 0x7f760c10ede0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.702+0000 7f761174e700 1 --2- 172.21.15.60:0/297165188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f760c06a7c0 0x7f760c06aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.702+0000 7f761174e700 1 --2- 172.21.15.60:0/297165188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c06b170 0x7f760c106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.702+0000 7f761174e700 1 -- 172.21.15.60:0/297165188 >> 172.21.15.60:0/297165188 conn(0x7f760c077b80 msgr2=0x7f760c077f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.702+0000 7f761174e700 1 -- 172.21.15.60:0/297165188 shutdown_connections 2024-03-04T12:24:50.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.702+0000 7f761174e700 1 -- 172.21.15.60:0/297165188 wait complete. 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.702+0000 7f761174e700 1 Processor -- start 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f761174e700 1 -- start start 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f761174e700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f760c06a7c0 0x7f760c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f761174e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760c06b170 0x7f760c198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f761174e700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c107260 0x7f760c19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f761174e700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c1121c0 con 0x7f760c06b170 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f761174e700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c112040 con 0x7f760c06a7c0 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f761174e700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c112340 con 0x7f760c107260 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f760a7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760c06b170 0x7f760c198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f760a7fc700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760c06b170 0x7f760c198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57628/0 (socket says 172.21.15.60:57628) 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f760a7fc700 1 -- 172.21.15.60:0/145673270 learned_addr learned my addr 172.21.15.60:0/145673270 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f760b7fe700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c107260 0x7f760c19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f760affd700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f760c06a7c0 0x7f760c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:50.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f760a7fc700 1 -- 172.21.15.60:0/145673270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c107260 msgr2=0x7f760c19ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f760a7fc700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c107260 0x7f760c19ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f760a7fc700 1 -- 172.21.15.60:0/145673270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f760c06a7c0 msgr2=0x7f760c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f760a7fc700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f760c06a7c0 0x7f760c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.703+0000 7f760a7fc700 1 -- 172.21.15.60:0/145673270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760c19d580 con 0x7f760c06b170 2024-03-04T12:24:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.704+0000 7f760a7fc700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760c06b170 0x7f760c198ab0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f760000d950 tx=0x7f760000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.704+0000 7f75f3fff700 1 -- 172.21.15.60:0/145673270 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600016070 con 0x7f760c06b170 2024-03-04T12:24:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.704+0000 7f761174e700 1 -- 172.21.15.60:0/145673270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760c19d870 con 0x7f760c06b170 2024-03-04T12:24:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.704+0000 7f75f3fff700 1 -- 172.21.15.60:0/145673270 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7600004d10 con 0x7f760c06b170 2024-03-04T12:24:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.704+0000 7f75f3fff700 1 -- 172.21.15.60:0/145673270 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600005600 con 0x7f760c06b170 2024-03-04T12:24:50.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.704+0000 7f761174e700 1 -- 172.21.15.60:0/145673270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760c1a50f0 con 0x7f760c06b170 2024-03-04T12:24:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.705+0000 7f761174e700 1 -- 172.21.15.60:0/145673270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760c1035e0 con 0x7f760c06b170 2024-03-04T12:24:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.706+0000 7f75f3fff700 1 -- 172.21.15.60:0/145673270 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f7600004220 con 0x7f760c06b170 2024-03-04T12:24:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.706+0000 7f75f3fff700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f75f40a1ea0 0x7f75f40a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:50.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.706+0000 7f760affd700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f75f40a1ea0 0x7f75f40a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:50.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.707+0000 7f75f3fff700 1 -- 172.21.15.60:0/145673270 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f76000cdff0 con 0x7f760c06b170 2024-03-04T12:24:50.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.707+0000 7f760affd700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f75f40a1ea0 0x7f75f40a4360 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00abd0 tx=0x7f75fc00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.711+0000 7f75f3fff700 1 -- 172.21.15.60:0/145673270 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f760001f020 con 0x7f760c06b170 2024-03-04T12:24:50.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.918+0000 7f761174e700 1 -- 172.21.15.60:0/145673270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f760c199900 con 0x7f760c06b170 2024-03-04T12:24:50.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.918+0000 7f75f3fff700 1 -- 172.21.15.60:0/145673270 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f760c199900 con 0x7f760c06b170 2024-03-04T12:24:50.924 INFO:teuthology.orchestra.run.smithi060.stdout:77309411408 2024-03-04T12:24:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.920+0000 7f761174e700 1 -- 172.21.15.60:0/145673270 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f75f40a1ea0 msgr2=0x7f75f40a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.920+0000 7f761174e700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f75f40a1ea0 0x7f75f40a4360 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00abd0 tx=0x7f75fc00a300 comp rx=0 tx=0).stop 2024-03-04T12:24:50.926 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.920+0000 7f761174e700 1 -- 172.21.15.60:0/145673270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760c06b170 msgr2=0x7f760c198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.920+0000 7f761174e700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760c06b170 0x7f760c198ab0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f760000d950 tx=0x7f760000dc60 comp rx=0 tx=0).stop 2024-03-04T12:24:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.920+0000 7f761174e700 1 -- 172.21.15.60:0/145673270 shutdown_connections 2024-03-04T12:24:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.920+0000 7f761174e700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f760c107260 0x7f760c19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.920+0000 7f761174e700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f760c06a7c0 0x7f760c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.920+0000 7f761174e700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f75f40a1ea0 0x7f75f40a4360 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.920+0000 7f761174e700 1 --2- 172.21.15.60:0/145673270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f760c06b170 0x7f760c198ab0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.920+0000 7f761174e700 1 -- 172.21.15.60:0/145673270 >> 172.21.15.60:0/145673270 conn(0x7f760c077b80 msgr2=0x7f760c0fef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.920+0000 7f761174e700 1 -- 172.21.15.60:0/145673270 shutdown_connections 2024-03-04T12:24:50.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:50.920+0000 7f761174e700 1 -- 172.21.15.60:0/145673270 wait complete. 2024-03-04T12:24:51.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:51 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/145673270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T12:24:51.539 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332142 got 219043332149 for osd.7 2024-03-04T12:24:51.540 DEBUG:teuthology.parallel:result is None 2024-03-04T12:24:51.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:51 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/145673270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T12:24:51.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:51 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/145673270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T12:24:52.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:52 smithi060 ceph-mon[32745]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:24:52.228 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411402 got 77309411408 for osd.2 2024-03-04T12:24:52.229 DEBUG:teuthology.parallel:result is None 2024-03-04T12:24:52.229 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-04T12:24:52.229 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T12:24:52.497 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:52.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:52 smithi086 ceph-mon[36851]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:24:52.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:52 smithi063 ceph-mon[36717]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:24:53.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:53 smithi086 ceph-mon[36851]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:53.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:53.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:53.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:24:53.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:24:53.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:53 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:53.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:53 smithi063 ceph-mon[36717]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:53.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:53.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:53.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:24:53.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:24:53.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:53 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:53.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:53 smithi060 ceph-mon[32745]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:53.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:53.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:53.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:24:53.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:24:53.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:53 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' 2024-03-04T12:24:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.344+0000 7f97e6983700 1 -- 172.21.15.60:0/4212302793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97e0100d20 msgr2=0x7f97e0101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.344+0000 7f97e6983700 1 --2- 172.21.15.60:0/4212302793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97e0100d20 0x7f97e0101180 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f97c8009990 tx=0x7f97c8009ca0 comp rx=0 tx=0).stop 2024-03-04T12:24:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.344+0000 7f97e6983700 1 -- 172.21.15.60:0/4212302793 shutdown_connections 2024-03-04T12:24:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.344+0000 7f97e6983700 1 --2- 172.21.15.60:0/4212302793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97e01016c0 0x7f97e010dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.344+0000 7f97e6983700 1 --2- 172.21.15.60:0/4212302793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97e0106d60 0x7f97e0107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.344+0000 7f97e6983700 1 --2- 172.21.15.60:0/4212302793 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97e0100d20 0x7f97e0101180 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.344+0000 7f97e6983700 1 -- 172.21.15.60:0/4212302793 >> 172.21.15.60:0/4212302793 conn(0x7f97e0077c50 msgr2=0x7f97e0078060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:54.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.344+0000 7f97e6983700 1 -- 172.21.15.60:0/4212302793 shutdown_connections 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.344+0000 7f97e6983700 1 -- 172.21.15.60:0/4212302793 wait complete. 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97e6983700 1 Processor -- start 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97e6983700 1 -- start start 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97e6983700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97e0100d20 0x7f97e01029d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97e6983700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97e01016c0 0x7f97e0102f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97e6983700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97e0106d60 0x7f97e0103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97e6983700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e0111060 con 0x7f97e0106d60 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97e6983700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e0110ee0 con 0x7f97e0100d20 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97e6983700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e01111e0 con 0x7f97e01016c0 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97e4f20700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97e0106d60 0x7f97e0103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97d7fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97e01016c0 0x7f97e0102f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97dffff700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97e0100d20 0x7f97e01029d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97d7fff700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97e01016c0 0x7f97e0102f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:49842/0 (socket says 172.21.15.60:49842) 2024-03-04T12:24:54.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97e4f20700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97e0106d60 0x7f97e0103450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:57650/0 (socket says 172.21.15.60:57650) 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97d7fff700 1 -- 172.21.15.60:0/1714907640 learned_addr learned my addr 172.21.15.60:0/1714907640 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97d7fff700 1 -- 172.21.15.60:0/1714907640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97e0100d20 msgr2=0x7f97e01029d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97d7fff700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97e0100d20 0x7f97e01029d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97d7fff700 1 -- 172.21.15.60:0/1714907640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97e0106d60 msgr2=0x7f97e0103450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97d7fff700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97e0106d60 0x7f97e0103450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97d7fff700 1 -- 172.21.15.60:0/1714907640 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e01a6b10 con 0x7f97e01016c0 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.345+0000 7f97e4f20700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97e0106d60 0x7f97e0103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.346+0000 7f97d7fff700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97e01016c0 0x7f97e0102f10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f97c8000c00 tx=0x7f97c8009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.346+0000 7f97ddffb700 1 -- 172.21.15.60:0/1714907640 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c80047f0 con 0x7f97e01016c0 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.346+0000 7f97e6983700 1 -- 172.21.15.60:0/1714907640 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e01a6da0 con 0x7f97e01016c0 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.346+0000 7f97ddffb700 1 -- 172.21.15.60:0/1714907640 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97c8005030 con 0x7f97e01016c0 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.346+0000 7f97ddffb700 1 -- 172.21.15.60:0/1714907640 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c8003ce0 con 0x7f97e01016c0 2024-03-04T12:24:54.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.346+0000 7f97e6983700 1 -- 172.21.15.60:0/1714907640 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e01a71a0 con 0x7f97e01016c0 2024-03-04T12:24:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.347+0000 7f97e6983700 1 -- 172.21.15.60:0/1714907640 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e0068ae0 con 0x7f97e01016c0 2024-03-04T12:24:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.348+0000 7f97ddffb700 1 -- 172.21.15.60:0/1714907640 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f97c80051e0 con 0x7f97e01016c0 2024-03-04T12:24:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.348+0000 7f97ddffb700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f97cc0a1ee0 0x7f97cc0a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:54.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.349+0000 7f97dffff700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f97cc0a1ee0 0x7f97cc0a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:54.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.349+0000 7f97ddffb700 1 -- 172.21.15.60:0/1714907640 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f97c80ced30 con 0x7f97e01016c0 2024-03-04T12:24:54.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.349+0000 7f97dffff700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f97cc0a1ee0 0x7f97cc0a43a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f97d0007b60 tx=0x7f97d00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:54.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.354+0000 7f97ddffb700 1 -- 172.21.15.60:0/1714907640 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f97c8013070 con 0x7f97e01016c0 2024-03-04T12:24:54.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.552+0000 7f97e6983700 1 -- 172.21.15.60:0/1714907640 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97e0069af0 con 0x7f97cc0a1ee0 2024-03-04T12:24:54.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.562+0000 7f97ddffb700 1 -- 172.21.15.60:0/1714907640 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340678 (secure 0 0 0) 0x7f97e0069af0 con 0x7f97cc0a1ee0 2024-03-04T12:24:54.573 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:24:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.565+0000 7f97e6983700 1 -- 172.21.15.60:0/1714907640 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f97cc0a1ee0 msgr2=0x7f97cc0a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.565+0000 7f97e6983700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f97cc0a1ee0 0x7f97cc0a43a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f97d0007b60 tx=0x7f97d00073d0 comp rx=0 tx=0).stop 2024-03-04T12:24:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.565+0000 7f97e6983700 1 -- 172.21.15.60:0/1714907640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97e01016c0 msgr2=0x7f97e0102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.565+0000 7f97e6983700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97e01016c0 0x7f97e0102f10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f97c8000c00 tx=0x7f97c8009580 comp rx=0 tx=0).stop 2024-03-04T12:24:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.565+0000 7f97e6983700 1 -- 172.21.15.60:0/1714907640 shutdown_connections 2024-03-04T12:24:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.565+0000 7f97e6983700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f97cc0a1ee0 0x7f97cc0a43a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.565+0000 7f97e6983700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f97e0106d60 0x7f97e0103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.565+0000 7f97e6983700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97e0100d20 0x7f97e01029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.565+0000 7f97e6983700 1 --2- 172.21.15.60:0/1714907640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97e01016c0 0x7f97e0102f10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:54.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.565+0000 7f97e6983700 1 -- 172.21.15.60:0/1714907640 >> 172.21.15.60:0/1714907640 conn(0x7f97e0077c50 msgr2=0x7f97e0100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:54.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.565+0000 7f97e6983700 1 -- 172.21.15.60:0/1714907640 shutdown_connections 2024-03-04T12:24:54.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:54.565+0000 7f97e6983700 1 -- 172.21.15.60:0/1714907640 wait complete. 2024-03-04T12:24:54.577 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-04T12:24:55.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:55 smithi060 ceph-mon[32745]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:24:55.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:55 smithi060 ceph-mon[32745]: from='client.24862 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:24:55.539 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-03-04T12:24:54.180002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658075,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":897,"num_read_kb":573,"num_write":422,"num_write_kb":3076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382332,"kb_used_data":12296,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559492,"statfs":{"total":1151940427776,"available":1151548919808,"internally_reserved":0,"allocated":12591104,"data_stored":7960127,"data_compressed":357048,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":145,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003094"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955770+0000","last_change":"2024-03-04T12:23:12.926673+0000","last_active":"2024-03-04T12:24:05.955770+0000","last_peered":"2024-03-04T12:24:05.955770+0000","last_clean":"2024-03-04T12:24:05.955770+0000","last_became_active":"2024-03-04T12:23:12.926299+0000","last_became_peered":"2024-03-04T12:23:12.926299+0000","last_unstale":"2024-03-04T12:24:05.955770+0000","last_undegraded":"2024-03-04T12:24:05.955770+0000","last_fullsized":"2024-03-04T12:24:05.955770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:19.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955021+0000","last_change":"2024-03-04T12:23:12.921758+0000","last_active":"2024-03-04T12:24:05.955021+0000","last_peered":"2024-03-04T12:24:05.955021+0000","last_clean":"2024-03-04T12:24:05.955021+0000","last_became_active":"2024-03-04T12:23:12.921317+0000","last_became_peered":"2024-03-04T12:23:12.921317+0000","last_unstale":"2024-03-04T12:24:05.955021+0000","last_undegraded":"2024-03-04T12:24:05.955021+0000","last_fullsized":"2024-03-04T12:24:05.955021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:20:25.179004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.956186+0000","last_change":"2024-03-04T12:23:12.925771+0000","last_active":"2024-03-04T12:24:05.956186+0000","last_peered":"2024-03-04T12:24:05.956186+0000","last_clean":"2024-03-04T12:24:05.956186+0000","last_became_active":"2024-03-04T12:23:12.925447+0000","last_became_peered":"2024-03-04T12:23:12.925447+0000","last_unstale":"2024-03-04T12:24:05.956186+0000","last_undegraded":"2024-03-04T12:24:05.956186+0000","last_fullsized":"2024-03-04T12:24:05.956186+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:12.380831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.959806+0000","last_change":"2024-03-04T12:24:05.959806+0000","last_active":"2024-03-04T12:24:05.959806+0000","last_peered":"2024-03-04T12:24:05.959806+0000","last_clean":"2024-03-04T12:24:05.959806+0000","last_became_active":"2024-03-04T12:24:05.959557+0000","last_became_peered":"2024-03-04T12:24:05.959557+0000","last_unstale":"2024-03-04T12:24:05.959806+0000","last_undegraded":"2024-03-04T12:24:05.959806+0000","last_fullsized":"2024-03-04T12:24:05.959806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:51.945517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.957611+0000","last_change":"2024-03-04T12:23:12.921118+0000","last_active":"2024-03-04T12:24:05.957611+0000","last_peered":"2024-03-04T12:24:05.957611+0000","last_clean":"2024-03-04T12:24:05.957611+0000","last_became_active":"2024-03-04T12:23:12.920127+0000","last_became_peered":"2024-03-04T12:23:12.920127+0000","last_unstale":"2024-03-04T12:24:05.957611+0000","last_undegraded":"2024-03-04T12:24:05.957611+0000","last_fullsized":"2024-03-04T12:24:05.957611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:22:10.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954447+0000","last_change":"2024-03-04T12:23:12.916751+0000","last_active":"2024-03-04T12:24:05.954447+0000","last_peered":"2024-03-04T12:24:05.954447+0000","last_clean":"2024-03-04T12:24:05.954447+0000","last_became_active":"2024-03-04T12:23:12.916404+0000","last_became_peered":"2024-03-04T12:23:12.916404+0000","last_unstale":"2024-03-04T12:24:05.954447+0000","last_undegraded":"2024-03-04T12:24:05.954447+0000","last_fullsized":"2024-03-04T12:24:05.954447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:39.998992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029608+0000","last_change":"2024-03-04T12:24:06.029608+0000","last_active":"2024-03-04T12:24:06.029608+0000","last_peered":"2024-03-04T12:24:06.029608+0000","last_clean":"2024-03-04T12:24:06.029608+0000","last_became_active":"2024-03-04T12:24:06.029021+0000","last_became_peered":"2024-03-04T12:24:06.029021+0000","last_unstale":"2024-03-04T12:24:06.029608+0000","last_undegraded":"2024-03-04T12:24:06.029608+0000","last_fullsized":"2024-03-04T12:24:06.029608+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:13.140511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955512+0000","last_change":"2024-03-04T12:23:12.917627+0000","last_active":"2024-03-04T12:24:05.955512+0000","last_peered":"2024-03-04T12:24:05.955512+0000","last_clean":"2024-03-04T12:24:05.955512+0000","last_became_active":"2024-03-04T12:23:12.917279+0000","last_became_peered":"2024-03-04T12:23:12.917279+0000","last_unstale":"2024-03-04T12:24:05.955512+0000","last_undegraded":"2024-03-04T12:24:05.955512+0000","last_fullsized":"2024-03-04T12:24:05.955512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:33:03.487273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029026+0000","last_change":"2024-03-04T12:23:12.922601+0000","last_active":"2024-03-04T12:24:06.029026+0000","last_peered":"2024-03-04T12:24:06.029026+0000","last_clean":"2024-03-04T12:24:06.029026+0000","last_became_active":"2024-03-04T12:23:12.922037+0000","last_became_peered":"2024-03-04T12:23:12.922037+0000","last_unstale":"2024-03-04T12:24:06.029026+0000","last_undegraded":"2024-03-04T12:24:06.029026+0000","last_fullsized":"2024-03-04T12:24:06.029026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:32.076439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954563+0000","last_change":"2024-03-04T12:23:12.916308+0000","last_active":"2024-03-04T12:24:05.954563+0000","last_peered":"2024-03-04T12:24:05.954563+0000","last_clean":"2024-03-04T12:24:05.954563+0000","last_became_active":"2024-03-04T12:23:12.915897+0000","last_became_peered":"2024-03-04T12:23:12.915897+0000","last_unstale":"2024-03-04T12:24:05.954563+0000","last_undegraded":"2024-03-04T12:24:05.954563+0000","last_fullsized":"2024-03-04T12:24:05.954563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:15:53.133053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955676+0000","last_change":"2024-03-04T12:23:12.930700+0000","last_active":"2024-03-04T12:24:05.955676+0000","last_peered":"2024-03-04T12:24:05.955676+0000","last_clean":"2024-03-04T12:24:05.955676+0000","last_became_active":"2024-03-04T12:23:12.930328+0000","last_became_peered":"2024-03-04T12:23:12.930328+0000","last_unstale":"2024-03-04T12:24:05.955676+0000","last_undegraded":"2024-03-04T12:24:05.955676+0000","last_fullsized":"2024-03-04T12:24:05.955676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:29.990527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953334+0000","last_change":"2024-03-04T12:23:12.915471+0000","last_active":"2024-03-04T12:24:05.953334+0000","last_peered":"2024-03-04T12:24:05.953334+0000","last_clean":"2024-03-04T12:24:05.953334+0000","last_became_active":"2024-03-04T12:23:12.913627+0000","last_became_peered":"2024-03-04T12:23:12.913627+0000","last_unstale":"2024-03-04T12:24:05.953334+0000","last_undegraded":"2024-03-04T12:24:05.953334+0000","last_fullsized":"2024-03-04T12:24:05.953334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:42:56.762782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028130+0000","last_change":"2024-03-04T12:23:12.899143+0000","last_active":"2024-03-04T12:24:06.028130+0000","last_peered":"2024-03-04T12:24:06.028130+0000","last_clean":"2024-03-04T12:24:06.028130+0000","last_became_active":"2024-03-04T12:23:12.898498+0000","last_became_peered":"2024-03-04T12:23:12.898498+0000","last_unstale":"2024-03-04T12:24:06.028130+0000","last_undegraded":"2024-03-04T12:24:06.028130+0000","last_fullsized":"2024-03-04T12:24:06.028130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:33.840683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955353+0000","last_change":"2024-03-04T12:23:12.920873+0000","last_active":"2024-03-04T12:24:05.955353+0000","last_peered":"2024-03-04T12:24:05.955353+0000","last_clean":"2024-03-04T12:24:05.955353+0000","last_became_active":"2024-03-04T12:23:12.919080+0000","last_became_peered":"2024-03-04T12:23:12.919080+0000","last_unstale":"2024-03-04T12:24:05.955353+0000","last_undegraded":"2024-03-04T12:24:05.955353+0000","last_fullsized":"2024-03-04T12:24:05.955353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:47:26.710624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.959024+0000","last_change":"2024-03-04T12:24:05.959024+0000","last_active":"2024-03-04T12:24:05.959024+0000","last_peered":"2024-03-04T12:24:05.959024+0000","last_clean":"2024-03-04T12:24:05.959024+0000","last_became_active":"2024-03-04T12:24:05.958750+0000","last_became_peered":"2024-03-04T12:24:05.958750+0000","last_unstale":"2024-03-04T12:24:05.959024+0000","last_undegraded":"2024-03-04T12:24:05.959024+0000","last_fullsized":"2024-03-04T12:24:05.959024+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:59:33.263357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028464+0000","last_change":"2024-03-04T12:23:12.923860+0000","last_active":"2024-03-04T12:24:06.028464+0000","last_peered":"2024-03-04T12:24:06.028464+0000","last_clean":"2024-03-04T12:24:06.028464+0000","last_became_active":"2024-03-04T12:23:12.923530+0000","last_became_peered":"2024-03-04T12:23:12.923530+0000","last_unstale":"2024-03-04T12:24:06.028464+0000","last_undegraded":"2024-03-04T12:24:06.028464+0000","last_fullsized":"2024-03-04T12:24:06.028464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:30:01.466458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955642+0000","last_change":"2024-03-04T12:23:12.927159+0000","last_active":"2024-03-04T12:24:05.955642+0000","last_peered":"2024-03-04T12:24:05.955642+0000","last_clean":"2024-03-04T12:24:05.955642+0000","last_became_active":"2024-03-04T12:23:12.926800+0000","last_became_peered":"2024-03-04T12:23:12.926800+0000","last_unstale":"2024-03-04T12:24:05.955642+0000","last_undegraded":"2024-03-04T12:24:05.955642+0000","last_fullsized":"2024-03-04T12:24:05.955642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:52.633648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952564+0000","last_change":"2024-03-04T12:23:12.920607+0000","last_active":"2024-03-04T12:24:05.952564+0000","last_peered":"2024-03-04T12:24:05.952564+0000","last_clean":"2024-03-04T12:24:05.952564+0000","last_became_active":"2024-03-04T12:23:12.920257+0000","last_became_peered":"2024-03-04T12:23:12.920257+0000","last_unstale":"2024-03-04T12:24:05.952564+0000","last_undegraded":"2024-03-04T12:24:05.952564+0000","last_fullsized":"2024-03-04T12:24:05.952564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:39.353673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953934+0000","last_change":"2024-03-04T12:23:12.906635+0000","last_active":"2024-03-04T12:24:05.953934+0000","last_peered":"2024-03-04T12:24:05.953934+0000","last_clean":"2024-03-04T12:24:05.953934+0000","last_became_active":"2024-03-04T12:23:12.906243+0000","last_became_peered":"2024-03-04T12:23:12.906243+0000","last_unstale":"2024-03-04T12:24:05.953934+0000","last_undegraded":"2024-03-04T12:24:05.953934+0000","last_fullsized":"2024-03-04T12:24:05.953934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:44.375934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953355+0000","last_change":"2024-03-04T12:23:16.951685+0000","last_active":"2024-03-04T12:24:05.953355+0000","last_peered":"2024-03-04T12:24:05.953355+0000","last_clean":"2024-03-04T12:24:05.953355+0000","last_became_active":"2024-03-04T12:23:16.951329+0000","last_became_peered":"2024-03-04T12:23:16.951329+0000","last_unstale":"2024-03-04T12:24:05.953355+0000","last_undegraded":"2024-03-04T12:24:05.953355+0000","last_fullsized":"2024-03-04T12:24:05.953355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:03:27.233466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":110,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:49.007039+0000","last_change":"2024-03-04T12:23:11.900681+0000","last_active":"2024-03-04T12:24:49.007039+0000","last_peered":"2024-03-04T12:24:49.007039+0000","last_clean":"2024-03-04T12:24:49.007039+0000","last_became_active":"2024-03-04T12:23:11.900365+0000","last_became_peered":"2024-03-04T12:23:11.900365+0000","last_unstale":"2024-03-04T12:24:49.007039+0000","last_undegraded":"2024-03-04T12:24:49.007039+0000","last_fullsized":"2024-03-04T12:24:49.007039+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:53.052513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.030707+0000","last_change":"2024-03-04T12:24:06.030707+0000","last_active":"2024-03-04T12:24:06.030707+0000","last_peered":"2024-03-04T12:24:06.030707+0000","last_clean":"2024-03-04T12:24:06.030707+0000","last_became_active":"2024-03-04T12:24:06.030306+0000","last_became_peered":"2024-03-04T12:24:06.030306+0000","last_unstale":"2024-03-04T12:24:06.030707+0000","last_undegraded":"2024-03-04T12:24:06.030707+0000","last_fullsized":"2024-03-04T12:24:06.030707+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:59:18.939734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952650+0000","last_change":"2024-03-04T12:23:16.953704+0000","last_active":"2024-03-04T12:24:05.952650+0000","last_peered":"2024-03-04T12:24:05.952650+0000","last_clean":"2024-03-04T12:24:05.952650+0000","last_became_active":"2024-03-04T12:23:16.953177+0000","last_became_peered":"2024-03-04T12:23:16.953177+0000","last_unstale":"2024-03-04T12:24:05.952650+0000","last_undegraded":"2024-03-04T12:24:05.952650+0000","last_fullsized":"2024-03-04T12:24:05.952650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:37:00.469767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955828+0000","last_change":"2024-03-04T12:23:11.904271+0000","last_active":"2024-03-04T12:24:05.955828+0000","last_peered":"2024-03-04T12:24:05.955828+0000","last_clean":"2024-03-04T12:24:05.955828+0000","last_became_active":"2024-03-04T12:23:11.903963+0000","last_became_peered":"2024-03-04T12:23:11.903963+0000","last_unstale":"2024-03-04T12:24:05.955828+0000","last_undegraded":"2024-03-04T12:24:05.955828+0000","last_fullsized":"2024-03-04T12:24:05.955828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:44.774621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.956538+0000","last_change":"2024-03-04T12:23:12.925750+0000","last_active":"2024-03-04T12:24:05.956538+0000","last_peered":"2024-03-04T12:24:05.956538+0000","last_clean":"2024-03-04T12:24:05.956538+0000","last_became_active":"2024-03-04T12:23:12.925380+0000","last_became_peered":"2024-03-04T12:23:12.925380+0000","last_unstale":"2024-03-04T12:24:05.956538+0000","last_undegraded":"2024-03-04T12:24:05.956538+0000","last_fullsized":"2024-03-04T12:24:05.956538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:14.665127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952468+0000","last_change":"2024-03-04T12:23:16.952303+0000","last_active":"2024-03-04T12:24:05.952468+0000","last_peered":"2024-03-04T12:24:05.952468+0000","last_clean":"2024-03-04T12:24:05.952468+0000","last_became_active":"2024-03-04T12:23:16.951666+0000","last_became_peered":"2024-03-04T12:23:16.951666+0000","last_unstale":"2024-03-04T12:24:05.952468+0000","last_undegraded":"2024-03-04T12:24:05.952468+0000","last_fullsized":"2024-03-04T12:24:05.952468+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:41.326808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.959106+0000","last_change":"2024-03-04T12:23:11.903871+0000","last_active":"2024-03-04T12:24:05.959106+0000","last_peered":"2024-03-04T12:24:05.959106+0000","last_clean":"2024-03-04T12:24:05.959106+0000","last_became_active":"2024-03-04T12:23:11.902735+0000","last_became_peered":"2024-03-04T12:23:11.902735+0000","last_unstale":"2024-03-04T12:24:05.959106+0000","last_undegraded":"2024-03-04T12:24:05.959106+0000","last_fullsized":"2024-03-04T12:24:05.959106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:45:09.897740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028321+0000","last_change":"2024-03-04T12:23:12.924504+0000","last_active":"2024-03-04T12:24:06.028321+0000","last_peered":"2024-03-04T12:24:06.028321+0000","last_clean":"2024-03-04T12:24:06.028321+0000","last_became_active":"2024-03-04T12:23:12.922934+0000","last_became_peered":"2024-03-04T12:23:12.922934+0000","last_unstale":"2024-03-04T12:24:06.028321+0000","last_undegraded":"2024-03-04T12:24:06.028321+0000","last_fullsized":"2024-03-04T12:24:06.028321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:28.974601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952232+0000","last_change":"2024-03-04T12:23:16.952324+0000","last_active":"2024-03-04T12:24:05.952232+0000","last_peered":"2024-03-04T12:24:05.952232+0000","last_clean":"2024-03-04T12:24:05.952232+0000","last_became_active":"2024-03-04T12:23:16.951145+0000","last_became_peered":"2024-03-04T12:23:16.951145+0000","last_unstale":"2024-03-04T12:24:05.952232+0000","last_undegraded":"2024-03-04T12:24:05.952232+0000","last_fullsized":"2024-03-04T12:24:05.952232+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:20:47.029523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.958858+0000","last_change":"2024-03-04T12:23:11.904391+0000","last_active":"2024-03-04T12:24:05.958858+0000","last_peered":"2024-03-04T12:24:05.958858+0000","last_clean":"2024-03-04T12:24:05.958858+0000","last_became_active":"2024-03-04T12:23:11.903244+0000","last_became_peered":"2024-03-04T12:23:11.903244+0000","last_unstale":"2024-03-04T12:24:05.958858+0000","last_undegraded":"2024-03-04T12:24:05.958858+0000","last_fullsized":"2024-03-04T12:24:05.958858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:27.590418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029058+0000","last_change":"2024-03-04T12:23:12.925623+0000","last_active":"2024-03-04T12:24:06.029058+0000","last_peered":"2024-03-04T12:24:06.029058+0000","last_clean":"2024-03-04T12:24:06.029058+0000","last_became_active":"2024-03-04T12:23:12.925243+0000","last_became_peered":"2024-03-04T12:23:12.925243+0000","last_unstale":"2024-03-04T12:24:06.029058+0000","last_undegraded":"2024-03-04T12:24:06.029058+0000","last_fullsized":"2024-03-04T12:24:06.029058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:11:04.142655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953361+0000","last_change":"2024-03-04T12:23:16.946064+0000","last_active":"2024-03-04T12:24:05.953361+0000","last_peered":"2024-03-04T12:24:05.953361+0000","last_clean":"2024-03-04T12:24:05.953361+0000","last_became_active":"2024-03-04T12:23:16.945342+0000","last_became_peered":"2024-03-04T12:23:16.945342+0000","last_unstale":"2024-03-04T12:24:05.953361+0000","last_undegraded":"2024-03-04T12:24:05.953361+0000","last_fullsized":"2024-03-04T12:24:05.953361+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:43:37.500216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:50.997352+0000","last_change":"2024-03-04T12:23:11.899478+0000","last_active":"2024-03-04T12:24:50.997352+0000","last_peered":"2024-03-04T12:24:50.997352+0000","last_clean":"2024-03-04T12:24:50.997352+0000","last_became_active":"2024-03-04T12:23:11.899153+0000","last_became_peered":"2024-03-04T12:23:11.899153+0000","last_unstale":"2024-03-04T12:24:50.997352+0000","last_undegraded":"2024-03-04T12:24:50.997352+0000","last_fullsized":"2024-03-04T12:24:50.997352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:10.709054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.958202+0000","last_change":"2024-03-04T12:23:12.908560+0000","last_active":"2024-03-04T12:24:05.958202+0000","last_peered":"2024-03-04T12:24:05.958202+0000","last_clean":"2024-03-04T12:24:05.958202+0000","last_became_active":"2024-03-04T12:23:12.908221+0000","last_became_peered":"2024-03-04T12:23:12.908221+0000","last_unstale":"2024-03-04T12:24:05.958202+0000","last_undegraded":"2024-03-04T12:24:05.958202+0000","last_fullsized":"2024-03-04T12:24:05.958202+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:28:06.545945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.031257+0000","last_change":"2024-03-04T12:24:06.031257+0000","last_active":"2024-03-04T12:24:06.031257+0000","last_peered":"2024-03-04T12:24:06.031257+0000","last_clean":"2024-03-04T12:24:06.031257+0000","last_became_active":"2024-03-04T12:24:06.030409+0000","last_became_peered":"2024-03-04T12:24:06.030409+0000","last_unstale":"2024-03-04T12:24:06.031257+0000","last_undegraded":"2024-03-04T12:24:06.031257+0000","last_fullsized":"2024-03-04T12:24:06.031257+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:31.403224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954320+0000","last_change":"2024-03-04T12:23:11.898360+0000","last_active":"2024-03-04T12:24:05.954320+0000","last_peered":"2024-03-04T12:24:05.954320+0000","last_clean":"2024-03-04T12:24:05.954320+0000","last_became_active":"2024-03-04T12:23:11.897988+0000","last_became_peered":"2024-03-04T12:23:11.897988+0000","last_unstale":"2024-03-04T12:24:05.954320+0000","last_undegraded":"2024-03-04T12:24:05.954320+0000","last_fullsized":"2024-03-04T12:24:05.954320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:17.448032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028454+0000","last_change":"2024-03-04T12:23:12.918392+0000","last_active":"2024-03-04T12:24:06.028454+0000","last_peered":"2024-03-04T12:24:06.028454+0000","last_clean":"2024-03-04T12:24:06.028454+0000","last_became_active":"2024-03-04T12:23:12.917656+0000","last_became_peered":"2024-03-04T12:23:12.917656+0000","last_unstale":"2024-03-04T12:24:06.028454+0000","last_undegraded":"2024-03-04T12:24:06.028454+0000","last_fullsized":"2024-03-04T12:24:06.028454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:44:14.545925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955607+0000","last_change":"2024-03-04T12:23:16.941841+0000","last_active":"2024-03-04T12:24:05.955607+0000","last_peered":"2024-03-04T12:24:05.955607+0000","last_clean":"2024-03-04T12:24:05.955607+0000","last_became_active":"2024-03-04T12:23:16.941449+0000","last_became_peered":"2024-03-04T12:23:16.941449+0000","last_unstale":"2024-03-04T12:24:05.955607+0000","last_undegraded":"2024-03-04T12:24:05.955607+0000","last_fullsized":"2024-03-04T12:24:05.955607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:50:35.451834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028130+0000","last_change":"2024-03-04T12:23:11.906723+0000","last_active":"2024-03-04T12:24:06.028130+0000","last_peered":"2024-03-04T12:24:06.028130+0000","last_clean":"2024-03-04T12:24:06.028130+0000","last_became_active":"2024-03-04T12:23:11.906357+0000","last_became_peered":"2024-03-04T12:23:11.906357+0000","last_unstale":"2024-03-04T12:24:06.028130+0000","last_undegraded":"2024-03-04T12:24:06.028130+0000","last_fullsized":"2024-03-04T12:24:06.028130+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:09.592236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955549+0000","last_change":"2024-03-04T12:23:12.916435+0000","last_active":"2024-03-04T12:24:05.955549+0000","last_peered":"2024-03-04T12:24:05.955549+0000","last_clean":"2024-03-04T12:24:05.955549+0000","last_became_active":"2024-03-04T12:23:12.915607+0000","last_became_peered":"2024-03-04T12:23:12.915607+0000","last_unstale":"2024-03-04T12:24:05.955549+0000","last_undegraded":"2024-03-04T12:24:05.955549+0000","last_fullsized":"2024-03-04T12:24:05.955549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:41:32.900344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954910+0000","last_change":"2024-03-04T12:23:16.944342+0000","last_active":"2024-03-04T12:24:05.954910+0000","last_peered":"2024-03-04T12:24:05.954910+0000","last_clean":"2024-03-04T12:24:05.954910+0000","last_became_active":"2024-03-04T12:23:16.943683+0000","last_became_peered":"2024-03-04T12:23:16.943683+0000","last_unstale":"2024-03-04T12:24:05.954910+0000","last_undegraded":"2024-03-04T12:24:05.954910+0000","last_fullsized":"2024-03-04T12:24:05.954910+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:19.658860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028539+0000","last_change":"2024-03-04T12:23:11.899194+0000","last_active":"2024-03-04T12:24:06.028539+0000","last_peered":"2024-03-04T12:24:06.028539+0000","last_clean":"2024-03-04T12:24:06.028539+0000","last_became_active":"2024-03-04T12:23:11.898109+0000","last_became_peered":"2024-03-04T12:23:11.898109+0000","last_unstale":"2024-03-04T12:24:06.028539+0000","last_undegraded":"2024-03-04T12:24:06.028539+0000","last_fullsized":"2024-03-04T12:24:06.028539+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:08.808202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.951684+0000","last_change":"2024-03-04T12:23:12.916549+0000","last_active":"2024-03-04T12:24:05.951684+0000","last_peered":"2024-03-04T12:24:05.951684+0000","last_clean":"2024-03-04T12:24:05.951684+0000","last_became_active":"2024-03-04T12:23:12.916204+0000","last_became_peered":"2024-03-04T12:23:12.916204+0000","last_unstale":"2024-03-04T12:24:05.951684+0000","last_undegraded":"2024-03-04T12:24:05.951684+0000","last_fullsized":"2024-03-04T12:24:05.951684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:56.412600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955949+0000","last_change":"2024-03-04T12:23:16.945903+0000","last_active":"2024-03-04T12:24:05.955949+0000","last_peered":"2024-03-04T12:24:05.955949+0000","last_clean":"2024-03-04T12:24:05.955949+0000","last_became_active":"2024-03-04T12:23:16.945535+0000","last_became_peered":"2024-03-04T12:23:16.945535+0000","last_unstale":"2024-03-04T12:24:05.955949+0000","last_undegraded":"2024-03-04T12:24:05.955949+0000","last_fullsized":"2024-03-04T12:24:05.955949+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:45:01.178370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":175,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:51.998164+0000","last_change":"2024-03-04T12:23:11.901570+0000","last_active":"2024-03-04T12:24:51.998164+0000","last_peered":"2024-03-04T12:24:51.998164+0000","last_clean":"2024-03-04T12:24:51.998164+0000","last_became_active":"2024-03-04T12:23:11.900702+0000","last_became_peered":"2024-03-04T12:23:11.900702+0000","last_unstale":"2024-03-04T12:24:51.998164+0000","last_undegraded":"2024-03-04T12:24:51.998164+0000","last_fullsized":"2024-03-04T12:24:51.998164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:18.758453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028696+0000","last_change":"2024-03-04T12:23:12.919571+0000","last_active":"2024-03-04T12:24:06.028696+0000","last_peered":"2024-03-04T12:24:06.028696+0000","last_clean":"2024-03-04T12:24:06.028696+0000","last_became_active":"2024-03-04T12:23:12.919143+0000","last_became_peered":"2024-03-04T12:23:12.919143+0000","last_unstale":"2024-03-04T12:24:06.028696+0000","last_undegraded":"2024-03-04T12:24:06.028696+0000","last_fullsized":"2024-03-04T12:24:06.028696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:25.459417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.951599+0000","last_change":"2024-03-04T12:23:16.946380+0000","last_active":"2024-03-04T12:24:05.951599+0000","last_peered":"2024-03-04T12:24:05.951599+0000","last_clean":"2024-03-04T12:24:05.951599+0000","last_became_active":"2024-03-04T12:23:16.945983+0000","last_became_peered":"2024-03-04T12:23:16.945983+0000","last_unstale":"2024-03-04T12:24:05.951599+0000","last_undegraded":"2024-03-04T12:24:05.951599+0000","last_fullsized":"2024-03-04T12:24:05.951599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:39:43.297619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955609+0000","last_change":"2024-03-04T12:23:11.897176+0000","last_active":"2024-03-04T12:24:05.955609+0000","last_peered":"2024-03-04T12:24:05.955609+0000","last_clean":"2024-03-04T12:24:05.955609+0000","last_became_active":"2024-03-04T12:23:11.896747+0000","last_became_peered":"2024-03-04T12:23:11.896747+0000","last_unstale":"2024-03-04T12:24:05.955609+0000","last_undegraded":"2024-03-04T12:24:05.955609+0000","last_fullsized":"2024-03-04T12:24:05.955609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:02.206704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.956455+0000","last_change":"2024-03-04T12:23:12.922021+0000","last_active":"2024-03-04T12:24:05.956455+0000","last_peered":"2024-03-04T12:24:05.956455+0000","last_clean":"2024-03-04T12:24:05.956455+0000","last_became_active":"2024-03-04T12:23:12.918724+0000","last_became_peered":"2024-03-04T12:23:12.918724+0000","last_unstale":"2024-03-04T12:24:05.956455+0000","last_undegraded":"2024-03-04T12:24:05.956455+0000","last_fullsized":"2024-03-04T12:24:05.956455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:49:09.017979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952895+0000","last_change":"2024-03-04T12:23:16.943271+0000","last_active":"2024-03-04T12:24:05.952895+0000","last_peered":"2024-03-04T12:24:05.952895+0000","last_clean":"2024-03-04T12:24:05.952895+0000","last_became_active":"2024-03-04T12:23:16.942400+0000","last_became_peered":"2024-03-04T12:23:16.942400+0000","last_unstale":"2024-03-04T12:24:05.952895+0000","last_undegraded":"2024-03-04T12:24:05.952895+0000","last_fullsized":"2024-03-04T12:24:05.952895+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:06.010960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028932+0000","last_change":"2024-03-04T12:23:11.896706+0000","last_active":"2024-03-04T12:24:06.028932+0000","last_peered":"2024-03-04T12:24:06.028932+0000","last_clean":"2024-03-04T12:24:06.028932+0000","last_became_active":"2024-03-04T12:23:11.896100+0000","last_became_peered":"2024-03-04T12:23:11.896100+0000","last_unstale":"2024-03-04T12:24:06.028932+0000","last_undegraded":"2024-03-04T12:24:06.028932+0000","last_fullsized":"2024-03-04T12:24:06.028932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:42:19.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028482+0000","last_change":"2024-03-04T12:23:12.923631+0000","last_active":"2024-03-04T12:24:06.028482+0000","last_peered":"2024-03-04T12:24:06.028482+0000","last_clean":"2024-03-04T12:24:06.028482+0000","last_became_active":"2024-03-04T12:23:12.922487+0000","last_became_peered":"2024-03-04T12:23:12.922487+0000","last_unstale":"2024-03-04T12:24:06.028482+0000","last_undegraded":"2024-03-04T12:24:06.028482+0000","last_fullsized":"2024-03-04T12:24:06.028482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:30:33.923064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.032470+0000","last_change":"2024-03-04T12:24:06.032075+0000","last_active":"2024-03-04T12:24:06.032470+0000","last_peered":"2024-03-04T12:24:06.032470+0000","last_clean":"2024-03-04T12:24:06.032470+0000","last_became_active":"2024-03-04T12:24:06.031639+0000","last_became_peered":"2024-03-04T12:24:06.031639+0000","last_unstale":"2024-03-04T12:24:06.032470+0000","last_undegraded":"2024-03-04T12:24:06.032470+0000","last_fullsized":"2024-03-04T12:24:06.032470+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:44:48.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'34","reported_seq":214,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:51.998187+0000","last_change":"2024-03-04T12:23:11.898903+0000","last_active":"2024-03-04T12:24:51.998187+0000","last_peered":"2024-03-04T12:24:51.998187+0000","last_clean":"2024-03-04T12:24:51.998187+0000","last_became_active":"2024-03-04T12:23:11.898477+0000","last_became_peered":"2024-03-04T12:23:11.898477+0000","last_unstale":"2024-03-04T12:24:51.998187+0000","last_undegraded":"2024-03-04T12:24:51.998187+0000","last_fullsized":"2024-03-04T12:24:51.998187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:36:58.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60269,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":153,"num_read_kb":135,"num_write":36,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.958633+0000","last_change":"2024-03-04T12:23:12.908497+0000","last_active":"2024-03-04T12:24:05.958633+0000","last_peered":"2024-03-04T12:24:05.958633+0000","last_clean":"2024-03-04T12:24:05.958633+0000","last_became_active":"2024-03-04T12:23:12.907847+0000","last_became_peered":"2024-03-04T12:23:12.907847+0000","last_unstale":"2024-03-04T12:24:05.958633+0000","last_undegraded":"2024-03-04T12:24:05.958633+0000","last_fullsized":"2024-03-04T12:24:05.958633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:51:44.592851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029273+0000","last_change":"2024-03-04T12:23:16.951844+0000","last_active":"2024-03-04T12:24:06.029273+0000","last_peered":"2024-03-04T12:24:06.029273+0000","last_clean":"2024-03-04T12:24:06.029273+0000","last_became_active":"2024-03-04T12:23:16.951491+0000","last_became_peered":"2024-03-04T12:23:16.951491+0000","last_unstale":"2024-03-04T12:24:06.029273+0000","last_undegraded":"2024-03-04T12:24:06.029273+0000","last_fullsized":"2024-03-04T12:24:06.029273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:54.098239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953662+0000","last_change":"2024-03-04T12:23:11.899592+0000","last_active":"2024-03-04T12:24:05.953662+0000","last_peered":"2024-03-04T12:24:05.953662+0000","last_clean":"2024-03-04T12:24:05.953662+0000","last_became_active":"2024-03-04T12:23:11.899275+0000","last_became_peered":"2024-03-04T12:23:11.899275+0000","last_unstale":"2024-03-04T12:24:05.953662+0000","last_undegraded":"2024-03-04T12:24:05.953662+0000","last_fullsized":"2024-03-04T12:24:05.953662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:02.184268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028014+0000","last_change":"2024-03-04T12:23:12.910084+0000","last_active":"2024-03-04T12:24:06.028014+0000","last_peered":"2024-03-04T12:24:06.028014+0000","last_clean":"2024-03-04T12:24:06.028014+0000","last_became_active":"2024-03-04T12:23:12.909719+0000","last_became_peered":"2024-03-04T12:23:12.909719+0000","last_unstale":"2024-03-04T12:24:06.028014+0000","last_undegraded":"2024-03-04T12:24:06.028014+0000","last_fullsized":"2024-03-04T12:24:06.028014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:16:41.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029579+0000","last_change":"2024-03-04T12:24:06.029579+0000","last_active":"2024-03-04T12:24:06.029579+0000","last_peered":"2024-03-04T12:24:06.029579+0000","last_clean":"2024-03-04T12:24:06.029579+0000","last_became_active":"2024-03-04T12:24:06.029042+0000","last_became_peered":"2024-03-04T12:24:06.029042+0000","last_unstale":"2024-03-04T12:24:06.029579+0000","last_undegraded":"2024-03-04T12:24:06.029579+0000","last_fullsized":"2024-03-04T12:24:06.029579+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:59:35.867212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029279+0000","last_change":"2024-03-04T12:23:11.897506+0000","last_active":"2024-03-04T12:24:06.029279+0000","last_peered":"2024-03-04T12:24:06.029279+0000","last_clean":"2024-03-04T12:24:06.029279+0000","last_became_active":"2024-03-04T12:23:11.897090+0000","last_became_peered":"2024-03-04T12:23:11.897090+0000","last_unstale":"2024-03-04T12:24:06.029279+0000","last_undegraded":"2024-03-04T12:24:06.029279+0000","last_fullsized":"2024-03-04T12:24:06.029279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:10:27.868713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955402+0000","last_change":"2024-03-04T12:23:12.921825+0000","last_active":"2024-03-04T12:24:05.955402+0000","last_peered":"2024-03-04T12:24:05.955402+0000","last_clean":"2024-03-04T12:24:05.955402+0000","last_became_active":"2024-03-04T12:23:12.921491+0000","last_became_peered":"2024-03-04T12:23:12.921491+0000","last_unstale":"2024-03-04T12:24:05.955402+0000","last_undegraded":"2024-03-04T12:24:05.955402+0000","last_fullsized":"2024-03-04T12:24:05.955402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:17:08.203502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952170+0000","last_change":"2024-03-04T12:23:16.951920+0000","last_active":"2024-03-04T12:24:05.952170+0000","last_peered":"2024-03-04T12:24:05.952170+0000","last_clean":"2024-03-04T12:24:05.952170+0000","last_became_active":"2024-03-04T12:23:16.951507+0000","last_became_peered":"2024-03-04T12:23:16.951507+0000","last_unstale":"2024-03-04T12:24:05.952170+0000","last_undegraded":"2024-03-04T12:24:05.952170+0000","last_fullsized":"2024-03-04T12:24:05.952170+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:27.673692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:50.996588+0000","last_change":"2024-03-04T12:23:11.897722+0000","last_active":"2024-03-04T12:24:50.996588+0000","last_peered":"2024-03-04T12:24:50.996588+0000","last_clean":"2024-03-04T12:24:50.996588+0000","last_became_active":"2024-03-04T12:23:11.897276+0000","last_became_peered":"2024-03-04T12:23:11.897276+0000","last_unstale":"2024-03-04T12:24:50.996588+0000","last_undegraded":"2024-03-04T12:24:50.996588+0000","last_fullsized":"2024-03-04T12:24:50.996588+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:37.968804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":77,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955021+0000","last_change":"2024-03-04T12:23:12.914773+0000","last_active":"2024-03-04T12:24:05.955021+0000","last_peered":"2024-03-04T12:24:05.955021+0000","last_clean":"2024-03-04T12:24:05.955021+0000","last_became_active":"2024-03-04T12:23:12.914431+0000","last_became_peered":"2024-03-04T12:23:12.914431+0000","last_unstale":"2024-03-04T12:24:05.955021+0000","last_undegraded":"2024-03-04T12:24:05.955021+0000","last_fullsized":"2024-03-04T12:24:05.955021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:50.234491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028425+0000","last_change":"2024-03-04T12:23:16.952405+0000","last_active":"2024-03-04T12:24:06.028425+0000","last_peered":"2024-03-04T12:24:06.028425+0000","last_clean":"2024-03-04T12:24:06.028425+0000","last_became_active":"2024-03-04T12:23:16.951915+0000","last_became_peered":"2024-03-04T12:23:16.951915+0000","last_unstale":"2024-03-04T12:24:06.028425+0000","last_undegraded":"2024-03-04T12:24:06.028425+0000","last_fullsized":"2024-03-04T12:24:06.028425+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:38:03.765883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.957615+0000","last_change":"2024-03-04T12:23:11.903267+0000","last_active":"2024-03-04T12:24:05.957615+0000","last_peered":"2024-03-04T12:24:05.957615+0000","last_clean":"2024-03-04T12:24:05.957615+0000","last_became_active":"2024-03-04T12:23:11.902870+0000","last_became_peered":"2024-03-04T12:23:11.902870+0000","last_unstale":"2024-03-04T12:24:05.957615+0000","last_undegraded":"2024-03-04T12:24:05.957615+0000","last_fullsized":"2024-03-04T12:24:05.957615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:33:43.679869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028314+0000","last_change":"2024-03-04T12:23:12.925303+0000","last_active":"2024-03-04T12:24:06.028314+0000","last_peered":"2024-03-04T12:24:06.028314+0000","last_clean":"2024-03-04T12:24:06.028314+0000","last_became_active":"2024-03-04T12:23:12.924945+0000","last_became_peered":"2024-03-04T12:23:12.924945+0000","last_unstale":"2024-03-04T12:24:06.028314+0000","last_undegraded":"2024-03-04T12:24:06.028314+0000","last_fullsized":"2024-03-04T12:24:06.028314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:49.680925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028148+0000","last_change":"2024-03-04T12:23:16.951457+0000","last_active":"2024-03-04T12:24:06.028148+0000","last_peered":"2024-03-04T12:24:06.028148+0000","last_clean":"2024-03-04T12:24:06.028148+0000","last_became_active":"2024-03-04T12:23:16.950835+0000","last_became_peered":"2024-03-04T12:23:16.950835+0000","last_unstale":"2024-03-04T12:24:06.028148+0000","last_undegraded":"2024-03-04T12:24:06.028148+0000","last_fullsized":"2024-03-04T12:24:06.028148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:39.337275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.959083+0000","last_change":"2024-03-04T12:23:11.903718+0000","last_active":"2024-03-04T12:24:05.959083+0000","last_peered":"2024-03-04T12:24:05.959083+0000","last_clean":"2024-03-04T12:24:05.959083+0000","last_became_active":"2024-03-04T12:23:11.902814+0000","last_became_peered":"2024-03-04T12:23:11.902814+0000","last_unstale":"2024-03-04T12:24:05.959083+0000","last_undegraded":"2024-03-04T12:24:05.959083+0000","last_fullsized":"2024-03-04T12:24:05.959083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:40:30.978576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.958392+0000","last_change":"2024-03-04T12:24:05.958392+0000","last_active":"2024-03-04T12:24:05.958392+0000","last_peered":"2024-03-04T12:24:05.958392+0000","last_clean":"2024-03-04T12:24:05.958392+0000","last_became_active":"2024-03-04T12:24:05.958012+0000","last_became_peered":"2024-03-04T12:24:05.958012+0000","last_unstale":"2024-03-04T12:24:05.958392+0000","last_undegraded":"2024-03-04T12:24:05.958392+0000","last_fullsized":"2024-03-04T12:24:05.958392+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:43:50.400090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.958152+0000","last_change":"2024-03-04T12:23:16.950046+0000","last_active":"2024-03-04T12:24:05.958152+0000","last_peered":"2024-03-04T12:24:05.958152+0000","last_clean":"2024-03-04T12:24:05.958152+0000","last_became_active":"2024-03-04T12:23:16.949632+0000","last_became_peered":"2024-03-04T12:23:16.949632+0000","last_unstale":"2024-03-04T12:24:05.958152+0000","last_undegraded":"2024-03-04T12:24:05.958152+0000","last_fullsized":"2024-03-04T12:24:05.958152+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:25.817212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952134+0000","last_change":"2024-03-04T12:23:11.897016+0000","last_active":"2024-03-04T12:24:05.952134+0000","last_peered":"2024-03-04T12:24:05.952134+0000","last_clean":"2024-03-04T12:24:05.952134+0000","last_became_active":"2024-03-04T12:23:11.896668+0000","last_became_peered":"2024-03-04T12:23:11.896668+0000","last_unstale":"2024-03-04T12:24:05.952134+0000","last_undegraded":"2024-03-04T12:24:05.952134+0000","last_fullsized":"2024-03-04T12:24:05.952134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:50:01.089658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954296+0000","last_change":"2024-03-04T12:23:12.908169+0000","last_active":"2024-03-04T12:24:05.954296+0000","last_peered":"2024-03-04T12:24:05.954296+0000","last_clean":"2024-03-04T12:24:05.954296+0000","last_became_active":"2024-03-04T12:23:12.907786+0000","last_became_peered":"2024-03-04T12:23:12.907786+0000","last_unstale":"2024-03-04T12:24:05.954296+0000","last_undegraded":"2024-03-04T12:24:05.954296+0000","last_fullsized":"2024-03-04T12:24:05.954296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:10.602216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028165+0000","last_change":"2024-03-04T12:23:16.944232+0000","last_active":"2024-03-04T12:24:06.028165+0000","last_peered":"2024-03-04T12:24:06.028165+0000","last_clean":"2024-03-04T12:24:06.028165+0000","last_became_active":"2024-03-04T12:23:16.943855+0000","last_became_peered":"2024-03-04T12:23:16.943855+0000","last_unstale":"2024-03-04T12:24:06.028165+0000","last_undegraded":"2024-03-04T12:24:06.028165+0000","last_fullsized":"2024-03-04T12:24:06.028165+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:37:12.083622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:52.012166+0000","last_change":"2024-03-04T12:23:11.899742+0000","last_active":"2024-03-04T12:24:52.012166+0000","last_peered":"2024-03-04T12:24:52.012166+0000","last_clean":"2024-03-04T12:24:52.012166+0000","last_became_active":"2024-03-04T12:23:11.898445+0000","last_became_peered":"2024-03-04T12:23:11.898445+0000","last_unstale":"2024-03-04T12:24:52.012166+0000","last_undegraded":"2024-03-04T12:24:52.012166+0000","last_fullsized":"2024-03-04T12:24:52.012166+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:13.128348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954237+0000","last_change":"2024-03-04T12:23:12.921321+0000","last_active":"2024-03-04T12:24:05.954237+0000","last_peered":"2024-03-04T12:24:05.954237+0000","last_clean":"2024-03-04T12:24:05.954237+0000","last_became_active":"2024-03-04T12:23:12.918119+0000","last_became_peered":"2024-03-04T12:23:12.918119+0000","last_unstale":"2024-03-04T12:24:05.954237+0000","last_undegraded":"2024-03-04T12:24:05.954237+0000","last_fullsized":"2024-03-04T12:24:05.954237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:31.693583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028259+0000","last_change":"2024-03-04T12:23:16.943887+0000","last_active":"2024-03-04T12:24:06.028259+0000","last_peered":"2024-03-04T12:24:06.028259+0000","last_clean":"2024-03-04T12:24:06.028259+0000","last_became_active":"2024-03-04T12:23:16.943487+0000","last_became_peered":"2024-03-04T12:23:16.943487+0000","last_unstale":"2024-03-04T12:24:06.028259+0000","last_undegraded":"2024-03-04T12:24:06.028259+0000","last_fullsized":"2024-03-04T12:24:06.028259+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:12.451176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028804+0000","last_change":"2024-03-04T12:23:12.918363+0000","last_active":"2024-03-04T12:24:06.028804+0000","last_peered":"2024-03-04T12:24:06.028804+0000","last_clean":"2024-03-04T12:24:06.028804+0000","last_became_active":"2024-03-04T12:23:12.917563+0000","last_became_peered":"2024-03-04T12:23:12.917563+0000","last_unstale":"2024-03-04T12:24:06.028804+0000","last_undegraded":"2024-03-04T12:24:06.028804+0000","last_fullsized":"2024-03-04T12:24:06.028804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:26:35.297513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":107,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:51.997586+0000","last_change":"2024-03-04T12:23:11.904072+0000","last_active":"2024-03-04T12:24:51.997586+0000","last_peered":"2024-03-04T12:24:51.997586+0000","last_clean":"2024-03-04T12:24:51.997586+0000","last_became_active":"2024-03-04T12:23:11.903097+0000","last_became_peered":"2024-03-04T12:23:11.903097+0000","last_unstale":"2024-03-04T12:24:51.997586+0000","last_undegraded":"2024-03-04T12:24:51.997586+0000","last_fullsized":"2024-03-04T12:24:51.997586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:36.606667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955543+0000","last_change":"2024-03-04T12:23:16.951019+0000","last_active":"2024-03-04T12:24:05.955543+0000","last_peered":"2024-03-04T12:24:05.955543+0000","last_clean":"2024-03-04T12:24:05.955543+0000","last_became_active":"2024-03-04T12:23:16.950524+0000","last_became_peered":"2024-03-04T12:23:16.950524+0000","last_unstale":"2024-03-04T12:24:05.955543+0000","last_undegraded":"2024-03-04T12:24:05.955543+0000","last_fullsized":"2024-03-04T12:24:05.955543+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:48.376539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954963+0000","last_change":"2024-03-04T12:23:12.927615+0000","last_active":"2024-03-04T12:24:05.954963+0000","last_peered":"2024-03-04T12:24:05.954963+0000","last_clean":"2024-03-04T12:24:05.954963+0000","last_became_active":"2024-03-04T12:23:12.927190+0000","last_became_peered":"2024-03-04T12:23:12.927190+0000","last_unstale":"2024-03-04T12:24:05.954963+0000","last_undegraded":"2024-03-04T12:24:05.954963+0000","last_fullsized":"2024-03-04T12:24:05.954963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:54:19.019030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952706+0000","last_change":"2024-03-04T12:23:11.893542+0000","last_active":"2024-03-04T12:24:05.952706+0000","last_peered":"2024-03-04T12:24:05.952706+0000","last_clean":"2024-03-04T12:24:05.952706+0000","last_became_active":"2024-03-04T12:23:11.893166+0000","last_became_peered":"2024-03-04T12:23:11.893166+0000","last_unstale":"2024-03-04T12:24:05.952706+0000","last_undegraded":"2024-03-04T12:24:05.952706+0000","last_fullsized":"2024-03-04T12:24:05.952706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:32.117863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954934+0000","last_change":"2024-03-04T12:21:55.865103+0000","last_active":"2024-03-04T12:24:05.954934+0000","last_peered":"2024-03-04T12:24:05.954934+0000","last_clean":"2024-03-04T12:24:05.954934+0000","last_became_active":"2024-03-04T12:21:55.864553+0000","last_became_peered":"2024-03-04T12:21:55.864553+0000","last_unstale":"2024-03-04T12:24:05.954934+0000","last_undegraded":"2024-03-04T12:24:05.954934+0000","last_fullsized":"2024-03-04T12:24:05.954934+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_clean_scrub_stamp":"2024-03-04T12:18:28.442423+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:31.052782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029538+0000","last_change":"2024-03-04T12:23:16.945852+0000","last_active":"2024-03-04T12:24:06.029538+0000","last_peered":"2024-03-04T12:24:06.029538+0000","last_clean":"2024-03-04T12:24:06.029538+0000","last_became_active":"2024-03-04T12:23:16.945496+0000","last_became_peered":"2024-03-04T12:23:16.945496+0000","last_unstale":"2024-03-04T12:24:06.029538+0000","last_undegraded":"2024-03-04T12:24:06.029538+0000","last_fullsized":"2024-03-04T12:24:06.029538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:35:10.880757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028066+0000","last_change":"2024-03-04T12:23:12.919219+0000","last_active":"2024-03-04T12:24:06.028066+0000","last_peered":"2024-03-04T12:24:06.028066+0000","last_clean":"2024-03-04T12:24:06.028066+0000","last_became_active":"2024-03-04T12:23:12.918897+0000","last_became_peered":"2024-03-04T12:23:12.918897+0000","last_unstale":"2024-03-04T12:24:06.028066+0000","last_undegraded":"2024-03-04T12:24:06.028066+0000","last_fullsized":"2024-03-04T12:24:06.028066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:24:32.594172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954885+0000","last_change":"2024-03-04T12:23:11.894544+0000","last_active":"2024-03-04T12:24:05.954885+0000","last_peered":"2024-03-04T12:24:05.954885+0000","last_clean":"2024-03-04T12:24:05.954885+0000","last_became_active":"2024-03-04T12:23:11.894188+0000","last_became_peered":"2024-03-04T12:23:11.894188+0000","last_unstale":"2024-03-04T12:24:05.954885+0000","last_undegraded":"2024-03-04T12:24:05.954885+0000","last_fullsized":"2024-03-04T12:24:05.954885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:28:14.687530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955450+0000","last_change":"2024-03-04T12:23:16.945510+0000","last_active":"2024-03-04T12:24:05.955450+0000","last_peered":"2024-03-04T12:24:05.955450+0000","last_clean":"2024-03-04T12:24:05.955450+0000","last_became_active":"2024-03-04T12:23:16.945140+0000","last_became_peered":"2024-03-04T12:23:16.945140+0000","last_unstale":"2024-03-04T12:24:05.955450+0000","last_undegraded":"2024-03-04T12:24:05.955450+0000","last_fullsized":"2024-03-04T12:24:05.955450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:44.102503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.958797+0000","last_change":"2024-03-04T12:23:11.904515+0000","last_active":"2024-03-04T12:24:05.958797+0000","last_peered":"2024-03-04T12:24:05.958797+0000","last_clean":"2024-03-04T12:24:05.958797+0000","last_became_active":"2024-03-04T12:23:11.904149+0000","last_became_peered":"2024-03-04T12:23:11.904149+0000","last_unstale":"2024-03-04T12:24:05.958797+0000","last_undegraded":"2024-03-04T12:24:05.958797+0000","last_fullsized":"2024-03-04T12:24:05.958797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:04:51.382626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028279+0000","last_change":"2024-03-04T12:23:12.908749+0000","last_active":"2024-03-04T12:24:06.028279+0000","last_peered":"2024-03-04T12:24:06.028279+0000","last_clean":"2024-03-04T12:24:06.028279+0000","last_became_active":"2024-03-04T12:23:12.908405+0000","last_became_peered":"2024-03-04T12:23:12.908405+0000","last_unstale":"2024-03-04T12:24:06.028279+0000","last_undegraded":"2024-03-04T12:24:06.028279+0000","last_fullsized":"2024-03-04T12:24:06.028279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:10.472953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955442+0000","last_change":"2024-03-04T12:23:16.951145+0000","last_active":"2024-03-04T12:24:05.955442+0000","last_peered":"2024-03-04T12:24:05.955442+0000","last_clean":"2024-03-04T12:24:05.955442+0000","last_became_active":"2024-03-04T12:23:16.950702+0000","last_became_peered":"2024-03-04T12:23:16.950702+0000","last_unstale":"2024-03-04T12:24:05.955442+0000","last_undegraded":"2024-03-04T12:24:05.955442+0000","last_fullsized":"2024-03-04T12:24:05.955442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:42.720616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'11","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952657+0000","last_change":"2024-03-04T12:23:11.893810+0000","last_active":"2024-03-04T12:24:05.952657+0000","last_peered":"2024-03-04T12:24:05.952657+0000","last_clean":"2024-03-04T12:24:05.952657+0000","last_became_active":"2024-03-04T12:23:11.893465+0000","last_became_peered":"2024-03-04T12:23:11.893465+0000","last_unstale":"2024-03-04T12:24:05.952657+0000","last_undegraded":"2024-03-04T12:24:05.952657+0000","last_fullsized":"2024-03-04T12:24:05.952657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:27:41.199786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955046+0000","last_change":"2024-03-04T12:23:12.925763+0000","last_active":"2024-03-04T12:24:05.955046+0000","last_peered":"2024-03-04T12:24:05.955046+0000","last_clean":"2024-03-04T12:24:05.955046+0000","last_became_active":"2024-03-04T12:23:12.925429+0000","last_became_peered":"2024-03-04T12:23:12.925429+0000","last_unstale":"2024-03-04T12:24:05.955046+0000","last_undegraded":"2024-03-04T12:24:05.955046+0000","last_fullsized":"2024-03-04T12:24:05.955046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:46.814459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953078+0000","last_change":"2024-03-04T12:23:16.942624+0000","last_active":"2024-03-04T12:24:05.953078+0000","last_peered":"2024-03-04T12:24:05.953078+0000","last_clean":"2024-03-04T12:24:05.953078+0000","last_became_active":"2024-03-04T12:23:16.942287+0000","last_became_peered":"2024-03-04T12:23:16.942287+0000","last_unstale":"2024-03-04T12:24:05.953078+0000","last_undegraded":"2024-03-04T12:24:05.953078+0000","last_fullsized":"2024-03-04T12:24:05.953078+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:59.479611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952809+0000","last_change":"2024-03-04T12:23:11.891504+0000","last_active":"2024-03-04T12:24:05.952809+0000","last_peered":"2024-03-04T12:24:05.952809+0000","last_clean":"2024-03-04T12:24:05.952809+0000","last_became_active":"2024-03-04T12:23:11.891134+0000","last_became_peered":"2024-03-04T12:23:11.891134+0000","last_unstale":"2024-03-04T12:24:05.952809+0000","last_undegraded":"2024-03-04T12:24:05.952809+0000","last_fullsized":"2024-03-04T12:24:05.952809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:41.481003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955120+0000","last_change":"2024-03-04T12:23:12.928989+0000","last_active":"2024-03-04T12:24:05.955120+0000","last_peered":"2024-03-04T12:24:05.955120+0000","last_clean":"2024-03-04T12:24:05.955120+0000","last_became_active":"2024-03-04T12:23:12.928609+0000","last_became_peered":"2024-03-04T12:23:12.928609+0000","last_unstale":"2024-03-04T12:24:05.955120+0000","last_undegraded":"2024-03-04T12:24:05.955120+0000","last_fullsized":"2024-03-04T12:24:05.955120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:39.979546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952912+0000","last_change":"2024-03-04T12:23:16.949590+0000","last_active":"2024-03-04T12:24:05.952912+0000","last_peered":"2024-03-04T12:24:05.952912+0000","last_clean":"2024-03-04T12:24:05.952912+0000","last_became_active":"2024-03-04T12:23:16.949086+0000","last_became_peered":"2024-03-04T12:23:16.949086+0000","last_unstale":"2024-03-04T12:24:05.952912+0000","last_undegraded":"2024-03-04T12:24:05.952912+0000","last_fullsized":"2024-03-04T12:24:05.952912+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:07:09.819097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028494+0000","last_change":"2024-03-04T12:23:11.894110+0000","last_active":"2024-03-04T12:24:06.028494+0000","last_peered":"2024-03-04T12:24:06.028494+0000","last_clean":"2024-03-04T12:24:06.028494+0000","last_became_active":"2024-03-04T12:23:11.893419+0000","last_became_peered":"2024-03-04T12:23:11.893419+0000","last_unstale":"2024-03-04T12:24:06.028494+0000","last_undegraded":"2024-03-04T12:24:06.028494+0000","last_fullsized":"2024-03-04T12:24:06.028494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:53.963989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.957824+0000","last_change":"2024-03-04T12:23:12.907403+0000","last_active":"2024-03-04T12:24:05.957824+0000","last_peered":"2024-03-04T12:24:05.957824+0000","last_clean":"2024-03-04T12:24:05.957824+0000","last_became_active":"2024-03-04T12:23:12.906674+0000","last_became_peered":"2024-03-04T12:23:12.906674+0000","last_unstale":"2024-03-04T12:24:05.957824+0000","last_undegraded":"2024-03-04T12:24:05.957824+0000","last_fullsized":"2024-03-04T12:24:05.957824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:49:01.133676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952321+0000","last_change":"2024-03-04T12:23:16.955113+0000","last_active":"2024-03-04T12:24:05.952321+0000","last_peered":"2024-03-04T12:24:05.952321+0000","last_clean":"2024-03-04T12:24:05.952321+0000","last_became_active":"2024-03-04T12:23:16.951624+0000","last_became_peered":"2024-03-04T12:23:16.951624+0000","last_unstale":"2024-03-04T12:24:05.952321+0000","last_undegraded":"2024-03-04T12:24:05.952321+0000","last_fullsized":"2024-03-04T12:24:05.952321+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:22:23.935460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955685+0000","last_change":"2024-03-04T12:23:11.900213+0000","last_active":"2024-03-04T12:24:05.955685+0000","last_peered":"2024-03-04T12:24:05.955685+0000","last_clean":"2024-03-04T12:24:05.955685+0000","last_became_active":"2024-03-04T12:23:11.899832+0000","last_became_peered":"2024-03-04T12:23:11.899832+0000","last_unstale":"2024-03-04T12:24:05.955685+0000","last_undegraded":"2024-03-04T12:24:05.955685+0000","last_fullsized":"2024-03-04T12:24:05.955685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:55:24.553309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954634+0000","last_change":"2024-03-04T12:23:12.913315+0000","last_active":"2024-03-04T12:24:05.954634+0000","last_peered":"2024-03-04T12:24:05.954634+0000","last_clean":"2024-03-04T12:24:05.954634+0000","last_became_active":"2024-03-04T12:23:12.912850+0000","last_became_peered":"2024-03-04T12:23:12.912850+0000","last_unstale":"2024-03-04T12:24:05.954634+0000","last_undegraded":"2024-03-04T12:24:05.954634+0000","last_fullsized":"2024-03-04T12:24:05.954634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:55.140866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953110+0000","last_change":"2024-03-04T12:23:16.943295+0000","last_active":"2024-03-04T12:24:05.953110+0000","last_peered":"2024-03-04T12:24:05.953110+0000","last_clean":"2024-03-04T12:24:05.953110+0000","last_became_active":"2024-03-04T12:23:16.942853+0000","last_became_peered":"2024-03-04T12:23:16.942853+0000","last_unstale":"2024-03-04T12:24:05.953110+0000","last_undegraded":"2024-03-04T12:24:05.953110+0000","last_fullsized":"2024-03-04T12:24:05.953110+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:05:51.162179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953604+0000","last_change":"2024-03-04T12:23:11.897870+0000","last_active":"2024-03-04T12:24:05.953604+0000","last_peered":"2024-03-04T12:24:05.953604+0000","last_clean":"2024-03-04T12:24:05.953604+0000","last_became_active":"2024-03-04T12:23:11.897443+0000","last_became_peered":"2024-03-04T12:23:11.897443+0000","last_unstale":"2024-03-04T12:24:05.953604+0000","last_undegraded":"2024-03-04T12:24:05.953604+0000","last_fullsized":"2024-03-04T12:24:05.953604+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:49:37.875541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955202+0000","last_change":"2024-03-04T12:23:12.916441+0000","last_active":"2024-03-04T12:24:05.955202+0000","last_peered":"2024-03-04T12:24:05.955202+0000","last_clean":"2024-03-04T12:24:05.955202+0000","last_became_active":"2024-03-04T12:23:12.916078+0000","last_became_peered":"2024-03-04T12:23:12.916078+0000","last_unstale":"2024-03-04T12:24:05.955202+0000","last_undegraded":"2024-03-04T12:24:05.955202+0000","last_fullsized":"2024-03-04T12:24:05.955202+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:45:23.663627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952474+0000","last_change":"2024-03-04T12:23:16.953341+0000","last_active":"2024-03-04T12:24:05.952474+0000","last_peered":"2024-03-04T12:24:05.952474+0000","last_clean":"2024-03-04T12:24:05.952474+0000","last_became_active":"2024-03-04T12:23:16.952878+0000","last_became_peered":"2024-03-04T12:23:16.952878+0000","last_unstale":"2024-03-04T12:24:05.952474+0000","last_undegraded":"2024-03-04T12:24:05.952474+0000","last_fullsized":"2024-03-04T12:24:05.952474+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:08:55.575263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.957748+0000","last_change":"2024-03-04T12:23:11.903182+0000","last_active":"2024-03-04T12:24:05.957748+0000","last_peered":"2024-03-04T12:24:05.957748+0000","last_clean":"2024-03-04T12:24:05.957748+0000","last_became_active":"2024-03-04T12:23:11.902709+0000","last_became_peered":"2024-03-04T12:23:11.902709+0000","last_unstale":"2024-03-04T12:24:05.957748+0000","last_undegraded":"2024-03-04T12:24:05.957748+0000","last_fullsized":"2024-03-04T12:24:05.957748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:13:33.438929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955801+0000","last_change":"2024-03-04T12:23:12.931339+0000","last_active":"2024-03-04T12:24:05.955801+0000","last_peered":"2024-03-04T12:24:05.955801+0000","last_clean":"2024-03-04T12:24:05.955801+0000","last_became_active":"2024-03-04T12:23:12.930213+0000","last_became_peered":"2024-03-04T12:23:12.930213+0000","last_unstale":"2024-03-04T12:24:05.955801+0000","last_undegraded":"2024-03-04T12:24:05.955801+0000","last_fullsized":"2024-03-04T12:24:05.955801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:55:23.084139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028104+0000","last_change":"2024-03-04T12:23:16.948723+0000","last_active":"2024-03-04T12:24:06.028104+0000","last_peered":"2024-03-04T12:24:06.028104+0000","last_clean":"2024-03-04T12:24:06.028104+0000","last_became_active":"2024-03-04T12:23:16.948256+0000","last_became_peered":"2024-03-04T12:23:16.948256+0000","last_unstale":"2024-03-04T12:24:06.028104+0000","last_undegraded":"2024-03-04T12:24:06.028104+0000","last_fullsized":"2024-03-04T12:24:06.028104+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:22.060517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:51.011591+0000","last_change":"2024-03-04T12:23:11.898475+0000","last_active":"2024-03-04T12:24:51.011591+0000","last_peered":"2024-03-04T12:24:51.011591+0000","last_clean":"2024-03-04T12:24:51.011591+0000","last_became_active":"2024-03-04T12:23:11.898117+0000","last_became_peered":"2024-03-04T12:23:11.898117+0000","last_unstale":"2024-03-04T12:24:51.011591+0000","last_undegraded":"2024-03-04T12:24:51.011591+0000","last_fullsized":"2024-03-04T12:24:51.011591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:21:25.119591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955735+0000","last_change":"2024-03-04T12:23:12.929772+0000","last_active":"2024-03-04T12:24:05.955735+0000","last_peered":"2024-03-04T12:24:05.955735+0000","last_clean":"2024-03-04T12:24:05.955735+0000","last_became_active":"2024-03-04T12:23:12.929405+0000","last_became_peered":"2024-03-04T12:23:12.929405+0000","last_unstale":"2024-03-04T12:24:05.955735+0000","last_undegraded":"2024-03-04T12:24:05.955735+0000","last_fullsized":"2024-03-04T12:24:05.955735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:22:42.874989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954869+0000","last_change":"2024-03-04T12:23:16.941715+0000","last_active":"2024-03-04T12:24:05.954869+0000","last_peered":"2024-03-04T12:24:05.954869+0000","last_clean":"2024-03-04T12:24:05.954869+0000","last_became_active":"2024-03-04T12:23:16.941298+0000","last_became_peered":"2024-03-04T12:23:16.941298+0000","last_unstale":"2024-03-04T12:24:05.954869+0000","last_undegraded":"2024-03-04T12:24:05.954869+0000","last_fullsized":"2024-03-04T12:24:05.954869+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:09:02.019531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029116+0000","last_change":"2024-03-04T12:23:11.896413+0000","last_active":"2024-03-04T12:24:06.029116+0000","last_peered":"2024-03-04T12:24:06.029116+0000","last_clean":"2024-03-04T12:24:06.029116+0000","last_became_active":"2024-03-04T12:23:11.895679+0000","last_became_peered":"2024-03-04T12:23:11.895679+0000","last_unstale":"2024-03-04T12:24:06.029116+0000","last_undegraded":"2024-03-04T12:24:06.029116+0000","last_fullsized":"2024-03-04T12:24:06.029116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:25:41.967530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955720+0000","last_change":"2024-03-04T12:23:12.920468+0000","last_active":"2024-03-04T12:24:05.955720+0000","last_peered":"2024-03-04T12:24:05.955720+0000","last_clean":"2024-03-04T12:24:05.955720+0000","last_became_active":"2024-03-04T12:23:12.920113+0000","last_became_peered":"2024-03-04T12:23:12.920113+0000","last_unstale":"2024-03-04T12:24:05.955720+0000","last_undegraded":"2024-03-04T12:24:05.955720+0000","last_fullsized":"2024-03-04T12:24:05.955720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:00:02.283518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.031522+0000","last_change":"2024-03-04T12:24:06.030444+0000","last_active":"2024-03-04T12:24:06.031522+0000","last_peered":"2024-03-04T12:24:06.031522+0000","last_clean":"2024-03-04T12:24:06.031522+0000","last_became_active":"2024-03-04T12:24:06.029934+0000","last_became_peered":"2024-03-04T12:24:06.029934+0000","last_unstale":"2024-03-04T12:24:06.031522+0000","last_undegraded":"2024-03-04T12:24:06.031522+0000","last_fullsized":"2024-03-04T12:24:06.031522+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:08:16.820509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028376+0000","last_change":"2024-03-04T12:23:11.903323+0000","last_active":"2024-03-04T12:24:06.028376+0000","last_peered":"2024-03-04T12:24:06.028376+0000","last_clean":"2024-03-04T12:24:06.028376+0000","last_became_active":"2024-03-04T12:23:11.902987+0000","last_became_peered":"2024-03-04T12:23:11.902987+0000","last_unstale":"2024-03-04T12:24:06.028376+0000","last_undegraded":"2024-03-04T12:24:06.028376+0000","last_fullsized":"2024-03-04T12:24:06.028376+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:14.044391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.956314+0000","last_change":"2024-03-04T12:23:12.913491+0000","last_active":"2024-03-04T12:24:05.956314+0000","last_peered":"2024-03-04T12:24:05.956314+0000","last_clean":"2024-03-04T12:24:05.956314+0000","last_became_active":"2024-03-04T12:23:12.912103+0000","last_became_peered":"2024-03-04T12:23:12.912103+0000","last_unstale":"2024-03-04T12:24:05.956314+0000","last_undegraded":"2024-03-04T12:24:05.956314+0000","last_fullsized":"2024-03-04T12:24:05.956314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:05:38.033596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954760+0000","last_change":"2024-03-04T12:23:12.908437+0000","last_active":"2024-03-04T12:24:05.954760+0000","last_peered":"2024-03-04T12:24:05.954760+0000","last_clean":"2024-03-04T12:24:05.954760+0000","last_became_active":"2024-03-04T12:23:12.908071+0000","last_became_peered":"2024-03-04T12:23:12.908071+0000","last_unstale":"2024-03-04T12:24:05.954760+0000","last_undegraded":"2024-03-04T12:24:05.954760+0000","last_fullsized":"2024-03-04T12:24:05.954760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:20.112645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953550+0000","last_change":"2024-03-04T12:23:12.921645+0000","last_active":"2024-03-04T12:24:05.953550+0000","last_peered":"2024-03-04T12:24:05.953550+0000","last_clean":"2024-03-04T12:24:05.953550+0000","last_became_active":"2024-03-04T12:23:12.921275+0000","last_became_peered":"2024-03-04T12:23:12.921275+0000","last_unstale":"2024-03-04T12:24:05.953550+0000","last_undegraded":"2024-03-04T12:24:05.953550+0000","last_fullsized":"2024-03-04T12:24:05.953550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:56:17.740782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953081+0000","last_change":"2024-03-04T12:23:12.919813+0000","last_active":"2024-03-04T12:24:05.953081+0000","last_peered":"2024-03-04T12:24:05.953081+0000","last_clean":"2024-03-04T12:24:05.953081+0000","last_became_active":"2024-03-04T12:23:12.919469+0000","last_became_peered":"2024-03-04T12:23:12.919469+0000","last_unstale":"2024-03-04T12:24:05.953081+0000","last_undegraded":"2024-03-04T12:24:05.953081+0000","last_fullsized":"2024-03-04T12:24:05.953081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:46.720268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955109+0000","last_change":"2024-03-04T12:23:12.918476+0000","last_active":"2024-03-04T12:24:05.955109+0000","last_peered":"2024-03-04T12:24:05.955109+0000","last_clean":"2024-03-04T12:24:05.955109+0000","last_became_active":"2024-03-04T12:23:12.917743+0000","last_became_peered":"2024-03-04T12:23:12.917743+0000","last_unstale":"2024-03-04T12:24:05.955109+0000","last_undegraded":"2024-03-04T12:24:05.955109+0000","last_fullsized":"2024-03-04T12:24:05.955109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:20.246769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028261+0000","last_change":"2024-03-04T12:23:12.918254+0000","last_active":"2024-03-04T12:24:06.028261+0000","last_peered":"2024-03-04T12:24:06.028261+0000","last_clean":"2024-03-04T12:24:06.028261+0000","last_became_active":"2024-03-04T12:23:12.917922+0000","last_became_peered":"2024-03-04T12:23:12.917922+0000","last_unstale":"2024-03-04T12:24:06.028261+0000","last_undegraded":"2024-03-04T12:24:06.028261+0000","last_fullsized":"2024-03-04T12:24:06.028261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:48:20.847984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955644+0000","last_change":"2024-03-04T12:23:12.910368+0000","last_active":"2024-03-04T12:24:05.955644+0000","last_peered":"2024-03-04T12:24:05.955644+0000","last_clean":"2024-03-04T12:24:05.955644+0000","last_became_active":"2024-03-04T12:23:12.910017+0000","last_became_peered":"2024-03-04T12:23:12.910017+0000","last_unstale":"2024-03-04T12:24:05.955644+0000","last_undegraded":"2024-03-04T12:24:05.955644+0000","last_fullsized":"2024-03-04T12:24:05.955644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:15:22.093644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953961+0000","last_change":"2024-03-04T12:23:12.921021+0000","last_active":"2024-03-04T12:24:05.953961+0000","last_peered":"2024-03-04T12:24:05.953961+0000","last_clean":"2024-03-04T12:24:05.953961+0000","last_became_active":"2024-03-04T12:23:12.920600+0000","last_became_peered":"2024-03-04T12:23:12.920600+0000","last_unstale":"2024-03-04T12:24:05.953961+0000","last_undegraded":"2024-03-04T12:24:05.953961+0000","last_fullsized":"2024-03-04T12:24:05.953961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:00:31.706900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.957607+0000","last_change":"2024-03-04T12:24:05.957607+0000","last_active":"2024-03-04T12:24:05.957607+0000","last_peered":"2024-03-04T12:24:05.957607+0000","last_clean":"2024-03-04T12:24:05.957607+0000","last_became_active":"2024-03-04T12:24:05.957335+0000","last_became_peered":"2024-03-04T12:24:05.957335+0000","last_unstale":"2024-03-04T12:24:05.957607+0000","last_undegraded":"2024-03-04T12:24:05.957607+0000","last_fullsized":"2024-03-04T12:24:05.957607+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:12.926219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028671+0000","last_change":"2024-03-04T12:23:12.911394+0000","last_active":"2024-03-04T12:24:06.028671+0000","last_peered":"2024-03-04T12:24:06.028671+0000","last_clean":"2024-03-04T12:24:06.028671+0000","last_became_active":"2024-03-04T12:23:12.911061+0000","last_became_peered":"2024-03-04T12:23:12.911061+0000","last_unstale":"2024-03-04T12:24:06.028671+0000","last_undegraded":"2024-03-04T12:24:06.028671+0000","last_fullsized":"2024-03-04T12:24:06.028671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:39.880383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955603+0000","last_change":"2024-03-04T12:23:12.911356+0000","last_active":"2024-03-04T12:24:05.955603+0000","last_peered":"2024-03-04T12:24:05.955603+0000","last_clean":"2024-03-04T12:24:05.955603+0000","last_became_active":"2024-03-04T12:23:12.911029+0000","last_became_peered":"2024-03-04T12:23:12.911029+0000","last_unstale":"2024-03-04T12:24:05.955603+0000","last_undegraded":"2024-03-04T12:24:05.955603+0000","last_fullsized":"2024-03-04T12:24:05.955603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:52.611625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028495+0000","last_change":"2024-03-04T12:23:12.924921+0000","last_active":"2024-03-04T12:24:06.028495+0000","last_peered":"2024-03-04T12:24:06.028495+0000","last_clean":"2024-03-04T12:24:06.028495+0000","last_became_active":"2024-03-04T12:23:12.924558+0000","last_became_peered":"2024-03-04T12:23:12.924558+0000","last_unstale":"2024-03-04T12:24:06.028495+0000","last_undegraded":"2024-03-04T12:24:06.028495+0000","last_fullsized":"2024-03-04T12:24:06.028495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:10:53.238440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028167+0000","last_change":"2024-03-04T12:23:12.913106+0000","last_active":"2024-03-04T12:24:06.028167+0000","last_peered":"2024-03-04T12:24:06.028167+0000","last_clean":"2024-03-04T12:24:06.028167+0000","last_became_active":"2024-03-04T12:23:12.912741+0000","last_became_peered":"2024-03-04T12:23:12.912741+0000","last_unstale":"2024-03-04T12:24:06.028167+0000","last_undegraded":"2024-03-04T12:24:06.028167+0000","last_fullsized":"2024-03-04T12:24:06.028167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:46:21.406383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955450+0000","last_change":"2024-03-04T12:23:12.918082+0000","last_active":"2024-03-04T12:24:05.955450+0000","last_peered":"2024-03-04T12:24:05.955450+0000","last_clean":"2024-03-04T12:24:05.955450+0000","last_became_active":"2024-03-04T12:23:12.917754+0000","last_became_peered":"2024-03-04T12:23:12.917754+0000","last_unstale":"2024-03-04T12:24:05.955450+0000","last_undegraded":"2024-03-04T12:24:05.955450+0000","last_fullsized":"2024-03-04T12:24:05.955450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:56.601779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286257,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67473,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":761,"num_read_kb":454,"num_write":239,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":202419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":501441,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645352,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":1148917,"data_compressed":57051,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1032192,"data_stored":505467,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1024000,"data_stored":501749,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":1098637,"data_compressed":57051,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":496875,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1069056,"data_stored":558174,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1028096,"data_stored":498331,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1089536,"data_stored":558785,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":496583,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":501340,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1085440,"data_stored":1093828,"data_compressed":57051,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T12:24:55.545 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T12:24:55.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:55 smithi086 ceph-mon[36851]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:24:55.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:55 smithi086 ceph-mon[36851]: from='client.24862 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:24:55.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:55 smithi063 ceph-mon[36717]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:24:55.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:55 smithi063 ceph-mon[36717]: from='client.24862 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:24:55.797 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:57.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.517+0000 7fb94df4d700 1 -- 172.21.15.60:0/3911541051 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94806a520 msgr2=0x7fb94806a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:57.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.517+0000 7fb94df4d700 1 --2- 172.21.15.60:0/3911541051 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94806a520 0x7fb94806a980 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0099c0 tx=0x7fb93c009cd0 comp rx=0 tx=0).stop 2024-03-04T12:24:57.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.518+0000 7fb94df4d700 1 -- 172.21.15.60:0/3911541051 shutdown_connections 2024-03-04T12:24:57.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.518+0000 7fb94df4d700 1 --2- 172.21.15.60:0/3911541051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb94806aec0 0x7fb94810ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:57.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.518+0000 7fb94df4d700 1 --2- 172.21.15.60:0/3911541051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb948107ea0 0x7fb948108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:57.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.518+0000 7fb94df4d700 1 --2- 172.21.15.60:0/3911541051 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94806a520 0x7fb94806a980 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:57.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.518+0000 7fb94df4d700 1 -- 172.21.15.60:0/3911541051 >> 172.21.15.60:0/3911541051 conn(0x7fb948076d40 msgr2=0x7fb948077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:57.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.518+0000 7fb94df4d700 1 -- 172.21.15.60:0/3911541051 shutdown_connections 2024-03-04T12:24:57.524 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.518+0000 7fb94df4d700 1 -- 172.21.15.60:0/3911541051 wait complete. 2024-03-04T12:24:57.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.518+0000 7fb94df4d700 1 Processor -- start 2024-03-04T12:24:57.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb94df4d700 1 -- start start 2024-03-04T12:24:57.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb94df4d700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb94806a520 0x7fb9481985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:57.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb94df4d700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94806aec0 0x7fb948198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:57.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb94df4d700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb948107ea0 0x7fb94819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:57.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb94df4d700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb948112120 con 0x7fb94806aec0 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb94df4d700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb948111fa0 con 0x7fb94806a520 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb94df4d700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9481122a0 con 0x7fb948107ea0 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb946ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94806aec0 0x7fb948198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb946ffd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94806aec0 0x7fb948198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46502/0 (socket says 172.21.15.60:46502) 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb946ffd700 1 -- 172.21.15.60:0/3931434052 learned_addr learned my addr 172.21.15.60:0/3931434052 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb9477fe700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb94806a520 0x7fb9481985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb947fff700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb948107ea0 0x7fb94819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb946ffd700 1 -- 172.21.15.60:0/3931434052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb948107ea0 msgr2=0x7fb94819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb946ffd700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb948107ea0 0x7fb94819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb946ffd700 1 -- 172.21.15.60:0/3931434052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb94806a520 msgr2=0x7fb9481985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb946ffd700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb94806a520 0x7fb9481985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.519+0000 7fb946ffd700 1 -- 172.21.15.60:0/3931434052 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94819d600 con 0x7fb94806aec0 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.520+0000 7fb946ffd700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94806aec0 0x7fb948198b30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0059d0 tx=0x7fb93c0038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:57.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.520+0000 7fb944ff9700 1 -- 172.21.15.60:0/3931434052 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c027070 con 0x7fb94806aec0 2024-03-04T12:24:57.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.520+0000 7fb944ff9700 1 -- 172.21.15.60:0/3931434052 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb93c004050 con 0x7fb94806aec0 2024-03-04T12:24:57.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.520+0000 7fb94df4d700 1 -- 172.21.15.60:0/3931434052 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94819d890 con 0x7fb94806aec0 2024-03-04T12:24:57.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.520+0000 7fb944ff9700 1 -- 172.21.15.60:0/3931434052 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c022db0 con 0x7fb94806aec0 2024-03-04T12:24:57.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.520+0000 7fb94df4d700 1 -- 172.21.15.60:0/3931434052 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9481a5170 con 0x7fb94806aec0 2024-03-04T12:24:57.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.521+0000 7fb94df4d700 1 -- 172.21.15.60:0/3931434052 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb948103550 con 0x7fb94806aec0 2024-03-04T12:24:57.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.523+0000 7fb944ff9700 1 -- 172.21.15.60:0/3931434052 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fb93c02a070 con 0x7fb94806aec0 2024-03-04T12:24:57.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.523+0000 7fb944ff9700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb9300a1f20 0x7fb9300a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:24:57.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.523+0000 7fb9477fe700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb9300a1f20 0x7fb9300a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:24:57.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.523+0000 7fb944ff9700 1 -- 172.21.15.60:0/3931434052 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb93c0cf5d0 con 0x7fb94806aec0 2024-03-04T12:24:57.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.524+0000 7fb9477fe700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb9300a1f20 0x7fb9300a43e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb938005950 tx=0x7fb93800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:24:57.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.527+0000 7fb944ff9700 1 -- 172.21.15.60:0/3931434052 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb93c031020 con 0x7fb94806aec0 2024-03-04T12:24:57.533 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:57 smithi060 ceph-mon[32745]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:57.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:57 smithi086 ceph-mon[36851]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:57.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:57 smithi063 ceph-mon[36717]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:24:57.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.728+0000 7fb94df4d700 1 -- 172.21.15.60:0/3931434052 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb948199950 con 0x7fb9300a1f20 2024-03-04T12:24:57.743 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.737+0000 7fb944ff9700 1 -- 172.21.15.60:0/3931434052 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340678 (secure 0 0 0) 0x7fb948199950 con 0x7fb9300a1f20 2024-03-04T12:24:57.748 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:24:57.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.740+0000 7fb94df4d700 1 -- 172.21.15.60:0/3931434052 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb9300a1f20 msgr2=0x7fb9300a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:57.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.740+0000 7fb94df4d700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb9300a1f20 0x7fb9300a43e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb938005950 tx=0x7fb93800b410 comp rx=0 tx=0).stop 2024-03-04T12:24:57.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.740+0000 7fb94df4d700 1 -- 172.21.15.60:0/3931434052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94806aec0 msgr2=0x7fb948198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:24:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.740+0000 7fb94df4d700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94806aec0 0x7fb948198b30 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0059d0 tx=0x7fb93c0038b0 comp rx=0 tx=0).stop 2024-03-04T12:24:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.741+0000 7fb94df4d700 1 -- 172.21.15.60:0/3931434052 shutdown_connections 2024-03-04T12:24:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.741+0000 7fb94df4d700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb948107ea0 0x7fb94819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.741+0000 7fb94df4d700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb94806a520 0x7fb9481985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.741+0000 7fb94df4d700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fb9300a1f20 0x7fb9300a43e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.741+0000 7fb94df4d700 1 --2- 172.21.15.60:0/3931434052 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fb94806aec0 0x7fb948198b30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:24:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.741+0000 7fb94df4d700 1 -- 172.21.15.60:0/3931434052 >> 172.21.15.60:0/3931434052 conn(0x7fb948076d40 msgr2=0x7fb9480ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:24:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.741+0000 7fb94df4d700 1 -- 172.21.15.60:0/3931434052 shutdown_connections 2024-03-04T12:24:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:24:57.741+0000 7fb94df4d700 1 -- 172.21.15.60:0/3931434052 wait complete. 2024-03-04T12:24:57.751 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-04T12:24:58.526 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-03-04T12:24:56.180581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658075,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":902,"num_read_kb":578,"num_write":422,"num_write_kb":3076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":382332,"kb_used_data":12296,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124559492,"statfs":{"total":1151940427776,"available":1151548919808,"internally_reserved":0,"allocated":12591104,"data_stored":7960127,"data_compressed":357048,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003223"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955770+0000","last_change":"2024-03-04T12:23:12.926673+0000","last_active":"2024-03-04T12:24:05.955770+0000","last_peered":"2024-03-04T12:24:05.955770+0000","last_clean":"2024-03-04T12:24:05.955770+0000","last_became_active":"2024-03-04T12:23:12.926299+0000","last_became_peered":"2024-03-04T12:23:12.926299+0000","last_unstale":"2024-03-04T12:24:05.955770+0000","last_undegraded":"2024-03-04T12:24:05.955770+0000","last_fullsized":"2024-03-04T12:24:05.955770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:19.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955021+0000","last_change":"2024-03-04T12:23:12.921758+0000","last_active":"2024-03-04T12:24:05.955021+0000","last_peered":"2024-03-04T12:24:05.955021+0000","last_clean":"2024-03-04T12:24:05.955021+0000","last_became_active":"2024-03-04T12:23:12.921317+0000","last_became_peered":"2024-03-04T12:23:12.921317+0000","last_unstale":"2024-03-04T12:24:05.955021+0000","last_undegraded":"2024-03-04T12:24:05.955021+0000","last_fullsized":"2024-03-04T12:24:05.955021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:20:25.179004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.956186+0000","last_change":"2024-03-04T12:23:12.925771+0000","last_active":"2024-03-04T12:24:05.956186+0000","last_peered":"2024-03-04T12:24:05.956186+0000","last_clean":"2024-03-04T12:24:05.956186+0000","last_became_active":"2024-03-04T12:23:12.925447+0000","last_became_peered":"2024-03-04T12:23:12.925447+0000","last_unstale":"2024-03-04T12:24:05.956186+0000","last_undegraded":"2024-03-04T12:24:05.956186+0000","last_fullsized":"2024-03-04T12:24:05.956186+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:12.380831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.959806+0000","last_change":"2024-03-04T12:24:05.959806+0000","last_active":"2024-03-04T12:24:05.959806+0000","last_peered":"2024-03-04T12:24:05.959806+0000","last_clean":"2024-03-04T12:24:05.959806+0000","last_became_active":"2024-03-04T12:24:05.959557+0000","last_became_peered":"2024-03-04T12:24:05.959557+0000","last_unstale":"2024-03-04T12:24:05.959806+0000","last_undegraded":"2024-03-04T12:24:05.959806+0000","last_fullsized":"2024-03-04T12:24:05.959806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:51.945517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.957611+0000","last_change":"2024-03-04T12:23:12.921118+0000","last_active":"2024-03-04T12:24:05.957611+0000","last_peered":"2024-03-04T12:24:05.957611+0000","last_clean":"2024-03-04T12:24:05.957611+0000","last_became_active":"2024-03-04T12:23:12.920127+0000","last_became_peered":"2024-03-04T12:23:12.920127+0000","last_unstale":"2024-03-04T12:24:05.957611+0000","last_undegraded":"2024-03-04T12:24:05.957611+0000","last_fullsized":"2024-03-04T12:24:05.957611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:22:10.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954447+0000","last_change":"2024-03-04T12:23:12.916751+0000","last_active":"2024-03-04T12:24:05.954447+0000","last_peered":"2024-03-04T12:24:05.954447+0000","last_clean":"2024-03-04T12:24:05.954447+0000","last_became_active":"2024-03-04T12:23:12.916404+0000","last_became_peered":"2024-03-04T12:23:12.916404+0000","last_unstale":"2024-03-04T12:24:05.954447+0000","last_undegraded":"2024-03-04T12:24:05.954447+0000","last_fullsized":"2024-03-04T12:24:05.954447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:39.998992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029608+0000","last_change":"2024-03-04T12:24:06.029608+0000","last_active":"2024-03-04T12:24:06.029608+0000","last_peered":"2024-03-04T12:24:06.029608+0000","last_clean":"2024-03-04T12:24:06.029608+0000","last_became_active":"2024-03-04T12:24:06.029021+0000","last_became_peered":"2024-03-04T12:24:06.029021+0000","last_unstale":"2024-03-04T12:24:06.029608+0000","last_undegraded":"2024-03-04T12:24:06.029608+0000","last_fullsized":"2024-03-04T12:24:06.029608+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:13.140511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955512+0000","last_change":"2024-03-04T12:23:12.917627+0000","last_active":"2024-03-04T12:24:05.955512+0000","last_peered":"2024-03-04T12:24:05.955512+0000","last_clean":"2024-03-04T12:24:05.955512+0000","last_became_active":"2024-03-04T12:23:12.917279+0000","last_became_peered":"2024-03-04T12:23:12.917279+0000","last_unstale":"2024-03-04T12:24:05.955512+0000","last_undegraded":"2024-03-04T12:24:05.955512+0000","last_fullsized":"2024-03-04T12:24:05.955512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:33:03.487273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029026+0000","last_change":"2024-03-04T12:23:12.922601+0000","last_active":"2024-03-04T12:24:06.029026+0000","last_peered":"2024-03-04T12:24:06.029026+0000","last_clean":"2024-03-04T12:24:06.029026+0000","last_became_active":"2024-03-04T12:23:12.922037+0000","last_became_peered":"2024-03-04T12:23:12.922037+0000","last_unstale":"2024-03-04T12:24:06.029026+0000","last_undegraded":"2024-03-04T12:24:06.029026+0000","last_fullsized":"2024-03-04T12:24:06.029026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:32.076439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954563+0000","last_change":"2024-03-04T12:23:12.916308+0000","last_active":"2024-03-04T12:24:05.954563+0000","last_peered":"2024-03-04T12:24:05.954563+0000","last_clean":"2024-03-04T12:24:05.954563+0000","last_became_active":"2024-03-04T12:23:12.915897+0000","last_became_peered":"2024-03-04T12:23:12.915897+0000","last_unstale":"2024-03-04T12:24:05.954563+0000","last_undegraded":"2024-03-04T12:24:05.954563+0000","last_fullsized":"2024-03-04T12:24:05.954563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:15:53.133053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955676+0000","last_change":"2024-03-04T12:23:12.930700+0000","last_active":"2024-03-04T12:24:05.955676+0000","last_peered":"2024-03-04T12:24:05.955676+0000","last_clean":"2024-03-04T12:24:05.955676+0000","last_became_active":"2024-03-04T12:23:12.930328+0000","last_became_peered":"2024-03-04T12:23:12.930328+0000","last_unstale":"2024-03-04T12:24:05.955676+0000","last_undegraded":"2024-03-04T12:24:05.955676+0000","last_fullsized":"2024-03-04T12:24:05.955676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:29.990527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953334+0000","last_change":"2024-03-04T12:23:12.915471+0000","last_active":"2024-03-04T12:24:05.953334+0000","last_peered":"2024-03-04T12:24:05.953334+0000","last_clean":"2024-03-04T12:24:05.953334+0000","last_became_active":"2024-03-04T12:23:12.913627+0000","last_became_peered":"2024-03-04T12:23:12.913627+0000","last_unstale":"2024-03-04T12:24:05.953334+0000","last_undegraded":"2024-03-04T12:24:05.953334+0000","last_fullsized":"2024-03-04T12:24:05.953334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:42:56.762782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028130+0000","last_change":"2024-03-04T12:23:12.899143+0000","last_active":"2024-03-04T12:24:06.028130+0000","last_peered":"2024-03-04T12:24:06.028130+0000","last_clean":"2024-03-04T12:24:06.028130+0000","last_became_active":"2024-03-04T12:23:12.898498+0000","last_became_peered":"2024-03-04T12:23:12.898498+0000","last_unstale":"2024-03-04T12:24:06.028130+0000","last_undegraded":"2024-03-04T12:24:06.028130+0000","last_fullsized":"2024-03-04T12:24:06.028130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:33.840683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955353+0000","last_change":"2024-03-04T12:23:12.920873+0000","last_active":"2024-03-04T12:24:05.955353+0000","last_peered":"2024-03-04T12:24:05.955353+0000","last_clean":"2024-03-04T12:24:05.955353+0000","last_became_active":"2024-03-04T12:23:12.919080+0000","last_became_peered":"2024-03-04T12:23:12.919080+0000","last_unstale":"2024-03-04T12:24:05.955353+0000","last_undegraded":"2024-03-04T12:24:05.955353+0000","last_fullsized":"2024-03-04T12:24:05.955353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:47:26.710624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.959024+0000","last_change":"2024-03-04T12:24:05.959024+0000","last_active":"2024-03-04T12:24:05.959024+0000","last_peered":"2024-03-04T12:24:05.959024+0000","last_clean":"2024-03-04T12:24:05.959024+0000","last_became_active":"2024-03-04T12:24:05.958750+0000","last_became_peered":"2024-03-04T12:24:05.958750+0000","last_unstale":"2024-03-04T12:24:05.959024+0000","last_undegraded":"2024-03-04T12:24:05.959024+0000","last_fullsized":"2024-03-04T12:24:05.959024+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:59:33.263357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028464+0000","last_change":"2024-03-04T12:23:12.923860+0000","last_active":"2024-03-04T12:24:06.028464+0000","last_peered":"2024-03-04T12:24:06.028464+0000","last_clean":"2024-03-04T12:24:06.028464+0000","last_became_active":"2024-03-04T12:23:12.923530+0000","last_became_peered":"2024-03-04T12:23:12.923530+0000","last_unstale":"2024-03-04T12:24:06.028464+0000","last_undegraded":"2024-03-04T12:24:06.028464+0000","last_fullsized":"2024-03-04T12:24:06.028464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:30:01.466458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955642+0000","last_change":"2024-03-04T12:23:12.927159+0000","last_active":"2024-03-04T12:24:05.955642+0000","last_peered":"2024-03-04T12:24:05.955642+0000","last_clean":"2024-03-04T12:24:05.955642+0000","last_became_active":"2024-03-04T12:23:12.926800+0000","last_became_peered":"2024-03-04T12:23:12.926800+0000","last_unstale":"2024-03-04T12:24:05.955642+0000","last_undegraded":"2024-03-04T12:24:05.955642+0000","last_fullsized":"2024-03-04T12:24:05.955642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:52.633648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952564+0000","last_change":"2024-03-04T12:23:12.920607+0000","last_active":"2024-03-04T12:24:05.952564+0000","last_peered":"2024-03-04T12:24:05.952564+0000","last_clean":"2024-03-04T12:24:05.952564+0000","last_became_active":"2024-03-04T12:23:12.920257+0000","last_became_peered":"2024-03-04T12:23:12.920257+0000","last_unstale":"2024-03-04T12:24:05.952564+0000","last_undegraded":"2024-03-04T12:24:05.952564+0000","last_fullsized":"2024-03-04T12:24:05.952564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:39.353673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953934+0000","last_change":"2024-03-04T12:23:12.906635+0000","last_active":"2024-03-04T12:24:05.953934+0000","last_peered":"2024-03-04T12:24:05.953934+0000","last_clean":"2024-03-04T12:24:05.953934+0000","last_became_active":"2024-03-04T12:23:12.906243+0000","last_became_peered":"2024-03-04T12:23:12.906243+0000","last_unstale":"2024-03-04T12:24:05.953934+0000","last_undegraded":"2024-03-04T12:24:05.953934+0000","last_fullsized":"2024-03-04T12:24:05.953934+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:44.375934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953355+0000","last_change":"2024-03-04T12:23:16.951685+0000","last_active":"2024-03-04T12:24:05.953355+0000","last_peered":"2024-03-04T12:24:05.953355+0000","last_clean":"2024-03-04T12:24:05.953355+0000","last_became_active":"2024-03-04T12:23:16.951329+0000","last_became_peered":"2024-03-04T12:23:16.951329+0000","last_unstale":"2024-03-04T12:24:05.953355+0000","last_undegraded":"2024-03-04T12:24:05.953355+0000","last_fullsized":"2024-03-04T12:24:05.953355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:03:27.233466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":115,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:54.014123+0000","last_change":"2024-03-04T12:23:11.900681+0000","last_active":"2024-03-04T12:24:54.014123+0000","last_peered":"2024-03-04T12:24:54.014123+0000","last_clean":"2024-03-04T12:24:54.014123+0000","last_became_active":"2024-03-04T12:23:11.900365+0000","last_became_peered":"2024-03-04T12:23:11.900365+0000","last_unstale":"2024-03-04T12:24:54.014123+0000","last_undegraded":"2024-03-04T12:24:54.014123+0000","last_fullsized":"2024-03-04T12:24:54.014123+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:53.052513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.030707+0000","last_change":"2024-03-04T12:24:06.030707+0000","last_active":"2024-03-04T12:24:06.030707+0000","last_peered":"2024-03-04T12:24:06.030707+0000","last_clean":"2024-03-04T12:24:06.030707+0000","last_became_active":"2024-03-04T12:24:06.030306+0000","last_became_peered":"2024-03-04T12:24:06.030306+0000","last_unstale":"2024-03-04T12:24:06.030707+0000","last_undegraded":"2024-03-04T12:24:06.030707+0000","last_fullsized":"2024-03-04T12:24:06.030707+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:59:18.939734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952650+0000","last_change":"2024-03-04T12:23:16.953704+0000","last_active":"2024-03-04T12:24:05.952650+0000","last_peered":"2024-03-04T12:24:05.952650+0000","last_clean":"2024-03-04T12:24:05.952650+0000","last_became_active":"2024-03-04T12:23:16.953177+0000","last_became_peered":"2024-03-04T12:23:16.953177+0000","last_unstale":"2024-03-04T12:24:05.952650+0000","last_undegraded":"2024-03-04T12:24:05.952650+0000","last_fullsized":"2024-03-04T12:24:05.952650+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:37:00.469767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955828+0000","last_change":"2024-03-04T12:23:11.904271+0000","last_active":"2024-03-04T12:24:05.955828+0000","last_peered":"2024-03-04T12:24:05.955828+0000","last_clean":"2024-03-04T12:24:05.955828+0000","last_became_active":"2024-03-04T12:23:11.903963+0000","last_became_peered":"2024-03-04T12:23:11.903963+0000","last_unstale":"2024-03-04T12:24:05.955828+0000","last_undegraded":"2024-03-04T12:24:05.955828+0000","last_fullsized":"2024-03-04T12:24:05.955828+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:44.774621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.956538+0000","last_change":"2024-03-04T12:23:12.925750+0000","last_active":"2024-03-04T12:24:05.956538+0000","last_peered":"2024-03-04T12:24:05.956538+0000","last_clean":"2024-03-04T12:24:05.956538+0000","last_became_active":"2024-03-04T12:23:12.925380+0000","last_became_peered":"2024-03-04T12:23:12.925380+0000","last_unstale":"2024-03-04T12:24:05.956538+0000","last_undegraded":"2024-03-04T12:24:05.956538+0000","last_fullsized":"2024-03-04T12:24:05.956538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:14.665127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952468+0000","last_change":"2024-03-04T12:23:16.952303+0000","last_active":"2024-03-04T12:24:05.952468+0000","last_peered":"2024-03-04T12:24:05.952468+0000","last_clean":"2024-03-04T12:24:05.952468+0000","last_became_active":"2024-03-04T12:23:16.951666+0000","last_became_peered":"2024-03-04T12:23:16.951666+0000","last_unstale":"2024-03-04T12:24:05.952468+0000","last_undegraded":"2024-03-04T12:24:05.952468+0000","last_fullsized":"2024-03-04T12:24:05.952468+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:41.326808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.959106+0000","last_change":"2024-03-04T12:23:11.903871+0000","last_active":"2024-03-04T12:24:05.959106+0000","last_peered":"2024-03-04T12:24:05.959106+0000","last_clean":"2024-03-04T12:24:05.959106+0000","last_became_active":"2024-03-04T12:23:11.902735+0000","last_became_peered":"2024-03-04T12:23:11.902735+0000","last_unstale":"2024-03-04T12:24:05.959106+0000","last_undegraded":"2024-03-04T12:24:05.959106+0000","last_fullsized":"2024-03-04T12:24:05.959106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:45:09.897740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028321+0000","last_change":"2024-03-04T12:23:12.924504+0000","last_active":"2024-03-04T12:24:06.028321+0000","last_peered":"2024-03-04T12:24:06.028321+0000","last_clean":"2024-03-04T12:24:06.028321+0000","last_became_active":"2024-03-04T12:23:12.922934+0000","last_became_peered":"2024-03-04T12:23:12.922934+0000","last_unstale":"2024-03-04T12:24:06.028321+0000","last_undegraded":"2024-03-04T12:24:06.028321+0000","last_fullsized":"2024-03-04T12:24:06.028321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:28.974601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952232+0000","last_change":"2024-03-04T12:23:16.952324+0000","last_active":"2024-03-04T12:24:05.952232+0000","last_peered":"2024-03-04T12:24:05.952232+0000","last_clean":"2024-03-04T12:24:05.952232+0000","last_became_active":"2024-03-04T12:23:16.951145+0000","last_became_peered":"2024-03-04T12:23:16.951145+0000","last_unstale":"2024-03-04T12:24:05.952232+0000","last_undegraded":"2024-03-04T12:24:05.952232+0000","last_fullsized":"2024-03-04T12:24:05.952232+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:20:47.029523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.958858+0000","last_change":"2024-03-04T12:23:11.904391+0000","last_active":"2024-03-04T12:24:05.958858+0000","last_peered":"2024-03-04T12:24:05.958858+0000","last_clean":"2024-03-04T12:24:05.958858+0000","last_became_active":"2024-03-04T12:23:11.903244+0000","last_became_peered":"2024-03-04T12:23:11.903244+0000","last_unstale":"2024-03-04T12:24:05.958858+0000","last_undegraded":"2024-03-04T12:24:05.958858+0000","last_fullsized":"2024-03-04T12:24:05.958858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:27.590418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029058+0000","last_change":"2024-03-04T12:23:12.925623+0000","last_active":"2024-03-04T12:24:06.029058+0000","last_peered":"2024-03-04T12:24:06.029058+0000","last_clean":"2024-03-04T12:24:06.029058+0000","last_became_active":"2024-03-04T12:23:12.925243+0000","last_became_peered":"2024-03-04T12:23:12.925243+0000","last_unstale":"2024-03-04T12:24:06.029058+0000","last_undegraded":"2024-03-04T12:24:06.029058+0000","last_fullsized":"2024-03-04T12:24:06.029058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:11:04.142655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953361+0000","last_change":"2024-03-04T12:23:16.946064+0000","last_active":"2024-03-04T12:24:05.953361+0000","last_peered":"2024-03-04T12:24:05.953361+0000","last_clean":"2024-03-04T12:24:05.953361+0000","last_became_active":"2024-03-04T12:23:16.945342+0000","last_became_peered":"2024-03-04T12:23:16.945342+0000","last_unstale":"2024-03-04T12:24:05.953361+0000","last_undegraded":"2024-03-04T12:24:05.953361+0000","last_fullsized":"2024-03-04T12:24:05.953361+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:43:37.500216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:50.997352+0000","last_change":"2024-03-04T12:23:11.899478+0000","last_active":"2024-03-04T12:24:50.997352+0000","last_peered":"2024-03-04T12:24:50.997352+0000","last_clean":"2024-03-04T12:24:50.997352+0000","last_became_active":"2024-03-04T12:23:11.899153+0000","last_became_peered":"2024-03-04T12:23:11.899153+0000","last_unstale":"2024-03-04T12:24:50.997352+0000","last_undegraded":"2024-03-04T12:24:50.997352+0000","last_fullsized":"2024-03-04T12:24:50.997352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:10.709054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.958202+0000","last_change":"2024-03-04T12:23:12.908560+0000","last_active":"2024-03-04T12:24:05.958202+0000","last_peered":"2024-03-04T12:24:05.958202+0000","last_clean":"2024-03-04T12:24:05.958202+0000","last_became_active":"2024-03-04T12:23:12.908221+0000","last_became_peered":"2024-03-04T12:23:12.908221+0000","last_unstale":"2024-03-04T12:24:05.958202+0000","last_undegraded":"2024-03-04T12:24:05.958202+0000","last_fullsized":"2024-03-04T12:24:05.958202+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:28:06.545945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.031257+0000","last_change":"2024-03-04T12:24:06.031257+0000","last_active":"2024-03-04T12:24:06.031257+0000","last_peered":"2024-03-04T12:24:06.031257+0000","last_clean":"2024-03-04T12:24:06.031257+0000","last_became_active":"2024-03-04T12:24:06.030409+0000","last_became_peered":"2024-03-04T12:24:06.030409+0000","last_unstale":"2024-03-04T12:24:06.031257+0000","last_undegraded":"2024-03-04T12:24:06.031257+0000","last_fullsized":"2024-03-04T12:24:06.031257+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:31.403224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954320+0000","last_change":"2024-03-04T12:23:11.898360+0000","last_active":"2024-03-04T12:24:05.954320+0000","last_peered":"2024-03-04T12:24:05.954320+0000","last_clean":"2024-03-04T12:24:05.954320+0000","last_became_active":"2024-03-04T12:23:11.897988+0000","last_became_peered":"2024-03-04T12:23:11.897988+0000","last_unstale":"2024-03-04T12:24:05.954320+0000","last_undegraded":"2024-03-04T12:24:05.954320+0000","last_fullsized":"2024-03-04T12:24:05.954320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:17.448032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028454+0000","last_change":"2024-03-04T12:23:12.918392+0000","last_active":"2024-03-04T12:24:06.028454+0000","last_peered":"2024-03-04T12:24:06.028454+0000","last_clean":"2024-03-04T12:24:06.028454+0000","last_became_active":"2024-03-04T12:23:12.917656+0000","last_became_peered":"2024-03-04T12:23:12.917656+0000","last_unstale":"2024-03-04T12:24:06.028454+0000","last_undegraded":"2024-03-04T12:24:06.028454+0000","last_fullsized":"2024-03-04T12:24:06.028454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:44:14.545925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955607+0000","last_change":"2024-03-04T12:23:16.941841+0000","last_active":"2024-03-04T12:24:05.955607+0000","last_peered":"2024-03-04T12:24:05.955607+0000","last_clean":"2024-03-04T12:24:05.955607+0000","last_became_active":"2024-03-04T12:23:16.941449+0000","last_became_peered":"2024-03-04T12:23:16.941449+0000","last_unstale":"2024-03-04T12:24:05.955607+0000","last_undegraded":"2024-03-04T12:24:05.955607+0000","last_fullsized":"2024-03-04T12:24:05.955607+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:50:35.451834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028130+0000","last_change":"2024-03-04T12:23:11.906723+0000","last_active":"2024-03-04T12:24:06.028130+0000","last_peered":"2024-03-04T12:24:06.028130+0000","last_clean":"2024-03-04T12:24:06.028130+0000","last_became_active":"2024-03-04T12:23:11.906357+0000","last_became_peered":"2024-03-04T12:23:11.906357+0000","last_unstale":"2024-03-04T12:24:06.028130+0000","last_undegraded":"2024-03-04T12:24:06.028130+0000","last_fullsized":"2024-03-04T12:24:06.028130+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:09.592236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955549+0000","last_change":"2024-03-04T12:23:12.916435+0000","last_active":"2024-03-04T12:24:05.955549+0000","last_peered":"2024-03-04T12:24:05.955549+0000","last_clean":"2024-03-04T12:24:05.955549+0000","last_became_active":"2024-03-04T12:23:12.915607+0000","last_became_peered":"2024-03-04T12:23:12.915607+0000","last_unstale":"2024-03-04T12:24:05.955549+0000","last_undegraded":"2024-03-04T12:24:05.955549+0000","last_fullsized":"2024-03-04T12:24:05.955549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:41:32.900344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954910+0000","last_change":"2024-03-04T12:23:16.944342+0000","last_active":"2024-03-04T12:24:05.954910+0000","last_peered":"2024-03-04T12:24:05.954910+0000","last_clean":"2024-03-04T12:24:05.954910+0000","last_became_active":"2024-03-04T12:23:16.943683+0000","last_became_peered":"2024-03-04T12:23:16.943683+0000","last_unstale":"2024-03-04T12:24:05.954910+0000","last_undegraded":"2024-03-04T12:24:05.954910+0000","last_fullsized":"2024-03-04T12:24:05.954910+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:19.658860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028539+0000","last_change":"2024-03-04T12:23:11.899194+0000","last_active":"2024-03-04T12:24:06.028539+0000","last_peered":"2024-03-04T12:24:06.028539+0000","last_clean":"2024-03-04T12:24:06.028539+0000","last_became_active":"2024-03-04T12:23:11.898109+0000","last_became_peered":"2024-03-04T12:23:11.898109+0000","last_unstale":"2024-03-04T12:24:06.028539+0000","last_undegraded":"2024-03-04T12:24:06.028539+0000","last_fullsized":"2024-03-04T12:24:06.028539+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:08.808202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.951684+0000","last_change":"2024-03-04T12:23:12.916549+0000","last_active":"2024-03-04T12:24:05.951684+0000","last_peered":"2024-03-04T12:24:05.951684+0000","last_clean":"2024-03-04T12:24:05.951684+0000","last_became_active":"2024-03-04T12:23:12.916204+0000","last_became_peered":"2024-03-04T12:23:12.916204+0000","last_unstale":"2024-03-04T12:24:05.951684+0000","last_undegraded":"2024-03-04T12:24:05.951684+0000","last_fullsized":"2024-03-04T12:24:05.951684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:56.412600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955949+0000","last_change":"2024-03-04T12:23:16.945903+0000","last_active":"2024-03-04T12:24:05.955949+0000","last_peered":"2024-03-04T12:24:05.955949+0000","last_clean":"2024-03-04T12:24:05.955949+0000","last_became_active":"2024-03-04T12:23:16.945535+0000","last_became_peered":"2024-03-04T12:23:16.945535+0000","last_unstale":"2024-03-04T12:24:05.955949+0000","last_undegraded":"2024-03-04T12:24:05.955949+0000","last_fullsized":"2024-03-04T12:24:05.955949+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:45:01.178370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":175,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:51.998164+0000","last_change":"2024-03-04T12:23:11.901570+0000","last_active":"2024-03-04T12:24:51.998164+0000","last_peered":"2024-03-04T12:24:51.998164+0000","last_clean":"2024-03-04T12:24:51.998164+0000","last_became_active":"2024-03-04T12:23:11.900702+0000","last_became_peered":"2024-03-04T12:23:11.900702+0000","last_unstale":"2024-03-04T12:24:51.998164+0000","last_undegraded":"2024-03-04T12:24:51.998164+0000","last_fullsized":"2024-03-04T12:24:51.998164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:18.758453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028696+0000","last_change":"2024-03-04T12:23:12.919571+0000","last_active":"2024-03-04T12:24:06.028696+0000","last_peered":"2024-03-04T12:24:06.028696+0000","last_clean":"2024-03-04T12:24:06.028696+0000","last_became_active":"2024-03-04T12:23:12.919143+0000","last_became_peered":"2024-03-04T12:23:12.919143+0000","last_unstale":"2024-03-04T12:24:06.028696+0000","last_undegraded":"2024-03-04T12:24:06.028696+0000","last_fullsized":"2024-03-04T12:24:06.028696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:25.459417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.951599+0000","last_change":"2024-03-04T12:23:16.946380+0000","last_active":"2024-03-04T12:24:05.951599+0000","last_peered":"2024-03-04T12:24:05.951599+0000","last_clean":"2024-03-04T12:24:05.951599+0000","last_became_active":"2024-03-04T12:23:16.945983+0000","last_became_peered":"2024-03-04T12:23:16.945983+0000","last_unstale":"2024-03-04T12:24:05.951599+0000","last_undegraded":"2024-03-04T12:24:05.951599+0000","last_fullsized":"2024-03-04T12:24:05.951599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:39:43.297619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955609+0000","last_change":"2024-03-04T12:23:11.897176+0000","last_active":"2024-03-04T12:24:05.955609+0000","last_peered":"2024-03-04T12:24:05.955609+0000","last_clean":"2024-03-04T12:24:05.955609+0000","last_became_active":"2024-03-04T12:23:11.896747+0000","last_became_peered":"2024-03-04T12:23:11.896747+0000","last_unstale":"2024-03-04T12:24:05.955609+0000","last_undegraded":"2024-03-04T12:24:05.955609+0000","last_fullsized":"2024-03-04T12:24:05.955609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:02.206704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.956455+0000","last_change":"2024-03-04T12:23:12.922021+0000","last_active":"2024-03-04T12:24:05.956455+0000","last_peered":"2024-03-04T12:24:05.956455+0000","last_clean":"2024-03-04T12:24:05.956455+0000","last_became_active":"2024-03-04T12:23:12.918724+0000","last_became_peered":"2024-03-04T12:23:12.918724+0000","last_unstale":"2024-03-04T12:24:05.956455+0000","last_undegraded":"2024-03-04T12:24:05.956455+0000","last_fullsized":"2024-03-04T12:24:05.956455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:49:09.017979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952895+0000","last_change":"2024-03-04T12:23:16.943271+0000","last_active":"2024-03-04T12:24:05.952895+0000","last_peered":"2024-03-04T12:24:05.952895+0000","last_clean":"2024-03-04T12:24:05.952895+0000","last_became_active":"2024-03-04T12:23:16.942400+0000","last_became_peered":"2024-03-04T12:23:16.942400+0000","last_unstale":"2024-03-04T12:24:05.952895+0000","last_undegraded":"2024-03-04T12:24:05.952895+0000","last_fullsized":"2024-03-04T12:24:05.952895+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:06.010960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028932+0000","last_change":"2024-03-04T12:23:11.896706+0000","last_active":"2024-03-04T12:24:06.028932+0000","last_peered":"2024-03-04T12:24:06.028932+0000","last_clean":"2024-03-04T12:24:06.028932+0000","last_became_active":"2024-03-04T12:23:11.896100+0000","last_became_peered":"2024-03-04T12:23:11.896100+0000","last_unstale":"2024-03-04T12:24:06.028932+0000","last_undegraded":"2024-03-04T12:24:06.028932+0000","last_fullsized":"2024-03-04T12:24:06.028932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:42:19.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028482+0000","last_change":"2024-03-04T12:23:12.923631+0000","last_active":"2024-03-04T12:24:06.028482+0000","last_peered":"2024-03-04T12:24:06.028482+0000","last_clean":"2024-03-04T12:24:06.028482+0000","last_became_active":"2024-03-04T12:23:12.922487+0000","last_became_peered":"2024-03-04T12:23:12.922487+0000","last_unstale":"2024-03-04T12:24:06.028482+0000","last_undegraded":"2024-03-04T12:24:06.028482+0000","last_fullsized":"2024-03-04T12:24:06.028482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:30:33.923064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.032470+0000","last_change":"2024-03-04T12:24:06.032075+0000","last_active":"2024-03-04T12:24:06.032470+0000","last_peered":"2024-03-04T12:24:06.032470+0000","last_clean":"2024-03-04T12:24:06.032470+0000","last_became_active":"2024-03-04T12:24:06.031639+0000","last_became_peered":"2024-03-04T12:24:06.031639+0000","last_unstale":"2024-03-04T12:24:06.032470+0000","last_undegraded":"2024-03-04T12:24:06.032470+0000","last_fullsized":"2024-03-04T12:24:06.032470+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:44:48.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'34","reported_seq":214,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:51.998187+0000","last_change":"2024-03-04T12:23:11.898903+0000","last_active":"2024-03-04T12:24:51.998187+0000","last_peered":"2024-03-04T12:24:51.998187+0000","last_clean":"2024-03-04T12:24:51.998187+0000","last_became_active":"2024-03-04T12:23:11.898477+0000","last_became_peered":"2024-03-04T12:23:11.898477+0000","last_unstale":"2024-03-04T12:24:51.998187+0000","last_undegraded":"2024-03-04T12:24:51.998187+0000","last_fullsized":"2024-03-04T12:24:51.998187+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:36:58.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60269,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":153,"num_read_kb":135,"num_write":36,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.958633+0000","last_change":"2024-03-04T12:23:12.908497+0000","last_active":"2024-03-04T12:24:05.958633+0000","last_peered":"2024-03-04T12:24:05.958633+0000","last_clean":"2024-03-04T12:24:05.958633+0000","last_became_active":"2024-03-04T12:23:12.907847+0000","last_became_peered":"2024-03-04T12:23:12.907847+0000","last_unstale":"2024-03-04T12:24:05.958633+0000","last_undegraded":"2024-03-04T12:24:05.958633+0000","last_fullsized":"2024-03-04T12:24:05.958633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:51:44.592851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029273+0000","last_change":"2024-03-04T12:23:16.951844+0000","last_active":"2024-03-04T12:24:06.029273+0000","last_peered":"2024-03-04T12:24:06.029273+0000","last_clean":"2024-03-04T12:24:06.029273+0000","last_became_active":"2024-03-04T12:23:16.951491+0000","last_became_peered":"2024-03-04T12:23:16.951491+0000","last_unstale":"2024-03-04T12:24:06.029273+0000","last_undegraded":"2024-03-04T12:24:06.029273+0000","last_fullsized":"2024-03-04T12:24:06.029273+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:54.098239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953662+0000","last_change":"2024-03-04T12:23:11.899592+0000","last_active":"2024-03-04T12:24:05.953662+0000","last_peered":"2024-03-04T12:24:05.953662+0000","last_clean":"2024-03-04T12:24:05.953662+0000","last_became_active":"2024-03-04T12:23:11.899275+0000","last_became_peered":"2024-03-04T12:23:11.899275+0000","last_unstale":"2024-03-04T12:24:05.953662+0000","last_undegraded":"2024-03-04T12:24:05.953662+0000","last_fullsized":"2024-03-04T12:24:05.953662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:02.184268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028014+0000","last_change":"2024-03-04T12:23:12.910084+0000","last_active":"2024-03-04T12:24:06.028014+0000","last_peered":"2024-03-04T12:24:06.028014+0000","last_clean":"2024-03-04T12:24:06.028014+0000","last_became_active":"2024-03-04T12:23:12.909719+0000","last_became_peered":"2024-03-04T12:23:12.909719+0000","last_unstale":"2024-03-04T12:24:06.028014+0000","last_undegraded":"2024-03-04T12:24:06.028014+0000","last_fullsized":"2024-03-04T12:24:06.028014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:16:41.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029579+0000","last_change":"2024-03-04T12:24:06.029579+0000","last_active":"2024-03-04T12:24:06.029579+0000","last_peered":"2024-03-04T12:24:06.029579+0000","last_clean":"2024-03-04T12:24:06.029579+0000","last_became_active":"2024-03-04T12:24:06.029042+0000","last_became_peered":"2024-03-04T12:24:06.029042+0000","last_unstale":"2024-03-04T12:24:06.029579+0000","last_undegraded":"2024-03-04T12:24:06.029579+0000","last_fullsized":"2024-03-04T12:24:06.029579+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:59:35.867212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029279+0000","last_change":"2024-03-04T12:23:11.897506+0000","last_active":"2024-03-04T12:24:06.029279+0000","last_peered":"2024-03-04T12:24:06.029279+0000","last_clean":"2024-03-04T12:24:06.029279+0000","last_became_active":"2024-03-04T12:23:11.897090+0000","last_became_peered":"2024-03-04T12:23:11.897090+0000","last_unstale":"2024-03-04T12:24:06.029279+0000","last_undegraded":"2024-03-04T12:24:06.029279+0000","last_fullsized":"2024-03-04T12:24:06.029279+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:10:27.868713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955402+0000","last_change":"2024-03-04T12:23:12.921825+0000","last_active":"2024-03-04T12:24:05.955402+0000","last_peered":"2024-03-04T12:24:05.955402+0000","last_clean":"2024-03-04T12:24:05.955402+0000","last_became_active":"2024-03-04T12:23:12.921491+0000","last_became_peered":"2024-03-04T12:23:12.921491+0000","last_unstale":"2024-03-04T12:24:05.955402+0000","last_undegraded":"2024-03-04T12:24:05.955402+0000","last_fullsized":"2024-03-04T12:24:05.955402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:17:08.203502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952170+0000","last_change":"2024-03-04T12:23:16.951920+0000","last_active":"2024-03-04T12:24:05.952170+0000","last_peered":"2024-03-04T12:24:05.952170+0000","last_clean":"2024-03-04T12:24:05.952170+0000","last_became_active":"2024-03-04T12:23:16.951507+0000","last_became_peered":"2024-03-04T12:23:16.951507+0000","last_unstale":"2024-03-04T12:24:05.952170+0000","last_undegraded":"2024-03-04T12:24:05.952170+0000","last_fullsized":"2024-03-04T12:24:05.952170+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:27.673692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":109,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:50.996588+0000","last_change":"2024-03-04T12:23:11.897722+0000","last_active":"2024-03-04T12:24:50.996588+0000","last_peered":"2024-03-04T12:24:50.996588+0000","last_clean":"2024-03-04T12:24:50.996588+0000","last_became_active":"2024-03-04T12:23:11.897276+0000","last_became_peered":"2024-03-04T12:23:11.897276+0000","last_unstale":"2024-03-04T12:24:50.996588+0000","last_undegraded":"2024-03-04T12:24:50.996588+0000","last_fullsized":"2024-03-04T12:24:50.996588+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:37.968804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":77,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955021+0000","last_change":"2024-03-04T12:23:12.914773+0000","last_active":"2024-03-04T12:24:05.955021+0000","last_peered":"2024-03-04T12:24:05.955021+0000","last_clean":"2024-03-04T12:24:05.955021+0000","last_became_active":"2024-03-04T12:23:12.914431+0000","last_became_peered":"2024-03-04T12:23:12.914431+0000","last_unstale":"2024-03-04T12:24:05.955021+0000","last_undegraded":"2024-03-04T12:24:05.955021+0000","last_fullsized":"2024-03-04T12:24:05.955021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:50.234491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028425+0000","last_change":"2024-03-04T12:23:16.952405+0000","last_active":"2024-03-04T12:24:06.028425+0000","last_peered":"2024-03-04T12:24:06.028425+0000","last_clean":"2024-03-04T12:24:06.028425+0000","last_became_active":"2024-03-04T12:23:16.951915+0000","last_became_peered":"2024-03-04T12:23:16.951915+0000","last_unstale":"2024-03-04T12:24:06.028425+0000","last_undegraded":"2024-03-04T12:24:06.028425+0000","last_fullsized":"2024-03-04T12:24:06.028425+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:38:03.765883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.957615+0000","last_change":"2024-03-04T12:23:11.903267+0000","last_active":"2024-03-04T12:24:05.957615+0000","last_peered":"2024-03-04T12:24:05.957615+0000","last_clean":"2024-03-04T12:24:05.957615+0000","last_became_active":"2024-03-04T12:23:11.902870+0000","last_became_peered":"2024-03-04T12:23:11.902870+0000","last_unstale":"2024-03-04T12:24:05.957615+0000","last_undegraded":"2024-03-04T12:24:05.957615+0000","last_fullsized":"2024-03-04T12:24:05.957615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:33:43.679869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028314+0000","last_change":"2024-03-04T12:23:12.925303+0000","last_active":"2024-03-04T12:24:06.028314+0000","last_peered":"2024-03-04T12:24:06.028314+0000","last_clean":"2024-03-04T12:24:06.028314+0000","last_became_active":"2024-03-04T12:23:12.924945+0000","last_became_peered":"2024-03-04T12:23:12.924945+0000","last_unstale":"2024-03-04T12:24:06.028314+0000","last_undegraded":"2024-03-04T12:24:06.028314+0000","last_fullsized":"2024-03-04T12:24:06.028314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:49.680925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028148+0000","last_change":"2024-03-04T12:23:16.951457+0000","last_active":"2024-03-04T12:24:06.028148+0000","last_peered":"2024-03-04T12:24:06.028148+0000","last_clean":"2024-03-04T12:24:06.028148+0000","last_became_active":"2024-03-04T12:23:16.950835+0000","last_became_peered":"2024-03-04T12:23:16.950835+0000","last_unstale":"2024-03-04T12:24:06.028148+0000","last_undegraded":"2024-03-04T12:24:06.028148+0000","last_fullsized":"2024-03-04T12:24:06.028148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:39.337275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.959083+0000","last_change":"2024-03-04T12:23:11.903718+0000","last_active":"2024-03-04T12:24:05.959083+0000","last_peered":"2024-03-04T12:24:05.959083+0000","last_clean":"2024-03-04T12:24:05.959083+0000","last_became_active":"2024-03-04T12:23:11.902814+0000","last_became_peered":"2024-03-04T12:23:11.902814+0000","last_unstale":"2024-03-04T12:24:05.959083+0000","last_undegraded":"2024-03-04T12:24:05.959083+0000","last_fullsized":"2024-03-04T12:24:05.959083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:40:30.978576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.958392+0000","last_change":"2024-03-04T12:24:05.958392+0000","last_active":"2024-03-04T12:24:05.958392+0000","last_peered":"2024-03-04T12:24:05.958392+0000","last_clean":"2024-03-04T12:24:05.958392+0000","last_became_active":"2024-03-04T12:24:05.958012+0000","last_became_peered":"2024-03-04T12:24:05.958012+0000","last_unstale":"2024-03-04T12:24:05.958392+0000","last_undegraded":"2024-03-04T12:24:05.958392+0000","last_fullsized":"2024-03-04T12:24:05.958392+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:43:50.400090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.958152+0000","last_change":"2024-03-04T12:23:16.950046+0000","last_active":"2024-03-04T12:24:05.958152+0000","last_peered":"2024-03-04T12:24:05.958152+0000","last_clean":"2024-03-04T12:24:05.958152+0000","last_became_active":"2024-03-04T12:23:16.949632+0000","last_became_peered":"2024-03-04T12:23:16.949632+0000","last_unstale":"2024-03-04T12:24:05.958152+0000","last_undegraded":"2024-03-04T12:24:05.958152+0000","last_fullsized":"2024-03-04T12:24:05.958152+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:25.817212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952134+0000","last_change":"2024-03-04T12:23:11.897016+0000","last_active":"2024-03-04T12:24:05.952134+0000","last_peered":"2024-03-04T12:24:05.952134+0000","last_clean":"2024-03-04T12:24:05.952134+0000","last_became_active":"2024-03-04T12:23:11.896668+0000","last_became_peered":"2024-03-04T12:23:11.896668+0000","last_unstale":"2024-03-04T12:24:05.952134+0000","last_undegraded":"2024-03-04T12:24:05.952134+0000","last_fullsized":"2024-03-04T12:24:05.952134+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:50:01.089658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954296+0000","last_change":"2024-03-04T12:23:12.908169+0000","last_active":"2024-03-04T12:24:05.954296+0000","last_peered":"2024-03-04T12:24:05.954296+0000","last_clean":"2024-03-04T12:24:05.954296+0000","last_became_active":"2024-03-04T12:23:12.907786+0000","last_became_peered":"2024-03-04T12:23:12.907786+0000","last_unstale":"2024-03-04T12:24:05.954296+0000","last_undegraded":"2024-03-04T12:24:05.954296+0000","last_fullsized":"2024-03-04T12:24:05.954296+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:10.602216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028165+0000","last_change":"2024-03-04T12:23:16.944232+0000","last_active":"2024-03-04T12:24:06.028165+0000","last_peered":"2024-03-04T12:24:06.028165+0000","last_clean":"2024-03-04T12:24:06.028165+0000","last_became_active":"2024-03-04T12:23:16.943855+0000","last_became_peered":"2024-03-04T12:23:16.943855+0000","last_unstale":"2024-03-04T12:24:06.028165+0000","last_undegraded":"2024-03-04T12:24:06.028165+0000","last_fullsized":"2024-03-04T12:24:06.028165+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:37:12.083622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:52.012166+0000","last_change":"2024-03-04T12:23:11.899742+0000","last_active":"2024-03-04T12:24:52.012166+0000","last_peered":"2024-03-04T12:24:52.012166+0000","last_clean":"2024-03-04T12:24:52.012166+0000","last_became_active":"2024-03-04T12:23:11.898445+0000","last_became_peered":"2024-03-04T12:23:11.898445+0000","last_unstale":"2024-03-04T12:24:52.012166+0000","last_undegraded":"2024-03-04T12:24:52.012166+0000","last_fullsized":"2024-03-04T12:24:52.012166+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:13.128348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954237+0000","last_change":"2024-03-04T12:23:12.921321+0000","last_active":"2024-03-04T12:24:05.954237+0000","last_peered":"2024-03-04T12:24:05.954237+0000","last_clean":"2024-03-04T12:24:05.954237+0000","last_became_active":"2024-03-04T12:23:12.918119+0000","last_became_peered":"2024-03-04T12:23:12.918119+0000","last_unstale":"2024-03-04T12:24:05.954237+0000","last_undegraded":"2024-03-04T12:24:05.954237+0000","last_fullsized":"2024-03-04T12:24:05.954237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:31.693583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028259+0000","last_change":"2024-03-04T12:23:16.943887+0000","last_active":"2024-03-04T12:24:06.028259+0000","last_peered":"2024-03-04T12:24:06.028259+0000","last_clean":"2024-03-04T12:24:06.028259+0000","last_became_active":"2024-03-04T12:23:16.943487+0000","last_became_peered":"2024-03-04T12:23:16.943487+0000","last_unstale":"2024-03-04T12:24:06.028259+0000","last_undegraded":"2024-03-04T12:24:06.028259+0000","last_fullsized":"2024-03-04T12:24:06.028259+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:12.451176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028804+0000","last_change":"2024-03-04T12:23:12.918363+0000","last_active":"2024-03-04T12:24:06.028804+0000","last_peered":"2024-03-04T12:24:06.028804+0000","last_clean":"2024-03-04T12:24:06.028804+0000","last_became_active":"2024-03-04T12:23:12.917563+0000","last_became_peered":"2024-03-04T12:23:12.917563+0000","last_unstale":"2024-03-04T12:24:06.028804+0000","last_undegraded":"2024-03-04T12:24:06.028804+0000","last_fullsized":"2024-03-04T12:24:06.028804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:26:35.297513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":107,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:51.997586+0000","last_change":"2024-03-04T12:23:11.904072+0000","last_active":"2024-03-04T12:24:51.997586+0000","last_peered":"2024-03-04T12:24:51.997586+0000","last_clean":"2024-03-04T12:24:51.997586+0000","last_became_active":"2024-03-04T12:23:11.903097+0000","last_became_peered":"2024-03-04T12:23:11.903097+0000","last_unstale":"2024-03-04T12:24:51.997586+0000","last_undegraded":"2024-03-04T12:24:51.997586+0000","last_fullsized":"2024-03-04T12:24:51.997586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:36.606667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955543+0000","last_change":"2024-03-04T12:23:16.951019+0000","last_active":"2024-03-04T12:24:05.955543+0000","last_peered":"2024-03-04T12:24:05.955543+0000","last_clean":"2024-03-04T12:24:05.955543+0000","last_became_active":"2024-03-04T12:23:16.950524+0000","last_became_peered":"2024-03-04T12:23:16.950524+0000","last_unstale":"2024-03-04T12:24:05.955543+0000","last_undegraded":"2024-03-04T12:24:05.955543+0000","last_fullsized":"2024-03-04T12:24:05.955543+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:48.376539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954963+0000","last_change":"2024-03-04T12:23:12.927615+0000","last_active":"2024-03-04T12:24:05.954963+0000","last_peered":"2024-03-04T12:24:05.954963+0000","last_clean":"2024-03-04T12:24:05.954963+0000","last_became_active":"2024-03-04T12:23:12.927190+0000","last_became_peered":"2024-03-04T12:23:12.927190+0000","last_unstale":"2024-03-04T12:24:05.954963+0000","last_undegraded":"2024-03-04T12:24:05.954963+0000","last_fullsized":"2024-03-04T12:24:05.954963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:54:19.019030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952706+0000","last_change":"2024-03-04T12:23:11.893542+0000","last_active":"2024-03-04T12:24:05.952706+0000","last_peered":"2024-03-04T12:24:05.952706+0000","last_clean":"2024-03-04T12:24:05.952706+0000","last_became_active":"2024-03-04T12:23:11.893166+0000","last_became_peered":"2024-03-04T12:23:11.893166+0000","last_unstale":"2024-03-04T12:24:05.952706+0000","last_undegraded":"2024-03-04T12:24:05.952706+0000","last_fullsized":"2024-03-04T12:24:05.952706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:32.117863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954934+0000","last_change":"2024-03-04T12:21:55.865103+0000","last_active":"2024-03-04T12:24:05.954934+0000","last_peered":"2024-03-04T12:24:05.954934+0000","last_clean":"2024-03-04T12:24:05.954934+0000","last_became_active":"2024-03-04T12:21:55.864553+0000","last_became_peered":"2024-03-04T12:21:55.864553+0000","last_unstale":"2024-03-04T12:24:05.954934+0000","last_undegraded":"2024-03-04T12:24:05.954934+0000","last_fullsized":"2024-03-04T12:24:05.954934+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_clean_scrub_stamp":"2024-03-04T12:18:28.442423+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:31.052782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029538+0000","last_change":"2024-03-04T12:23:16.945852+0000","last_active":"2024-03-04T12:24:06.029538+0000","last_peered":"2024-03-04T12:24:06.029538+0000","last_clean":"2024-03-04T12:24:06.029538+0000","last_became_active":"2024-03-04T12:23:16.945496+0000","last_became_peered":"2024-03-04T12:23:16.945496+0000","last_unstale":"2024-03-04T12:24:06.029538+0000","last_undegraded":"2024-03-04T12:24:06.029538+0000","last_fullsized":"2024-03-04T12:24:06.029538+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:35:10.880757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028066+0000","last_change":"2024-03-04T12:23:12.919219+0000","last_active":"2024-03-04T12:24:06.028066+0000","last_peered":"2024-03-04T12:24:06.028066+0000","last_clean":"2024-03-04T12:24:06.028066+0000","last_became_active":"2024-03-04T12:23:12.918897+0000","last_became_peered":"2024-03-04T12:23:12.918897+0000","last_unstale":"2024-03-04T12:24:06.028066+0000","last_undegraded":"2024-03-04T12:24:06.028066+0000","last_fullsized":"2024-03-04T12:24:06.028066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:24:32.594172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954885+0000","last_change":"2024-03-04T12:23:11.894544+0000","last_active":"2024-03-04T12:24:05.954885+0000","last_peered":"2024-03-04T12:24:05.954885+0000","last_clean":"2024-03-04T12:24:05.954885+0000","last_became_active":"2024-03-04T12:23:11.894188+0000","last_became_peered":"2024-03-04T12:23:11.894188+0000","last_unstale":"2024-03-04T12:24:05.954885+0000","last_undegraded":"2024-03-04T12:24:05.954885+0000","last_fullsized":"2024-03-04T12:24:05.954885+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:28:14.687530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955450+0000","last_change":"2024-03-04T12:23:16.945510+0000","last_active":"2024-03-04T12:24:05.955450+0000","last_peered":"2024-03-04T12:24:05.955450+0000","last_clean":"2024-03-04T12:24:05.955450+0000","last_became_active":"2024-03-04T12:23:16.945140+0000","last_became_peered":"2024-03-04T12:23:16.945140+0000","last_unstale":"2024-03-04T12:24:05.955450+0000","last_undegraded":"2024-03-04T12:24:05.955450+0000","last_fullsized":"2024-03-04T12:24:05.955450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:44.102503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.958797+0000","last_change":"2024-03-04T12:23:11.904515+0000","last_active":"2024-03-04T12:24:05.958797+0000","last_peered":"2024-03-04T12:24:05.958797+0000","last_clean":"2024-03-04T12:24:05.958797+0000","last_became_active":"2024-03-04T12:23:11.904149+0000","last_became_peered":"2024-03-04T12:23:11.904149+0000","last_unstale":"2024-03-04T12:24:05.958797+0000","last_undegraded":"2024-03-04T12:24:05.958797+0000","last_fullsized":"2024-03-04T12:24:05.958797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:04:51.382626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028279+0000","last_change":"2024-03-04T12:23:12.908749+0000","last_active":"2024-03-04T12:24:06.028279+0000","last_peered":"2024-03-04T12:24:06.028279+0000","last_clean":"2024-03-04T12:24:06.028279+0000","last_became_active":"2024-03-04T12:23:12.908405+0000","last_became_peered":"2024-03-04T12:23:12.908405+0000","last_unstale":"2024-03-04T12:24:06.028279+0000","last_undegraded":"2024-03-04T12:24:06.028279+0000","last_fullsized":"2024-03-04T12:24:06.028279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:10.472953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955442+0000","last_change":"2024-03-04T12:23:16.951145+0000","last_active":"2024-03-04T12:24:05.955442+0000","last_peered":"2024-03-04T12:24:05.955442+0000","last_clean":"2024-03-04T12:24:05.955442+0000","last_became_active":"2024-03-04T12:23:16.950702+0000","last_became_peered":"2024-03-04T12:23:16.950702+0000","last_unstale":"2024-03-04T12:24:05.955442+0000","last_undegraded":"2024-03-04T12:24:05.955442+0000","last_fullsized":"2024-03-04T12:24:05.955442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:42.720616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'11","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952657+0000","last_change":"2024-03-04T12:23:11.893810+0000","last_active":"2024-03-04T12:24:05.952657+0000","last_peered":"2024-03-04T12:24:05.952657+0000","last_clean":"2024-03-04T12:24:05.952657+0000","last_became_active":"2024-03-04T12:23:11.893465+0000","last_became_peered":"2024-03-04T12:23:11.893465+0000","last_unstale":"2024-03-04T12:24:05.952657+0000","last_undegraded":"2024-03-04T12:24:05.952657+0000","last_fullsized":"2024-03-04T12:24:05.952657+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:27:41.199786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955046+0000","last_change":"2024-03-04T12:23:12.925763+0000","last_active":"2024-03-04T12:24:05.955046+0000","last_peered":"2024-03-04T12:24:05.955046+0000","last_clean":"2024-03-04T12:24:05.955046+0000","last_became_active":"2024-03-04T12:23:12.925429+0000","last_became_peered":"2024-03-04T12:23:12.925429+0000","last_unstale":"2024-03-04T12:24:05.955046+0000","last_undegraded":"2024-03-04T12:24:05.955046+0000","last_fullsized":"2024-03-04T12:24:05.955046+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:46.814459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953078+0000","last_change":"2024-03-04T12:23:16.942624+0000","last_active":"2024-03-04T12:24:05.953078+0000","last_peered":"2024-03-04T12:24:05.953078+0000","last_clean":"2024-03-04T12:24:05.953078+0000","last_became_active":"2024-03-04T12:23:16.942287+0000","last_became_peered":"2024-03-04T12:23:16.942287+0000","last_unstale":"2024-03-04T12:24:05.953078+0000","last_undegraded":"2024-03-04T12:24:05.953078+0000","last_fullsized":"2024-03-04T12:24:05.953078+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:59.479611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952809+0000","last_change":"2024-03-04T12:23:11.891504+0000","last_active":"2024-03-04T12:24:05.952809+0000","last_peered":"2024-03-04T12:24:05.952809+0000","last_clean":"2024-03-04T12:24:05.952809+0000","last_became_active":"2024-03-04T12:23:11.891134+0000","last_became_peered":"2024-03-04T12:23:11.891134+0000","last_unstale":"2024-03-04T12:24:05.952809+0000","last_undegraded":"2024-03-04T12:24:05.952809+0000","last_fullsized":"2024-03-04T12:24:05.952809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:41.481003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955120+0000","last_change":"2024-03-04T12:23:12.928989+0000","last_active":"2024-03-04T12:24:05.955120+0000","last_peered":"2024-03-04T12:24:05.955120+0000","last_clean":"2024-03-04T12:24:05.955120+0000","last_became_active":"2024-03-04T12:23:12.928609+0000","last_became_peered":"2024-03-04T12:23:12.928609+0000","last_unstale":"2024-03-04T12:24:05.955120+0000","last_undegraded":"2024-03-04T12:24:05.955120+0000","last_fullsized":"2024-03-04T12:24:05.955120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:39.979546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952912+0000","last_change":"2024-03-04T12:23:16.949590+0000","last_active":"2024-03-04T12:24:05.952912+0000","last_peered":"2024-03-04T12:24:05.952912+0000","last_clean":"2024-03-04T12:24:05.952912+0000","last_became_active":"2024-03-04T12:23:16.949086+0000","last_became_peered":"2024-03-04T12:23:16.949086+0000","last_unstale":"2024-03-04T12:24:05.952912+0000","last_undegraded":"2024-03-04T12:24:05.952912+0000","last_fullsized":"2024-03-04T12:24:05.952912+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:07:09.819097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028494+0000","last_change":"2024-03-04T12:23:11.894110+0000","last_active":"2024-03-04T12:24:06.028494+0000","last_peered":"2024-03-04T12:24:06.028494+0000","last_clean":"2024-03-04T12:24:06.028494+0000","last_became_active":"2024-03-04T12:23:11.893419+0000","last_became_peered":"2024-03-04T12:23:11.893419+0000","last_unstale":"2024-03-04T12:24:06.028494+0000","last_undegraded":"2024-03-04T12:24:06.028494+0000","last_fullsized":"2024-03-04T12:24:06.028494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:53.963989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.957824+0000","last_change":"2024-03-04T12:23:12.907403+0000","last_active":"2024-03-04T12:24:05.957824+0000","last_peered":"2024-03-04T12:24:05.957824+0000","last_clean":"2024-03-04T12:24:05.957824+0000","last_became_active":"2024-03-04T12:23:12.906674+0000","last_became_peered":"2024-03-04T12:23:12.906674+0000","last_unstale":"2024-03-04T12:24:05.957824+0000","last_undegraded":"2024-03-04T12:24:05.957824+0000","last_fullsized":"2024-03-04T12:24:05.957824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:49:01.133676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952321+0000","last_change":"2024-03-04T12:23:16.955113+0000","last_active":"2024-03-04T12:24:05.952321+0000","last_peered":"2024-03-04T12:24:05.952321+0000","last_clean":"2024-03-04T12:24:05.952321+0000","last_became_active":"2024-03-04T12:23:16.951624+0000","last_became_peered":"2024-03-04T12:23:16.951624+0000","last_unstale":"2024-03-04T12:24:05.952321+0000","last_undegraded":"2024-03-04T12:24:05.952321+0000","last_fullsized":"2024-03-04T12:24:05.952321+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:22:23.935460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955685+0000","last_change":"2024-03-04T12:23:11.900213+0000","last_active":"2024-03-04T12:24:05.955685+0000","last_peered":"2024-03-04T12:24:05.955685+0000","last_clean":"2024-03-04T12:24:05.955685+0000","last_became_active":"2024-03-04T12:23:11.899832+0000","last_became_peered":"2024-03-04T12:23:11.899832+0000","last_unstale":"2024-03-04T12:24:05.955685+0000","last_undegraded":"2024-03-04T12:24:05.955685+0000","last_fullsized":"2024-03-04T12:24:05.955685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:55:24.553309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954634+0000","last_change":"2024-03-04T12:23:12.913315+0000","last_active":"2024-03-04T12:24:05.954634+0000","last_peered":"2024-03-04T12:24:05.954634+0000","last_clean":"2024-03-04T12:24:05.954634+0000","last_became_active":"2024-03-04T12:23:12.912850+0000","last_became_peered":"2024-03-04T12:23:12.912850+0000","last_unstale":"2024-03-04T12:24:05.954634+0000","last_undegraded":"2024-03-04T12:24:05.954634+0000","last_fullsized":"2024-03-04T12:24:05.954634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:55.140866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953110+0000","last_change":"2024-03-04T12:23:16.943295+0000","last_active":"2024-03-04T12:24:05.953110+0000","last_peered":"2024-03-04T12:24:05.953110+0000","last_clean":"2024-03-04T12:24:05.953110+0000","last_became_active":"2024-03-04T12:23:16.942853+0000","last_became_peered":"2024-03-04T12:23:16.942853+0000","last_unstale":"2024-03-04T12:24:05.953110+0000","last_undegraded":"2024-03-04T12:24:05.953110+0000","last_fullsized":"2024-03-04T12:24:05.953110+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:05:51.162179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953604+0000","last_change":"2024-03-04T12:23:11.897870+0000","last_active":"2024-03-04T12:24:05.953604+0000","last_peered":"2024-03-04T12:24:05.953604+0000","last_clean":"2024-03-04T12:24:05.953604+0000","last_became_active":"2024-03-04T12:23:11.897443+0000","last_became_peered":"2024-03-04T12:23:11.897443+0000","last_unstale":"2024-03-04T12:24:05.953604+0000","last_undegraded":"2024-03-04T12:24:05.953604+0000","last_fullsized":"2024-03-04T12:24:05.953604+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:49:37.875541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955202+0000","last_change":"2024-03-04T12:23:12.916441+0000","last_active":"2024-03-04T12:24:05.955202+0000","last_peered":"2024-03-04T12:24:05.955202+0000","last_clean":"2024-03-04T12:24:05.955202+0000","last_became_active":"2024-03-04T12:23:12.916078+0000","last_became_peered":"2024-03-04T12:23:12.916078+0000","last_unstale":"2024-03-04T12:24:05.955202+0000","last_undegraded":"2024-03-04T12:24:05.955202+0000","last_fullsized":"2024-03-04T12:24:05.955202+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:45:23.663627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.952474+0000","last_change":"2024-03-04T12:23:16.953341+0000","last_active":"2024-03-04T12:24:05.952474+0000","last_peered":"2024-03-04T12:24:05.952474+0000","last_clean":"2024-03-04T12:24:05.952474+0000","last_became_active":"2024-03-04T12:23:16.952878+0000","last_became_peered":"2024-03-04T12:23:16.952878+0000","last_unstale":"2024-03-04T12:24:05.952474+0000","last_undegraded":"2024-03-04T12:24:05.952474+0000","last_fullsized":"2024-03-04T12:24:05.952474+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:08:55.575263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.957748+0000","last_change":"2024-03-04T12:23:11.903182+0000","last_active":"2024-03-04T12:24:05.957748+0000","last_peered":"2024-03-04T12:24:05.957748+0000","last_clean":"2024-03-04T12:24:05.957748+0000","last_became_active":"2024-03-04T12:23:11.902709+0000","last_became_peered":"2024-03-04T12:23:11.902709+0000","last_unstale":"2024-03-04T12:24:05.957748+0000","last_undegraded":"2024-03-04T12:24:05.957748+0000","last_fullsized":"2024-03-04T12:24:05.957748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:13:33.438929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955801+0000","last_change":"2024-03-04T12:23:12.931339+0000","last_active":"2024-03-04T12:24:05.955801+0000","last_peered":"2024-03-04T12:24:05.955801+0000","last_clean":"2024-03-04T12:24:05.955801+0000","last_became_active":"2024-03-04T12:23:12.930213+0000","last_became_peered":"2024-03-04T12:23:12.930213+0000","last_unstale":"2024-03-04T12:24:05.955801+0000","last_undegraded":"2024-03-04T12:24:05.955801+0000","last_fullsized":"2024-03-04T12:24:05.955801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:55:23.084139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028104+0000","last_change":"2024-03-04T12:23:16.948723+0000","last_active":"2024-03-04T12:24:06.028104+0000","last_peered":"2024-03-04T12:24:06.028104+0000","last_clean":"2024-03-04T12:24:06.028104+0000","last_became_active":"2024-03-04T12:23:16.948256+0000","last_became_peered":"2024-03-04T12:23:16.948256+0000","last_unstale":"2024-03-04T12:24:06.028104+0000","last_undegraded":"2024-03-04T12:24:06.028104+0000","last_fullsized":"2024-03-04T12:24:06.028104+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:22.060517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:51.011591+0000","last_change":"2024-03-04T12:23:11.898475+0000","last_active":"2024-03-04T12:24:51.011591+0000","last_peered":"2024-03-04T12:24:51.011591+0000","last_clean":"2024-03-04T12:24:51.011591+0000","last_became_active":"2024-03-04T12:23:11.898117+0000","last_became_peered":"2024-03-04T12:23:11.898117+0000","last_unstale":"2024-03-04T12:24:51.011591+0000","last_undegraded":"2024-03-04T12:24:51.011591+0000","last_fullsized":"2024-03-04T12:24:51.011591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:21:25.119591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955735+0000","last_change":"2024-03-04T12:23:12.929772+0000","last_active":"2024-03-04T12:24:05.955735+0000","last_peered":"2024-03-04T12:24:05.955735+0000","last_clean":"2024-03-04T12:24:05.955735+0000","last_became_active":"2024-03-04T12:23:12.929405+0000","last_became_peered":"2024-03-04T12:23:12.929405+0000","last_unstale":"2024-03-04T12:24:05.955735+0000","last_undegraded":"2024-03-04T12:24:05.955735+0000","last_fullsized":"2024-03-04T12:24:05.955735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:22:42.874989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954869+0000","last_change":"2024-03-04T12:23:16.941715+0000","last_active":"2024-03-04T12:24:05.954869+0000","last_peered":"2024-03-04T12:24:05.954869+0000","last_clean":"2024-03-04T12:24:05.954869+0000","last_became_active":"2024-03-04T12:23:16.941298+0000","last_became_peered":"2024-03-04T12:23:16.941298+0000","last_unstale":"2024-03-04T12:24:05.954869+0000","last_undegraded":"2024-03-04T12:24:05.954869+0000","last_fullsized":"2024-03-04T12:24:05.954869+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:09:02.019531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.029116+0000","last_change":"2024-03-04T12:23:11.896413+0000","last_active":"2024-03-04T12:24:06.029116+0000","last_peered":"2024-03-04T12:24:06.029116+0000","last_clean":"2024-03-04T12:24:06.029116+0000","last_became_active":"2024-03-04T12:23:11.895679+0000","last_became_peered":"2024-03-04T12:23:11.895679+0000","last_unstale":"2024-03-04T12:24:06.029116+0000","last_undegraded":"2024-03-04T12:24:06.029116+0000","last_fullsized":"2024-03-04T12:24:06.029116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:25:41.967530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955720+0000","last_change":"2024-03-04T12:23:12.920468+0000","last_active":"2024-03-04T12:24:05.955720+0000","last_peered":"2024-03-04T12:24:05.955720+0000","last_clean":"2024-03-04T12:24:05.955720+0000","last_became_active":"2024-03-04T12:23:12.920113+0000","last_became_peered":"2024-03-04T12:23:12.920113+0000","last_unstale":"2024-03-04T12:24:05.955720+0000","last_undegraded":"2024-03-04T12:24:05.955720+0000","last_fullsized":"2024-03-04T12:24:05.955720+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:00:02.283518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.031522+0000","last_change":"2024-03-04T12:24:06.030444+0000","last_active":"2024-03-04T12:24:06.031522+0000","last_peered":"2024-03-04T12:24:06.031522+0000","last_clean":"2024-03-04T12:24:06.031522+0000","last_became_active":"2024-03-04T12:24:06.029934+0000","last_became_peered":"2024-03-04T12:24:06.029934+0000","last_unstale":"2024-03-04T12:24:06.031522+0000","last_undegraded":"2024-03-04T12:24:06.031522+0000","last_fullsized":"2024-03-04T12:24:06.031522+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:08:16.820509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028376+0000","last_change":"2024-03-04T12:23:11.903323+0000","last_active":"2024-03-04T12:24:06.028376+0000","last_peered":"2024-03-04T12:24:06.028376+0000","last_clean":"2024-03-04T12:24:06.028376+0000","last_became_active":"2024-03-04T12:23:11.902987+0000","last_became_peered":"2024-03-04T12:23:11.902987+0000","last_unstale":"2024-03-04T12:24:06.028376+0000","last_undegraded":"2024-03-04T12:24:06.028376+0000","last_fullsized":"2024-03-04T12:24:06.028376+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:14.044391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.956314+0000","last_change":"2024-03-04T12:23:12.913491+0000","last_active":"2024-03-04T12:24:05.956314+0000","last_peered":"2024-03-04T12:24:05.956314+0000","last_clean":"2024-03-04T12:24:05.956314+0000","last_became_active":"2024-03-04T12:23:12.912103+0000","last_became_peered":"2024-03-04T12:23:12.912103+0000","last_unstale":"2024-03-04T12:24:05.956314+0000","last_undegraded":"2024-03-04T12:24:05.956314+0000","last_fullsized":"2024-03-04T12:24:05.956314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:05:38.033596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.954760+0000","last_change":"2024-03-04T12:23:12.908437+0000","last_active":"2024-03-04T12:24:05.954760+0000","last_peered":"2024-03-04T12:24:05.954760+0000","last_clean":"2024-03-04T12:24:05.954760+0000","last_became_active":"2024-03-04T12:23:12.908071+0000","last_became_peered":"2024-03-04T12:23:12.908071+0000","last_unstale":"2024-03-04T12:24:05.954760+0000","last_undegraded":"2024-03-04T12:24:05.954760+0000","last_fullsized":"2024-03-04T12:24:05.954760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:20.112645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953550+0000","last_change":"2024-03-04T12:23:12.921645+0000","last_active":"2024-03-04T12:24:05.953550+0000","last_peered":"2024-03-04T12:24:05.953550+0000","last_clean":"2024-03-04T12:24:05.953550+0000","last_became_active":"2024-03-04T12:23:12.921275+0000","last_became_peered":"2024-03-04T12:23:12.921275+0000","last_unstale":"2024-03-04T12:24:05.953550+0000","last_undegraded":"2024-03-04T12:24:05.953550+0000","last_fullsized":"2024-03-04T12:24:05.953550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:56:17.740782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953081+0000","last_change":"2024-03-04T12:23:12.919813+0000","last_active":"2024-03-04T12:24:05.953081+0000","last_peered":"2024-03-04T12:24:05.953081+0000","last_clean":"2024-03-04T12:24:05.953081+0000","last_became_active":"2024-03-04T12:23:12.919469+0000","last_became_peered":"2024-03-04T12:23:12.919469+0000","last_unstale":"2024-03-04T12:24:05.953081+0000","last_undegraded":"2024-03-04T12:24:05.953081+0000","last_fullsized":"2024-03-04T12:24:05.953081+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:46.720268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955109+0000","last_change":"2024-03-04T12:23:12.918476+0000","last_active":"2024-03-04T12:24:05.955109+0000","last_peered":"2024-03-04T12:24:05.955109+0000","last_clean":"2024-03-04T12:24:05.955109+0000","last_became_active":"2024-03-04T12:23:12.917743+0000","last_became_peered":"2024-03-04T12:23:12.917743+0000","last_unstale":"2024-03-04T12:24:05.955109+0000","last_undegraded":"2024-03-04T12:24:05.955109+0000","last_fullsized":"2024-03-04T12:24:05.955109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:20.246769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028261+0000","last_change":"2024-03-04T12:23:12.918254+0000","last_active":"2024-03-04T12:24:06.028261+0000","last_peered":"2024-03-04T12:24:06.028261+0000","last_clean":"2024-03-04T12:24:06.028261+0000","last_became_active":"2024-03-04T12:23:12.917922+0000","last_became_peered":"2024-03-04T12:23:12.917922+0000","last_unstale":"2024-03-04T12:24:06.028261+0000","last_undegraded":"2024-03-04T12:24:06.028261+0000","last_fullsized":"2024-03-04T12:24:06.028261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:48:20.847984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955644+0000","last_change":"2024-03-04T12:23:12.910368+0000","last_active":"2024-03-04T12:24:05.955644+0000","last_peered":"2024-03-04T12:24:05.955644+0000","last_clean":"2024-03-04T12:24:05.955644+0000","last_became_active":"2024-03-04T12:23:12.910017+0000","last_became_peered":"2024-03-04T12:23:12.910017+0000","last_unstale":"2024-03-04T12:24:05.955644+0000","last_undegraded":"2024-03-04T12:24:05.955644+0000","last_fullsized":"2024-03-04T12:24:05.955644+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:15:22.093644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.953961+0000","last_change":"2024-03-04T12:23:12.921021+0000","last_active":"2024-03-04T12:24:05.953961+0000","last_peered":"2024-03-04T12:24:05.953961+0000","last_clean":"2024-03-04T12:24:05.953961+0000","last_became_active":"2024-03-04T12:23:12.920600+0000","last_became_peered":"2024-03-04T12:23:12.920600+0000","last_unstale":"2024-03-04T12:24:05.953961+0000","last_undegraded":"2024-03-04T12:24:05.953961+0000","last_fullsized":"2024-03-04T12:24:05.953961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:00:31.706900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.957607+0000","last_change":"2024-03-04T12:24:05.957607+0000","last_active":"2024-03-04T12:24:05.957607+0000","last_peered":"2024-03-04T12:24:05.957607+0000","last_clean":"2024-03-04T12:24:05.957607+0000","last_became_active":"2024-03-04T12:24:05.957335+0000","last_became_peered":"2024-03-04T12:24:05.957335+0000","last_unstale":"2024-03-04T12:24:05.957607+0000","last_undegraded":"2024-03-04T12:24:05.957607+0000","last_fullsized":"2024-03-04T12:24:05.957607+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:12.926219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028671+0000","last_change":"2024-03-04T12:23:12.911394+0000","last_active":"2024-03-04T12:24:06.028671+0000","last_peered":"2024-03-04T12:24:06.028671+0000","last_clean":"2024-03-04T12:24:06.028671+0000","last_became_active":"2024-03-04T12:23:12.911061+0000","last_became_peered":"2024-03-04T12:23:12.911061+0000","last_unstale":"2024-03-04T12:24:06.028671+0000","last_undegraded":"2024-03-04T12:24:06.028671+0000","last_fullsized":"2024-03-04T12:24:06.028671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:39.880383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955603+0000","last_change":"2024-03-04T12:23:12.911356+0000","last_active":"2024-03-04T12:24:05.955603+0000","last_peered":"2024-03-04T12:24:05.955603+0000","last_clean":"2024-03-04T12:24:05.955603+0000","last_became_active":"2024-03-04T12:23:12.911029+0000","last_became_peered":"2024-03-04T12:23:12.911029+0000","last_unstale":"2024-03-04T12:24:05.955603+0000","last_undegraded":"2024-03-04T12:24:05.955603+0000","last_fullsized":"2024-03-04T12:24:05.955603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:52.611625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028495+0000","last_change":"2024-03-04T12:23:12.924921+0000","last_active":"2024-03-04T12:24:06.028495+0000","last_peered":"2024-03-04T12:24:06.028495+0000","last_clean":"2024-03-04T12:24:06.028495+0000","last_became_active":"2024-03-04T12:23:12.924558+0000","last_became_peered":"2024-03-04T12:23:12.924558+0000","last_unstale":"2024-03-04T12:24:06.028495+0000","last_undegraded":"2024-03-04T12:24:06.028495+0000","last_fullsized":"2024-03-04T12:24:06.028495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:10:53.238440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:06.028167+0000","last_change":"2024-03-04T12:23:12.913106+0000","last_active":"2024-03-04T12:24:06.028167+0000","last_peered":"2024-03-04T12:24:06.028167+0000","last_clean":"2024-03-04T12:24:06.028167+0000","last_became_active":"2024-03-04T12:23:12.912741+0000","last_became_peered":"2024-03-04T12:23:12.912741+0000","last_unstale":"2024-03-04T12:24:06.028167+0000","last_undegraded":"2024-03-04T12:24:06.028167+0000","last_fullsized":"2024-03-04T12:24:06.028167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:46:21.406383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-04T12:24:05.955450+0000","last_change":"2024-03-04T12:23:12.918082+0000","last_active":"2024-03-04T12:24:05.955450+0000","last_peered":"2024-03-04T12:24:05.955450+0000","last_clean":"2024-03-04T12:24:05.955450+0000","last_became_active":"2024-03-04T12:23:12.917754+0000","last_became_peered":"2024-03-04T12:23:12.917754+0000","last_unstale":"2024-03-04T12:24:05.955450+0000","last_undegraded":"2024-03-04T12:24:05.955450+0000","last_fullsized":"2024-03-04T12:24:05.955450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:56.601779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286257,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67473,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":766,"num_read_kb":459,"num_write":239,"num_write_kb":136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":202419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383715,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":501441,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645352,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28148,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717004,"statfs":{"total":95995035648,"available":95966212096,"internally_reserved":0,"allocated":1150976,"data_stored":1148917,"data_compressed":57051,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808878,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1032192,"data_stored":505467,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070515,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1024000,"data_stored":501749,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":1098637,"data_compressed":57051,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495675,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":496875,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1069056,"data_stored":558174,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1028096,"data_stored":498331,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36408,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708744,"statfs":{"total":95995035648,"available":95957753856,"internally_reserved":0,"allocated":1089536,"data_stored":558785,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":496583,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":501340,"data_compressed":20655,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757314,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1085440,"data_stored":1093828,"data_compressed":57051,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":62396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T12:24:58.531 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-04T12:24:58.531 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-04T12:24:58.531 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-04T12:24:58.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph health --format=json 2024-03-04T12:24:58.556 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:58 smithi060 ceph-mon[32745]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:24:58.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:58 smithi086 ceph-mon[36851]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:24:58.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:58 smithi063 ceph-mon[36717]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:24:58.777 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:24:59.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:24:59 smithi086 ceph-mon[36851]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:24:59.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:24:59 smithi063 ceph-mon[36717]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:24:59.648 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:24:59 smithi060 ceph-mon[32745]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:25:00.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.611+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/3319411916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c106e90 msgr2=0x7fbf9c107270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:00.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.611+0000 7fbfa0dbe700 1 --2- 172.21.15.60:0/3319411916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c106e90 0x7fbf9c107270 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009990 tx=0x7fbf8c009ca0 comp rx=0 tx=0).stop 2024-03-04T12:25:00.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.611+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/3319411916 shutdown_connections 2024-03-04T12:25:00.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.611+0000 7fbfa0dbe700 1 --2- 172.21.15.60:0/3319411916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf9c100dd0 0x7fbf9c10ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:00.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.611+0000 7fbfa0dbe700 1 --2- 172.21.15.60:0/3319411916 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c106e90 0x7fbf9c107270 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:00.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.611+0000 7fbfa0dbe700 1 --2- 172.21.15.60:0/3319411916 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c100410 0x7fbf9c100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:00.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.611+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/3319411916 >> 172.21.15.60:0/3319411916 conn(0x7fbf9c077320 msgr2=0x7fbf9c0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:00.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.611+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/3319411916 shutdown_connections 2024-03-04T12:25:00.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.611+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/3319411916 wait complete. 2024-03-04T12:25:00.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbfa0dbe700 1 Processor -- start 2024-03-04T12:25:00.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbfa0dbe700 1 -- start start 2024-03-04T12:25:00.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbfa0dbe700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c100410 0x7fbf9c1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:00.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbfa0dbe700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf9c100dd0 0x7fbf9c198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:00.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbfa0dbe700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c106e90 0x7fbf9c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:00.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbfa0dbe700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c112270 con 0x7fbf9c100410 2024-03-04T12:25:00.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbfa0dbe700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1120f0 con 0x7fbf9c106e90 2024-03-04T12:25:00.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbfa0dbe700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1123f0 con 0x7fbf9c100dd0 2024-03-04T12:25:00.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbf9a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c100410 0x7fbf9c1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:00.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbf9a59c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c100410 0x7fbf9c1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46532/0 (socket says 172.21.15.60:46532) 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbf9a59c700 1 -- 172.21.15.60:0/2046836982 learned_addr learned my addr 172.21.15.60:0/2046836982 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbf99d9b700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf9c100dd0 0x7fbf9c198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.612+0000 7fbf9ad9d700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c106e90 0x7fbf9c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.613+0000 7fbf9a59c700 1 -- 172.21.15.60:0/2046836982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf9c100dd0 msgr2=0x7fbf9c198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.613+0000 7fbf9a59c700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf9c100dd0 0x7fbf9c198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.613+0000 7fbf9a59c700 1 -- 172.21.15.60:0/2046836982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c106e90 msgr2=0x7fbf9c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.613+0000 7fbf9a59c700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c106e90 0x7fbf9c19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.613+0000 7fbf9a59c700 1 -- 172.21.15.60:0/2046836982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c19d680 con 0x7fbf9c100410 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.613+0000 7fbf9a59c700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c100410 0x7fbf9c1986c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c000c00 tx=0x7fbf8c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.613+0000 7fbf8b7fe700 1 -- 172.21.15.60:0/2046836982 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c027070 con 0x7fbf9c100410 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.613+0000 7fbf8b7fe700 1 -- 172.21.15.60:0/2046836982 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf8c003780 con 0x7fbf9c100410 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.613+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/2046836982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c19d910 con 0x7fbf9c100410 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.613+0000 7fbf8b7fe700 1 -- 172.21.15.60:0/2046836982 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c022dd0 con 0x7fbf9c100410 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.613+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/2046836982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c1a51f0 con 0x7fbf9c100410 2024-03-04T12:25:00.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.614+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/2046836982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c10a460 con 0x7fbf9c100410 2024-03-04T12:25:00.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.615+0000 7fbf8b7fe700 1 -- 172.21.15.60:0/2046836982 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fbf8c02a070 con 0x7fbf9c100410 2024-03-04T12:25:00.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.616+0000 7fbf8b7fe700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbf840a1ea0 0x7fbf840a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:00.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.616+0000 7fbf99d9b700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbf840a1ea0 0x7fbf840a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:00.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.616+0000 7fbf8b7fe700 1 -- 172.21.15.60:0/2046836982 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fbf8c0cf370 con 0x7fbf9c100410 2024-03-04T12:25:00.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.616+0000 7fbf99d9b700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbf840a1ea0 0x7fbf840a4360 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c0ff980 tx=0x7fbf9000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:00.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.619+0000 7fbf8b7fe700 1 -- 172.21.15.60:0/2046836982 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbf8c013070 con 0x7fbf9c100410 2024-03-04T12:25:00.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.867+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/2046836982 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbf9c1999d0 con 0x7fbf9c100410 2024-03-04T12:25:00.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.867+0000 7fbf8b7fe700 1 -- 172.21.15.60:0/2046836982 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fbf8c09c3a0 con 0x7fbf9c100410 2024-03-04T12:25:00.874 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:00.874 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T12:25:00.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.869+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/2046836982 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbf840a1ea0 msgr2=0x7fbf840a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:00.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.869+0000 7fbfa0dbe700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbf840a1ea0 0x7fbf840a4360 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c0ff980 tx=0x7fbf9000b410 comp rx=0 tx=0).stop 2024-03-04T12:25:00.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.869+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/2046836982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c100410 msgr2=0x7fbf9c1986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:00.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.869+0000 7fbfa0dbe700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c100410 0x7fbf9c1986c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c000c00 tx=0x7fbf8c0047b0 comp rx=0 tx=0).stop 2024-03-04T12:25:00.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.869+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/2046836982 shutdown_connections 2024-03-04T12:25:00.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.869+0000 7fbfa0dbe700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c106e90 0x7fbf9c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:00.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.870+0000 7fbfa0dbe700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbf840a1ea0 0x7fbf840a4360 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:00.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.870+0000 7fbfa0dbe700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c100410 0x7fbf9c1986c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:00.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.870+0000 7fbfa0dbe700 1 --2- 172.21.15.60:0/2046836982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf9c100dd0 0x7fbf9c198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:00.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.870+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/2046836982 >> 172.21.15.60:0/2046836982 conn(0x7fbf9c077320 msgr2=0x7fbf9c104f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:00.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.870+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/2046836982 shutdown_connections 2024-03-04T12:25:00.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:00.870+0000 7fbfa0dbe700 1 -- 172.21.15.60:0/2046836982 wait complete. 2024-03-04T12:25:01.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:00 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2046836982' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T12:25:01.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:00 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2046836982' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T12:25:01.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:00 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2046836982' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T12:25:01.830 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-04T12:25:01.830 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:25:01.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 -- 172.21.15.60:0/2248556197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f841025b0 msgr2=0x7f0f84102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:01.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 --2- 172.21.15.60:0/2248556197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f841025b0 0x7f0f84102a30 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009900 tx=0x7f0f7002bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:01.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 -- 172.21.15.60:0/2248556197 shutdown_connections 2024-03-04T12:25:01.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 --2- 172.21.15.60:0/2248556197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f84102f70 0x7f0f84109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:01.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 --2- 172.21.15.60:0/2248556197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f841025b0 0x7f0f84102a30 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:01.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 --2- 172.21.15.60:0/2248556197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f841013c0 0x7f0f841017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:01.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 -- 172.21.15.60:0/2248556197 >> 172.21.15.60:0/2248556197 conn(0x7f0f840fc950 msgr2=0x7f0f840fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:01.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 -- 172.21.15.60:0/2248556197 shutdown_connections 2024-03-04T12:25:01.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 -- 172.21.15.60:0/2248556197 wait complete. 2024-03-04T12:25:01.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 Processor -- start 2024-03-04T12:25:01.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 -- start start 2024-03-04T12:25:01.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f841013c0 0x7f0f840750e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:01.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f841025b0 0x7f0f84073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:01.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f84102f70 0x7f0f84073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:01.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8410cfe0 con 0x7f0f841013c0 2024-03-04T12:25:01.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8410ce60 con 0x7f0f841025b0 2024-03-04T12:25:01.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.980+0000 7f0f8aba9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8410d160 con 0x7f0f84102f70 2024-03-04T12:25:01.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f88945640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f841013c0 0x7f0f840750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:01.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f88945640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f841013c0 0x7f0f840750e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46556/0 (socket says 172.21.15.60:46556) 2024-03-04T12:25:01.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f88945640 1 -- 172.21.15.60:0/898907355 learned_addr learned my addr 172.21.15.60:0/898907355 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:01.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f88945640 1 -- 172.21.15.60:0/898907355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f84102f70 msgr2=0x7f0f84073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:01.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f88945640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f84102f70 0x7f0f84073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:01.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f83fff640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f841025b0 0x7f0f84073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:01.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f88945640 1 -- 172.21.15.60:0/898907355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f841025b0 msgr2=0x7f0f84073720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:01.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f88945640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f841025b0 0x7f0f84073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:01.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f88945640 1 -- 172.21.15.60:0/898907355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f840744f0 con 0x7f0f841013c0 2024-03-04T12:25:01.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f88945640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f841013c0 0x7f0f840750e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800c970 tx=0x7f0f7800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:01.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f81ffb640 1 -- 172.21.15.60:0/898907355 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f78007c10 con 0x7f0f841013c0 2024-03-04T12:25:01.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f8aba9640 1 -- 172.21.15.60:0/898907355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f841a16c0 con 0x7f0f841013c0 2024-03-04T12:25:01.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f81ffb640 1 -- 172.21.15.60:0/898907355 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f78007db0 con 0x7f0f841013c0 2024-03-04T12:25:01.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f81ffb640 1 -- 172.21.15.60:0/898907355 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7800f940 con 0x7f0f841013c0 2024-03-04T12:25:01.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.981+0000 7f0f8aba9640 1 -- 172.21.15.60:0/898907355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f841a1c10 con 0x7f0f841013c0 2024-03-04T12:25:01.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.982+0000 7f0f8aba9640 1 -- 172.21.15.60:0/898907355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f84108610 con 0x7f0f841013c0 2024-03-04T12:25:01.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.983+0000 7f0f81ffb640 1 -- 172.21.15.60:0/898907355 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f0f7800fae0 con 0x7f0f841013c0 2024-03-04T12:25:01.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.984+0000 7f0f81ffb640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0f740a1e10 0x7f0f740a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:01.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.984+0000 7f0f83fff640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0f740a1e10 0x7f0f740a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:01.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.984+0000 7f0f81ffb640 1 -- 172.21.15.60:0/898907355 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f0f780ce760 con 0x7f0f841013c0 2024-03-04T12:25:01.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.985+0000 7f0f83fff640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0f740a1e10 0x7f0f740a42d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0f70036040 tx=0x7f0f70005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:01.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:01.989+0000 7f0f81ffb640 1 -- 172.21.15.60:0/898907355 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0f7809b760 con 0x7f0f841013c0 2024-03-04T12:25:02.176 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.170+0000 7f0f8aba9640 1 -- 172.21.15.60:0/898907355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f8402ceb0 con 0x7f0f841013c0 2024-03-04T12:25:02.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.172+0000 7f0f81ffb640 1 -- 172.21.15.60:0/898907355 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24468 (secure 0 0 0) 0x7f0f7809b060 con 0x7f0f841013c0 2024-03-04T12:25:02.178 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:02.178 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":91,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:24:05.942202+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:25:02.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:01 smithi060 ceph-mon[32745]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:25:02.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.174+0000 7f0f8aba9640 1 -- 172.21.15.60:0/898907355 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0f740a1e10 msgr2=0x7f0f740a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.174+0000 7f0f8aba9640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0f740a1e10 0x7f0f740a42d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0f70036040 tx=0x7f0f70005fd0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.174+0000 7f0f8aba9640 1 -- 172.21.15.60:0/898907355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f841013c0 msgr2=0x7f0f840750e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.174+0000 7f0f8aba9640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f841013c0 0x7f0f840750e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800c970 tx=0x7f0f7800ce30 comp rx=0 tx=0).stop 2024-03-04T12:25:02.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.174+0000 7f0f8aba9640 1 -- 172.21.15.60:0/898907355 shutdown_connections 2024-03-04T12:25:02.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.174+0000 7f0f8aba9640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0f740a1e10 0x7f0f740a42d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.174+0000 7f0f8aba9640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f84102f70 0x7f0f84073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.174+0000 7f0f8aba9640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f841025b0 0x7f0f84073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.174+0000 7f0f8aba9640 1 --2- 172.21.15.60:0/898907355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f841013c0 0x7f0f840750e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.174+0000 7f0f8aba9640 1 -- 172.21.15.60:0/898907355 >> 172.21.15.60:0/898907355 conn(0x7f0f840fc950 msgr2=0x7f0f841057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:02.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.174+0000 7f0f8aba9640 1 -- 172.21.15.60:0/898907355 shutdown_connections 2024-03-04T12:25:02.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.174+0000 7f0f8aba9640 1 -- 172.21.15.60:0/898907355 wait complete. 2024-03-04T12:25:02.192 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T12:23:10.438928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T12:23:11.472998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T12:23:15.162500+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T12:25:02.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:25:02.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.293+0000 7f86c996f640 1 -- 172.21.15.60:0/1339060909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c41070b0 msgr2=0x7f86c41094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.293+0000 7f86c996f640 1 --2- 172.21.15.60:0/1339060909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c41070b0 0x7f86c41094a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f86b4009ca0 tx=0x7f86b402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.293+0000 7f86c996f640 1 -- 172.21.15.60:0/1339060909 shutdown_connections 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.293+0000 7f86c996f640 1 --2- 172.21.15.60:0/1339060909 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c41070b0 0x7f86c41094a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.293+0000 7f86c996f640 1 --2- 172.21.15.60:0/1339060909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86c4104780 0x7f86c4106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.293+0000 7f86c996f640 1 --2- 172.21.15.60:0/1339060909 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86c4101e50 0x7f86c4104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.293+0000 7f86c996f640 1 -- 172.21.15.60:0/1339060909 >> 172.21.15.60:0/1339060909 conn(0x7f86c40fb7c0 msgr2=0x7f86c40fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.293+0000 7f86c996f640 1 -- 172.21.15.60:0/1339060909 shutdown_connections 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.293+0000 7f86c996f640 1 -- 172.21.15.60:0/1339060909 wait complete. 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.294+0000 7f86c996f640 1 Processor -- start 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.294+0000 7f86c996f640 1 -- start start 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.294+0000 7f86c996f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86c4101e50 0x7f86c4197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.294+0000 7f86c996f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86c4104780 0x7f86c4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.294+0000 7f86c996f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c41070b0 0x7f86c419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.294+0000 7f86c996f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c410cfe0 con 0x7f86c41070b0 2024-03-04T12:25:02.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.294+0000 7f86c996f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c410ce60 con 0x7f86c4101e50 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.294+0000 7f86c996f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c410d160 con 0x7f86c4104780 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.294+0000 7f86c37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c41070b0 0x7f86c419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.294+0000 7f86c37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c41070b0 0x7f86c419ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46572/0 (socket says 172.21.15.60:46572) 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.294+0000 7f86c37fe640 1 -- 172.21.15.60:0/59411963 learned_addr learned my addr 172.21.15.60:0/59411963 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.294+0000 7f86c27fc640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86c4104780 0x7f86c4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.295+0000 7f86c2ffd640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86c4101e50 0x7f86c4197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.295+0000 7f86c37fe640 1 -- 172.21.15.60:0/59411963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86c4104780 msgr2=0x7f86c4197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.295+0000 7f86c37fe640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86c4104780 0x7f86c4197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.295+0000 7f86c37fe640 1 -- 172.21.15.60:0/59411963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86c4101e50 msgr2=0x7f86c4197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.295+0000 7f86c37fe640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86c4101e50 0x7f86c4197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.295+0000 7f86c37fe640 1 -- 172.21.15.60:0/59411963 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c419f230 con 0x7f86c41070b0 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.295+0000 7f86c37fe640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c41070b0 0x7f86c419ebc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f86b402c700 tx=0x7f86b4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.295+0000 7f86c896d640 1 -- 172.21.15.60:0/59411963 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4039040 con 0x7f86c41070b0 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.295+0000 7f86c996f640 1 -- 172.21.15.60:0/59411963 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c419f4c0 con 0x7f86c41070b0 2024-03-04T12:25:02.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.295+0000 7f86c896d640 1 -- 172.21.15.60:0/59411963 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86b4044070 con 0x7f86c41070b0 2024-03-04T12:25:02.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.295+0000 7f86c896d640 1 -- 172.21.15.60:0/59411963 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b40042a0 con 0x7f86c41070b0 2024-03-04T12:25:02.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.295+0000 7f86c996f640 1 -- 172.21.15.60:0/59411963 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c419f9b0 con 0x7f86c41070b0 2024-03-04T12:25:02.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.296+0000 7f86c996f640 1 -- 172.21.15.60:0/59411963 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c40fd2b0 con 0x7f86c41070b0 2024-03-04T12:25:02.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.297+0000 7f86c896d640 1 -- 172.21.15.60:0/59411963 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f86b4003970 con 0x7f86c41070b0 2024-03-04T12:25:02.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.298+0000 7f86c896d640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f86b00a1e10 0x7f86b00a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:02.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.298+0000 7f86c2ffd640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f86b00a1e10 0x7f86b00a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:02.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.298+0000 7f86c896d640 1 -- 172.21.15.60:0/59411963 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f86b40f0d80 con 0x7f86c41070b0 2024-03-04T12:25:02.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.298+0000 7f86c2ffd640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f86b00a1e10 0x7f86b00a42d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f86bc005fd0 tx=0x7f86bc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:02.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.301+0000 7f86c896d640 1 -- 172.21.15.60:0/59411963 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f86b40bde30 con 0x7f86c41070b0 2024-03-04T12:25:02.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:01 smithi086 ceph-mon[36851]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:25:02.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:01 smithi063 ceph-mon[36717]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:25:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.482+0000 7f86c996f640 1 -- 172.21.15.60:0/59411963 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f86c402ceb0 con 0x7f86c41070b0 2024-03-04T12:25:02.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.482+0000 7f86c896d640 1 -- 172.21.15.60:0/59411963 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f86b40bd730 con 0x7f86c41070b0 2024-03-04T12:25:02.488 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:25:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.484+0000 7f86c996f640 1 -- 172.21.15.60:0/59411963 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f86b00a1e10 msgr2=0x7f86b00a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.484+0000 7f86c996f640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f86b00a1e10 0x7f86b00a42d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f86bc005fd0 tx=0x7f86bc005850 comp rx=0 tx=0).stop 2024-03-04T12:25:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.484+0000 7f86c996f640 1 -- 172.21.15.60:0/59411963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c41070b0 msgr2=0x7f86c419ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.484+0000 7f86c996f640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c41070b0 0x7f86c419ebc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f86b402c700 tx=0x7f86b4032f20 comp rx=0 tx=0).stop 2024-03-04T12:25:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.484+0000 7f86c996f640 1 -- 172.21.15.60:0/59411963 shutdown_connections 2024-03-04T12:25:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.484+0000 7f86c996f640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f86b00a1e10 0x7f86b00a42d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.484+0000 7f86c996f640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f86c41070b0 0x7f86c419ebc0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.484+0000 7f86c996f640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86c4104780 0x7f86c4197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.484+0000 7f86c996f640 1 --2- 172.21.15.60:0/59411963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86c4101e50 0x7f86c4197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.484+0000 7f86c996f640 1 -- 172.21.15.60:0/59411963 >> 172.21.15.60:0/59411963 conn(0x7f86c40fb7c0 msgr2=0x7f86c4102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.485+0000 7f86c996f640 1 -- 172.21.15.60:0/59411963 shutdown_connections 2024-03-04T12:25:02.491 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.485+0000 7f86c996f640 1 -- 172.21.15.60:0/59411963 wait complete. 2024-03-04T12:25:02.502 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T12:25:02.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.602+0000 7f5296fec640 1 -- 172.21.15.60:0/2193203401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5290101440 msgr2=0x7f52901018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.602+0000 7f5296fec640 1 --2- 172.21.15.60:0/2193203401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5290101440 0x7f52901018c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f527c009900 tx=0x7f527c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:02.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.602+0000 7f5296fec640 1 -- 172.21.15.60:0/2193203401 shutdown_connections 2024-03-04T12:25:02.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.602+0000 7f5296fec640 1 --2- 172.21.15.60:0/2193203401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5290101e00 0x7f5290108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.602+0000 7f5296fec640 1 --2- 172.21.15.60:0/2193203401 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5290101440 0x7f52901018c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.602+0000 7f5296fec640 1 --2- 172.21.15.60:0/2193203401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5290100250 0x7f5290100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.602+0000 7f5296fec640 1 -- 172.21.15.60:0/2193203401 >> 172.21.15.60:0/2193203401 conn(0x7f52900fb7c0 msgr2=0x7f52900fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:02.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.602+0000 7f5296fec640 1 -- 172.21.15.60:0/2193203401 shutdown_connections 2024-03-04T12:25:02.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.602+0000 7f5296fec640 1 -- 172.21.15.60:0/2193203401 wait complete. 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.602+0000 7f5296fec640 1 Processor -- start 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.602+0000 7f5296fec640 1 -- start start 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5296fec640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5290100250 0x7f52901976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5296fec640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5290101440 0x7f5290197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5296fec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5290101e00 0x7f529019ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5296fec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529010bee0 con 0x7f5290101e00 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5296fec640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529010bd60 con 0x7f5290100250 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5296fec640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529010c060 con 0x7f5290101440 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5295589640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5290101e00 0x7f529019ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5295589640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5290101e00 0x7f529019ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46582/0 (socket says 172.21.15.60:46582) 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5295589640 1 -- 172.21.15.60:0/2400388136 learned_addr learned my addr 172.21.15.60:0/2400388136 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5294d88640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5290100250 0x7f52901976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:02.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f528ffff640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5290101440 0x7f5290197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:02.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5295589640 1 -- 172.21.15.60:0/2400388136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5290101440 msgr2=0x7f5290197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5295589640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5290101440 0x7f5290197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5295589640 1 -- 172.21.15.60:0/2400388136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5290100250 msgr2=0x7f52901976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5295589640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5290100250 0x7f52901976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5295589640 1 -- 172.21.15.60:0/2400388136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529019f2e0 con 0x7f5290101e00 2024-03-04T12:25:02.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f528ffff640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5290101440 0x7f5290197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:02.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5295589640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5290101e00 0x7f529019ec70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5288009fd0 tx=0x7f528800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:02.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f528dffb640 1 -- 172.21.15.60:0/2400388136 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5288017070 con 0x7f5290101e00 2024-03-04T12:25:02.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f5296fec640 1 -- 172.21.15.60:0/2400388136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529019f5d0 con 0x7f5290101e00 2024-03-04T12:25:02.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f528dffb640 1 -- 172.21.15.60:0/2400388136 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5288003e50 con 0x7f5290101e00 2024-03-04T12:25:02.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.603+0000 7f528dffb640 1 -- 172.21.15.60:0/2400388136 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52880106c0 con 0x7f5290101e00 2024-03-04T12:25:02.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.604+0000 7f5296fec640 1 -- 172.21.15.60:0/2400388136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529019fb20 con 0x7f5290101e00 2024-03-04T12:25:02.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.605+0000 7f5296fec640 1 -- 172.21.15.60:0/2400388136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529019fd80 con 0x7f5290101e00 2024-03-04T12:25:02.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.606+0000 7f528dffb640 1 -- 172.21.15.60:0/2400388136 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f5288003970 con 0x7f5290101e00 2024-03-04T12:25:02.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.606+0000 7f528dffb640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f52800a1e10 0x7f52800a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:02.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.606+0000 7f5294d88640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f52800a1e10 0x7f52800a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:02.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.606+0000 7f528dffb640 1 -- 172.21.15.60:0/2400388136 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f52880ce070 con 0x7f5290101e00 2024-03-04T12:25:02.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.607+0000 7f5294d88640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f52800a1e10 0x7f52800a42d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f52901012a0 tx=0x7f528400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:02.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.610+0000 7f528dffb640 1 -- 172.21.15.60:0/2400388136 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f528809afe0 con 0x7f5290101e00 2024-03-04T12:25:02.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.791+0000 7f5296fec640 1 -- 172.21.15.60:0/2400388136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f52901a0110 con 0x7f5290101e00 2024-03-04T12:25:02.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.792+0000 7f528dffb640 1 -- 172.21.15.60:0/2400388136 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f528809a8e0 con 0x7f5290101e00 2024-03-04T12:25:02.799 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:02.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.794+0000 7f5296fec640 1 -- 172.21.15.60:0/2400388136 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f52800a1e10 msgr2=0x7f52800a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.794+0000 7f5296fec640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f52800a1e10 0x7f52800a42d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f52901012a0 tx=0x7f528400b390 comp rx=0 tx=0).stop 2024-03-04T12:25:02.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.794+0000 7f5296fec640 1 -- 172.21.15.60:0/2400388136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5290101e00 msgr2=0x7f529019ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.794+0000 7f5296fec640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5290101e00 0x7f529019ec70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5288009fd0 tx=0x7f528800b260 comp rx=0 tx=0).stop 2024-03-04T12:25:02.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.794+0000 7f5296fec640 1 -- 172.21.15.60:0/2400388136 shutdown_connections 2024-03-04T12:25:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.794+0000 7f5296fec640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f52800a1e10 0x7f52800a42d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.794+0000 7f5296fec640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5290101e00 0x7f529019ec70 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.794+0000 7f5296fec640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5290101440 0x7f5290197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.794+0000 7f5296fec640 1 --2- 172.21.15.60:0/2400388136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5290100250 0x7f52901976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.794+0000 7f5296fec640 1 -- 172.21.15.60:0/2400388136 >> 172.21.15.60:0/2400388136 conn(0x7f52900fb7c0 msgr2=0x7f52900fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.794+0000 7f5296fec640 1 -- 172.21.15.60:0/2400388136 shutdown_connections 2024-03-04T12:25:02.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.794+0000 7f5296fec640 1 -- 172.21.15.60:0/2400388136 wait complete. 2024-03-04T12:25:02.811 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T12:25:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.911+0000 7f710b73b640 1 -- 172.21.15.60:0/1625817778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7104101e00 msgr2=0x7f7104108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.911+0000 7f710b73b640 1 --2- 172.21.15.60:0/1625817778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7104101e00 0x7f7104108690 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009ca0 tx=0x7f70fc02c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.911+0000 7f710b73b640 1 -- 172.21.15.60:0/1625817778 shutdown_connections 2024-03-04T12:25:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.911+0000 7f710b73b640 1 --2- 172.21.15.60:0/1625817778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7104101e00 0x7f7104108690 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.911+0000 7f710b73b640 1 --2- 172.21.15.60:0/1625817778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7104101440 0x7f71041018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.911+0000 7f710b73b640 1 --2- 172.21.15.60:0/1625817778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7104100250 0x7f7104100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.911+0000 7f710b73b640 1 -- 172.21.15.60:0/1625817778 >> 172.21.15.60:0/1625817778 conn(0x7f71040fb7c0 msgr2=0x7f71040fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.911+0000 7f710b73b640 1 -- 172.21.15.60:0/1625817778 shutdown_connections 2024-03-04T12:25:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.911+0000 7f710b73b640 1 -- 172.21.15.60:0/1625817778 wait complete. 2024-03-04T12:25:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.911+0000 7f710b73b640 1 Processor -- start 2024-03-04T12:25:02.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.911+0000 7f710b73b640 1 -- start start 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f710b73b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7104100250 0x7f7104197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f710b73b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7104101440 0x7f7104197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f710b73b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7104101e00 0x7f710419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f710b73b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710410bee0 con 0x7f7104101440 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f710b73b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710410bd60 con 0x7f7104100250 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f710b73b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710410c060 con 0x7f7104101e00 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f7108cd6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7104101440 0x7f7104197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f7108cd6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7104101440 0x7f7104197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46598/0 (socket says 172.21.15.60:46598) 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f7108cd6640 1 -- 172.21.15.60:0/1378734271 learned_addr learned my addr 172.21.15.60:0/1378734271 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f71094d7640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7104100250 0x7f7104197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f7109cd8640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7104101e00 0x7f710419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f7108cd6640 1 -- 172.21.15.60:0/1378734271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7104101e00 msgr2=0x7f710419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f7108cd6640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7104101e00 0x7f710419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f7108cd6640 1 -- 172.21.15.60:0/1378734271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7104100250 msgr2=0x7f7104197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:02.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f7108cd6640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7104100250 0x7f7104197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:02.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f7108cd6640 1 -- 172.21.15.60:0/1378734271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710419f2b0 con 0x7f7104101440 2024-03-04T12:25:02.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f7109cd8640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7104101e00 0x7f710419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:02.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.912+0000 7f7108cd6640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7104101440 0x7f7104197bd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f70f800e9f0 tx=0x7f70f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:02.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.914+0000 7f70f67fc640 1 -- 172.21.15.60:0/1378734271 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f800cd80 con 0x7f7104101440 2024-03-04T12:25:02.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.914+0000 7f70f67fc640 1 -- 172.21.15.60:0/1378734271 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f70f8005500 con 0x7f7104101440 2024-03-04T12:25:02.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.914+0000 7f70f67fc640 1 -- 172.21.15.60:0/1378734271 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8010690 con 0x7f7104101440 2024-03-04T12:25:02.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.914+0000 7f710b73b640 1 -- 172.21.15.60:0/1378734271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710419f5a0 con 0x7f7104101440 2024-03-04T12:25:02.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.914+0000 7f710b73b640 1 -- 172.21.15.60:0/1378734271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710419faf0 con 0x7f7104101440 2024-03-04T12:25:02.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.915+0000 7f710b73b640 1 -- 172.21.15.60:0/1378734271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710419fdb0 con 0x7f7104101440 2024-03-04T12:25:02.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.916+0000 7f70f67fc640 1 -- 172.21.15.60:0/1378734271 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f70f80046c0 con 0x7f7104101440 2024-03-04T12:25:02.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.916+0000 7f70f67fc640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f70f00a1e90 0x7f70f00a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:02.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.917+0000 7f70f67fc640 1 -- 172.21.15.60:0/1378734271 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f70f80ce6e0 con 0x7f7104101440 2024-03-04T12:25:02.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.917+0000 7f71094d7640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f70f00a1e90 0x7f70f00a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:02.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.917+0000 7f71094d7640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f70f00a1e90 0x7f70f00a4350 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f71041012a0 tx=0x7f7100005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:02.927 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:02.921+0000 7f70f67fc640 1 -- 172.21.15.60:0/1378734271 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f70f809b650 con 0x7f7104101440 2024-03-04T12:25:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.103+0000 7f710b73b640 1 -- 172.21.15.60:0/1378734271 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f710419ff80 con 0x7f7104101440 2024-03-04T12:25:03.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.103+0000 7f70f67fc640 1 -- 172.21.15.60:0/1378734271 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f70f809af50 con 0x7f7104101440 2024-03-04T12:25:03.109 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-03-04T12:25:03.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.105+0000 7f710b73b640 1 -- 172.21.15.60:0/1378734271 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f70f00a1e90 msgr2=0x7f70f00a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.105+0000 7f710b73b640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f70f00a1e90 0x7f70f00a4350 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f71041012a0 tx=0x7f7100005f90 comp rx=0 tx=0).stop 2024-03-04T12:25:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.105+0000 7f710b73b640 1 -- 172.21.15.60:0/1378734271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7104101440 msgr2=0x7f7104197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.105+0000 7f710b73b640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7104101440 0x7f7104197bd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f70f800e9f0 tx=0x7f70f800edf0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.105+0000 7f710b73b640 1 -- 172.21.15.60:0/1378734271 shutdown_connections 2024-03-04T12:25:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.105+0000 7f710b73b640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f70f00a1e90 0x7f70f00a4350 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.105+0000 7f710b73b640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7104101e00 0x7f710419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.105+0000 7f710b73b640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7104101440 0x7f7104197bd0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.105+0000 7f710b73b640 1 --2- 172.21.15.60:0/1378734271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7104100250 0x7f7104197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.105+0000 7f710b73b640 1 -- 172.21.15.60:0/1378734271 >> 172.21.15.60:0/1378734271 conn(0x7f71040fb7c0 msgr2=0x7f71040fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.105+0000 7f710b73b640 1 -- 172.21.15.60:0/1378734271 shutdown_connections 2024-03-04T12:25:03.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.105+0000 7f710b73b640 1 -- 172.21.15.60:0/1378734271 wait complete. 2024-03-04T12:25:03.122 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:02 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/898907355' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:03.122 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:02 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/59411963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:03.122 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:02 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2400388136' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:03.123 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T12:25:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.223+0000 7fbea0bfa640 1 -- 172.21.15.60:0/3120460684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c101340 msgr2=0x7fbe9c101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.223+0000 7fbea0bfa640 1 --2- 172.21.15.60:0/3120460684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c101340 0x7fbe9c101760 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009900 tx=0x7fbe8c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.223+0000 7fbea0bfa640 1 -- 172.21.15.60:0/3120460684 shutdown_connections 2024-03-04T12:25:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.223+0000 7fbea0bfa640 1 --2- 172.21.15.60:0/3120460684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe9c102ef0 0x7fbe9c109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.223+0000 7fbea0bfa640 1 --2- 172.21.15.60:0/3120460684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe9c102530 0x7fbe9c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.223+0000 7fbea0bfa640 1 --2- 172.21.15.60:0/3120460684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c101340 0x7fbe9c101760 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.223+0000 7fbea0bfa640 1 -- 172.21.15.60:0/3120460684 >> 172.21.15.60:0/3120460684 conn(0x7fbe9c0fc8b0 msgr2=0x7fbe9c0fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:03.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.223+0000 7fbea0bfa640 1 -- 172.21.15.60:0/3120460684 shutdown_connections 2024-03-04T12:25:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.223+0000 7fbea0bfa640 1 -- 172.21.15.60:0/3120460684 wait complete. 2024-03-04T12:25:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.223+0000 7fbea0bfa640 1 Processor -- start 2024-03-04T12:25:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbea0bfa640 1 -- start start 2024-03-04T12:25:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbea0bfa640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c101340 0x7fbe9c1976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbea0bfa640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe9c102530 0x7fbe9c197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbea0bfa640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe9c102ef0 0x7fbe9c19ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbea0bfa640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c10cfd0 con 0x7fbe9c101340 2024-03-04T12:25:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbea0bfa640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c10ce50 con 0x7fbe9c102530 2024-03-04T12:25:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbea0bfa640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c10d150 con 0x7fbe9c102ef0 2024-03-04T12:25:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe9a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c101340 0x7fbe9c1976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:03.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe9a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c101340 0x7fbe9c1976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46612/0 (socket says 172.21.15.60:46612) 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe9a59c640 1 -- 172.21.15.60:0/2141302832 learned_addr learned my addr 172.21.15.60:0/2141302832 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe9ad9d640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe9c102ef0 0x7fbe9c19ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe93fff640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe9c102530 0x7fbe9c197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe9a59c640 1 -- 172.21.15.60:0/2141302832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe9c102ef0 msgr2=0x7fbe9c19ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe9a59c640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe9c102ef0 0x7fbe9c19ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe9a59c640 1 -- 172.21.15.60:0/2141302832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe9c102530 msgr2=0x7fbe9c197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe9a59c640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe9c102530 0x7fbe9c197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe9a59c640 1 -- 172.21.15.60:0/2141302832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c19f2d0 con 0x7fbe9c101340 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe9ad9d640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe9c102ef0 0x7fbe9c19ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe9a59c640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c101340 0x7fbe9c1976b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c005ab0 tx=0x7fbe8c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe937fe640 1 -- 172.21.15.60:0/2141302832 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c041070 con 0x7fbe9c101340 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe937fe640 1 -- 172.21.15.60:0/2141302832 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe8c0054e0 con 0x7fbe9c101340 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbea0bfa640 1 -- 172.21.15.60:0/2141302832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c19f560 con 0x7fbe9c101340 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.224+0000 7fbe937fe640 1 -- 172.21.15.60:0/2141302832 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c034440 con 0x7fbe9c101340 2024-03-04T12:25:03.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.225+0000 7fbea0bfa640 1 -- 172.21.15.60:0/2141302832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c19fa50 con 0x7fbe9c101340 2024-03-04T12:25:03.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.225+0000 7fbea0bfa640 1 -- 172.21.15.60:0/2141302832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c108730 con 0x7fbe9c101340 2024-03-04T12:25:03.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.227+0000 7fbe937fe640 1 -- 172.21.15.60:0/2141302832 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fbe8c044070 con 0x7fbe9c101340 2024-03-04T12:25:03.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.227+0000 7fbe937fe640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbe880a1e10 0x7fbe880a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:03.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.227+0000 7fbe937fe640 1 -- 172.21.15.60:0/2141302832 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fbe8c0efe60 con 0x7fbe9c101340 2024-03-04T12:25:03.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.227+0000 7fbe93fff640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbe880a1e10 0x7fbe880a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:03.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.228+0000 7fbe93fff640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbe880a1e10 0x7fbe880a42d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbe84005fd0 tx=0x7fbe84005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:03.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.231+0000 7fbe937fe640 1 -- 172.21.15.60:0/2141302832 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbe8c0bcee0 con 0x7fbe9c101340 2024-03-04T12:25:03.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:02 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/898907355' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:03.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:02 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/59411963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:03.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:02 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2400388136' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:03.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:02 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/898907355' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:03.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:02 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/59411963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:03.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:02 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2400388136' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:03.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.415+0000 7fbea0bfa640 1 -- 172.21.15.60:0/2141302832 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbe9c02d0d0 con 0x7fbe9c101340 2024-03-04T12:25:03.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.416+0000 7fbe937fe640 1 -- 172.21.15.60:0/2141302832 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbe8c0bc7e0 con 0x7fbe9c101340 2024-03-04T12:25:03.422 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:03.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.417+0000 7fbea0bfa640 1 -- 172.21.15.60:0/2141302832 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbe880a1e10 msgr2=0x7fbe880a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.417+0000 7fbea0bfa640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbe880a1e10 0x7fbe880a42d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbe84005fd0 tx=0x7fbe84005850 comp rx=0 tx=0).stop 2024-03-04T12:25:03.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.417+0000 7fbea0bfa640 1 -- 172.21.15.60:0/2141302832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c101340 msgr2=0x7fbe9c1976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.417+0000 7fbea0bfa640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c101340 0x7fbe9c1976b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c005ab0 tx=0x7fbe8c005dc0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.418+0000 7fbea0bfa640 1 -- 172.21.15.60:0/2141302832 shutdown_connections 2024-03-04T12:25:03.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.418+0000 7fbea0bfa640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fbe880a1e10 0x7fbe880a42d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.418+0000 7fbea0bfa640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe9c102ef0 0x7fbe9c19ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.418+0000 7fbea0bfa640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe9c102530 0x7fbe9c197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.418+0000 7fbea0bfa640 1 --2- 172.21.15.60:0/2141302832 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe9c101340 0x7fbe9c1976b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.418+0000 7fbea0bfa640 1 -- 172.21.15.60:0/2141302832 >> 172.21.15.60:0/2141302832 conn(0x7fbe9c0fc8b0 msgr2=0x7fbe9c105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:03.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.418+0000 7fbea0bfa640 1 -- 172.21.15.60:0/2141302832 shutdown_connections 2024-03-04T12:25:03.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.418+0000 7fbea0bfa640 1 -- 172.21.15.60:0/2141302832 wait complete. 2024-03-04T12:25:03.434 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:25:03.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.531+0000 7f95c4ef7640 1 -- 172.21.15.60:0/819358449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95c01025b0 msgr2=0x7f95c0102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.531+0000 7f95c4ef7640 1 --2- 172.21.15.60:0/819358449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95c01025b0 0x7f95c0102a30 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009900 tx=0x7f95a802bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.531+0000 7f95c4ef7640 1 -- 172.21.15.60:0/819358449 shutdown_connections 2024-03-04T12:25:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.531+0000 7f95c4ef7640 1 --2- 172.21.15.60:0/819358449 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c0102f70 0x7f95c0109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.531+0000 7f95c4ef7640 1 --2- 172.21.15.60:0/819358449 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95c01025b0 0x7f95c0102a30 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.531+0000 7f95c4ef7640 1 --2- 172.21.15.60:0/819358449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c01013c0 0x7f95c01017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.531+0000 7f95c4ef7640 1 -- 172.21.15.60:0/819358449 >> 172.21.15.60:0/819358449 conn(0x7f95c00fc950 msgr2=0x7f95c00fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.531+0000 7f95c4ef7640 1 -- 172.21.15.60:0/819358449 shutdown_connections 2024-03-04T12:25:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.531+0000 7f95c4ef7640 1 -- 172.21.15.60:0/819358449 wait complete. 2024-03-04T12:25:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95c4ef7640 1 Processor -- start 2024-03-04T12:25:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95c4ef7640 1 -- start start 2024-03-04T12:25:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95c4ef7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95c01013c0 0x7f95c01975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95c4ef7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c01025b0 0x7f95c0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:03.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95c4ef7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c0102f70 0x7f95c019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95c4ef7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c010cfe0 con 0x7f95c01013c0 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95c4ef7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c010ce60 con 0x7f95c01025b0 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95c4ef7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c010d160 con 0x7f95c0102f70 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95be59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95c01013c0 0x7f95c01975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95be59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95c01013c0 0x7f95c01975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46618/0 (socket says 172.21.15.60:46618) 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95be59c640 1 -- 172.21.15.60:0/273688905 learned_addr learned my addr 172.21.15.60:0/273688905 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95b7fff640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c01025b0 0x7f95c0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.532+0000 7f95bed9d640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c0102f70 0x7f95c019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.533+0000 7f95be59c640 1 -- 172.21.15.60:0/273688905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c0102f70 msgr2=0x7f95c019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.533+0000 7f95be59c640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c0102f70 0x7f95c019eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.533+0000 7f95be59c640 1 -- 172.21.15.60:0/273688905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c01025b0 msgr2=0x7f95c0197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.533+0000 7f95be59c640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c01025b0 0x7f95c0197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.533+0000 7f95be59c640 1 -- 172.21.15.60:0/273688905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c019f1d0 con 0x7f95c01013c0 2024-03-04T12:25:03.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.533+0000 7f95be59c640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95c01013c0 0x7f95c01975b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f95b000c970 tx=0x7f95b000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:03.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.533+0000 7f95b77fe640 1 -- 172.21.15.60:0/273688905 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b0007c10 con 0x7f95c01013c0 2024-03-04T12:25:03.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.533+0000 7f95b77fe640 1 -- 172.21.15.60:0/273688905 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95b0007db0 con 0x7f95c01013c0 2024-03-04T12:25:03.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.533+0000 7f95c4ef7640 1 -- 172.21.15.60:0/273688905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c019f4c0 con 0x7f95c01013c0 2024-03-04T12:25:03.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.533+0000 7f95b77fe640 1 -- 172.21.15.60:0/273688905 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b000f940 con 0x7f95c01013c0 2024-03-04T12:25:03.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.533+0000 7f95c4ef7640 1 -- 172.21.15.60:0/273688905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c019fa30 con 0x7f95c01013c0 2024-03-04T12:25:03.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.534+0000 7f95c4ef7640 1 -- 172.21.15.60:0/273688905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c0108680 con 0x7f95c01013c0 2024-03-04T12:25:03.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.535+0000 7f95b77fe640 1 -- 172.21.15.60:0/273688905 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f95b000fae0 con 0x7f95c01013c0 2024-03-04T12:25:03.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.536+0000 7f95b77fe640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f95ac0a1e90 0x7f95ac0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:03.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.536+0000 7f95b7fff640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f95ac0a1e90 0x7f95ac0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:03.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.536+0000 7f95b77fe640 1 -- 172.21.15.60:0/273688905 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f95b00ce7d0 con 0x7f95c01013c0 2024-03-04T12:25:03.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.536+0000 7f95b7fff640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f95ac0a1e90 0x7f95ac0a4350 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f95a802c7b0 tx=0x7f95a8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:03.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.539+0000 7f95b77fe640 1 -- 172.21.15.60:0/273688905 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f95b009b860 con 0x7f95c01013c0 2024-03-04T12:25:03.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.758+0000 7f95c4ef7640 1 -- 172.21.15.60:0/273688905 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95c002ceb0 con 0x7f95c01013c0 2024-03-04T12:25:03.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.759+0000 7f95b77fe640 1 -- 172.21.15.60:0/273688905 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11373 (secure 0 0 0) 0x7f95b009b160 con 0x7f95c01013c0 2024-03-04T12:25:03.765 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:03.766 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:23:41.896895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:25:03.766 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 28 2024-03-04T12:25:03.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.761+0000 7f95c4ef7640 1 -- 172.21.15.60:0/273688905 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f95ac0a1e90 msgr2=0x7f95ac0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.761+0000 7f95c4ef7640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f95ac0a1e90 0x7f95ac0a4350 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f95a802c7b0 tx=0x7f95a8005fd0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.761+0000 7f95c4ef7640 1 -- 172.21.15.60:0/273688905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95c01013c0 msgr2=0x7f95c01975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.761+0000 7f95c4ef7640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95c01013c0 0x7f95c01975b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f95b000c970 tx=0x7f95b000ce30 comp rx=0 tx=0).stop 2024-03-04T12:25:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.761+0000 7f95c4ef7640 1 -- 172.21.15.60:0/273688905 shutdown_connections 2024-03-04T12:25:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.761+0000 7f95c4ef7640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f95ac0a1e90 0x7f95ac0a4350 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.761+0000 7f95c4ef7640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c0102f70 0x7f95c019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.761+0000 7f95c4ef7640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c01025b0 0x7f95c0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.761+0000 7f95c4ef7640 1 --2- 172.21.15.60:0/273688905 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95c01013c0 0x7f95c01975b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.761+0000 7f95c4ef7640 1 -- 172.21.15.60:0/273688905 >> 172.21.15.60:0/273688905 conn(0x7f95c00fc950 msgr2=0x7f95c0105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.761+0000 7f95c4ef7640 1 -- 172.21.15.60:0/273688905 shutdown_connections 2024-03-04T12:25:03.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.761+0000 7f95c4ef7640 1 -- 172.21.15.60:0/273688905 wait complete. 2024-03-04T12:25:03.779 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:25:03.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.875+0000 7f061355b640 1 -- 172.21.15.60:0/307547527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f060c1047b0 msgr2=0x7f060c106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.875+0000 7f061355b640 1 --2- 172.21.15.60:0/307547527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f060c1047b0 0x7f060c106ba0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0600009900 tx=0x7f060002bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:03.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.875+0000 7f061355b640 1 -- 172.21.15.60:0/307547527 shutdown_connections 2024-03-04T12:25:03.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.875+0000 7f061355b640 1 --2- 172.21.15.60:0/307547527 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f060c1070e0 0x7f060c1094d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.875+0000 7f061355b640 1 --2- 172.21.15.60:0/307547527 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f060c1047b0 0x7f060c106ba0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.875+0000 7f061355b640 1 --2- 172.21.15.60:0/307547527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f060c101e80 0x7f060c104270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.875+0000 7f061355b640 1 -- 172.21.15.60:0/307547527 >> 172.21.15.60:0/307547527 conn(0x7f060c0fb7d0 msgr2=0x7f060c0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.875+0000 7f061355b640 1 -- 172.21.15.60:0/307547527 shutdown_connections 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.875+0000 7f061355b640 1 -- 172.21.15.60:0/307547527 wait complete. 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.876+0000 7f061355b640 1 Processor -- start 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.876+0000 7f061355b640 1 -- start start 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.876+0000 7f061355b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f060c101e80 0x7f060c1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.876+0000 7f061355b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f060c1047b0 0x7f060c197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.876+0000 7f061355b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f060c1070e0 0x7f060c19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.876+0000 7f061355b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c10d010 con 0x7f060c1070e0 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.876+0000 7f061355b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c10ce90 con 0x7f060c101e80 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.876+0000 7f061355b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060c10d190 con 0x7f060c1047b0 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.876+0000 7f0611af8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f060c1070e0 0x7f060c19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.876+0000 7f0611af8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f060c1070e0 0x7f060c19ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46640/0 (socket says 172.21.15.60:46640) 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.876+0000 7f0611af8640 1 -- 172.21.15.60:0/3544735661 learned_addr learned my addr 172.21.15.60:0/3544735661 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:03.885 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.876+0000 7f06112f7640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f060c101e80 0x7f060c1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.877+0000 7f0611af8640 1 -- 172.21.15.60:0/3544735661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f060c1047b0 msgr2=0x7f060c197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.877+0000 7f0610af6640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f060c1047b0 0x7f060c197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.877+0000 7f0611af8640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f060c1047b0 0x7f060c197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.877+0000 7f0611af8640 1 -- 172.21.15.60:0/3544735661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f060c101e80 msgr2=0x7f060c1976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.877+0000 7f0611af8640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f060c101e80 0x7f060c1976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.877+0000 7f0611af8640 1 -- 172.21.15.60:0/3544735661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060c19f2c0 con 0x7f060c1070e0 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.877+0000 7f0611af8640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f060c1070e0 0x7f060c19ec50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00cad0 tx=0x7f05fc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.877+0000 7f060a7fc640 1 -- 172.21.15.60:0/3544735661 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc00ae20 con 0x7f060c1070e0 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.877+0000 7f061355b640 1 -- 172.21.15.60:0/3544735661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060c19f5b0 con 0x7f060c1070e0 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.877+0000 7f060a7fc640 1 -- 172.21.15.60:0/3544735661 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05fc005020 con 0x7f060c1070e0 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.877+0000 7f060a7fc640 1 -- 172.21.15.60:0/3544735661 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05fc00f6f0 con 0x7f060c1070e0 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.877+0000 7f061355b640 1 -- 172.21.15.60:0/3544735661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060c19fb00 con 0x7f060c1070e0 2024-03-04T12:25:03.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.878+0000 7f061355b640 1 -- 172.21.15.60:0/3544735661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060c0fd2c0 con 0x7f060c1070e0 2024-03-04T12:25:03.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.879+0000 7f060a7fc640 1 -- 172.21.15.60:0/3544735661 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f05fc0051d0 con 0x7f060c1070e0 2024-03-04T12:25:03.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.879+0000 7f060a7fc640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f05f40a1e10 0x7f05f40a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:03.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.880+0000 7f06112f7640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f05f40a1e10 0x7f05f40a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:03.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.880+0000 7f060a7fc640 1 -- 172.21.15.60:0/3544735661 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f05fc0ce2c0 con 0x7f060c1070e0 2024-03-04T12:25:03.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.880+0000 7f06112f7640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f05f40a1e10 0x7f05f40a42d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0604005fd0 tx=0x7f060400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:03.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:03.884+0000 7f060a7fc640 1 -- 172.21.15.60:0/3544735661 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f05fc014030 con 0x7f060c1070e0 2024-03-04T12:25:04.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.069+0000 7f061355b640 1 -- 172.21.15.60:0/3544735661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f060c061f30 con 0x7f060c1070e0 2024-03-04T12:25:04.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.070+0000 7f060a7fc640 1 -- 172.21.15.60:0/3544735661 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24468 (secure 0 0 0) 0x7f05fc09b230 con 0x7f060c1070e0 2024-03-04T12:25:04.077 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:04.077 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":91,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:24:05.942202+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:25:04.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.072+0000 7f061355b640 1 -- 172.21.15.60:0/3544735661 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f05f40a1e10 msgr2=0x7f05f40a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.072+0000 7f061355b640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f05f40a1e10 0x7f05f40a42d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0604005fd0 tx=0x7f060400b390 comp rx=0 tx=0).stop 2024-03-04T12:25:04.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.073+0000 7f061355b640 1 -- 172.21.15.60:0/3544735661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f060c1070e0 msgr2=0x7f060c19ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.073+0000 7f061355b640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f060c1070e0 0x7f060c19ec50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00cad0 tx=0x7f05fc00cf90 comp rx=0 tx=0).stop 2024-03-04T12:25:04.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.073+0000 7f061355b640 1 -- 172.21.15.60:0/3544735661 shutdown_connections 2024-03-04T12:25:04.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.073+0000 7f061355b640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f05f40a1e10 0x7f05f40a42d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.073+0000 7f061355b640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f060c1070e0 0x7f060c19ec50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.073+0000 7f061355b640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f060c1047b0 0x7f060c197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.079 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.073+0000 7f061355b640 1 --2- 172.21.15.60:0/3544735661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f060c101e80 0x7f060c1976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.073+0000 7f061355b640 1 -- 172.21.15.60:0/3544735661 >> 172.21.15.60:0/3544735661 conn(0x7f060c0fb7d0 msgr2=0x7f060c1028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:04.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.073+0000 7f061355b640 1 -- 172.21.15.60:0/3544735661 shutdown_connections 2024-03-04T12:25:04.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.073+0000 7f061355b640 1 -- 172.21.15.60:0/3544735661 wait complete. 2024-03-04T12:25:04.089 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:03 smithi060 ceph-mon[32745]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:25:04.089 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:03 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1378734271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:04.089 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:03 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2141302832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:04.089 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:03 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/273688905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:04.090 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T12:23:10.438928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T12:23:11.472998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T12:23:15.162500+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T12:25:04.091 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:25:04.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.214+0000 7f2f8897e640 1 -- 172.21.15.60:0/2964704198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f84100250 msgr2=0x7f2f84100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.214+0000 7f2f8897e640 1 --2- 172.21.15.60:0/2964704198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f84100250 0x7f2f84100670 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2f74009900 tx=0x7f2f7402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:04.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.214+0000 7f2f8897e640 1 -- 172.21.15.60:0/2964704198 shutdown_connections 2024-03-04T12:25:04.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.214+0000 7f2f8897e640 1 --2- 172.21.15.60:0/2964704198 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84101e00 0x7f2f84108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.214+0000 7f2f8897e640 1 --2- 172.21.15.60:0/2964704198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f84101440 0x7f2f841018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.214+0000 7f2f8897e640 1 --2- 172.21.15.60:0/2964704198 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f84100250 0x7f2f84100670 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.214+0000 7f2f8897e640 1 -- 172.21.15.60:0/2964704198 >> 172.21.15.60:0/2964704198 conn(0x7f2f840fb7c0 msgr2=0x7f2f840fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:04.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.214+0000 7f2f8897e640 1 -- 172.21.15.60:0/2964704198 shutdown_connections 2024-03-04T12:25:04.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.214+0000 7f2f8897e640 1 -- 172.21.15.60:0/2964704198 wait complete. 2024-03-04T12:25:04.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.215+0000 7f2f8897e640 1 Processor -- start 2024-03-04T12:25:04.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.215+0000 7f2f8897e640 1 -- start start 2024-03-04T12:25:04.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.215+0000 7f2f8897e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f84100250 0x7f2f841976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:04.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.215+0000 7f2f8897e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84101440 0x7f2f84197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:04.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.215+0000 7f2f8897e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f84101e00 0x7f2f8419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:04.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.215+0000 7f2f8897e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8410bee0 con 0x7f2f84101e00 2024-03-04T12:25:04.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.215+0000 7f2f8897e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8410bd60 con 0x7f2f84101440 2024-03-04T12:25:04.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.215+0000 7f2f8897e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8410c060 con 0x7f2f84100250 2024-03-04T12:25:04.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.215+0000 7f2f82d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f84101e00 0x7f2f8419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:04.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.215+0000 7f2f82d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f84101e00 0x7f2f8419ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46658/0 (socket says 172.21.15.60:46658) 2024-03-04T12:25:04.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.215+0000 7f2f82d9d640 1 -- 172.21.15.60:0/877628799 learned_addr learned my addr 172.21.15.60:0/877628799 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:04.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f79d9b640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84101440 0x7f2f84197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:04.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f8259c640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f84100250 0x7f2f841976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f82d9d640 1 -- 172.21.15.60:0/877628799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f84100250 msgr2=0x7f2f841976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f82d9d640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f84100250 0x7f2f841976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f82d9d640 1 -- 172.21.15.60:0/877628799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84101440 msgr2=0x7f2f84197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f82d9d640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84101440 0x7f2f84197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f82d9d640 1 -- 172.21.15.60:0/877628799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8419f2a0 con 0x7f2f84101e00 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f79d9b640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84101440 0x7f2f84197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f82d9d640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f84101e00 0x7f2f8419ec30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c00eb10 tx=0x7f2f7c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f7bfff640 1 -- 172.21.15.60:0/877628799 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c004b20 con 0x7f2f84101e00 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f7bfff640 1 -- 172.21.15.60:0/877628799 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f7c004cc0 con 0x7f2f84101e00 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f7bfff640 1 -- 172.21.15.60:0/877628799 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c010760 con 0x7f2f84101e00 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f8897e640 1 -- 172.21.15.60:0/877628799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8419f590 con 0x7f2f84101e00 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.216+0000 7f2f8897e640 1 -- 172.21.15.60:0/877628799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8419fa90 con 0x7f2f84101e00 2024-03-04T12:25:04.225 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.217+0000 7f2f8897e640 1 -- 172.21.15.60:0/877628799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8419fd50 con 0x7f2f84101e00 2024-03-04T12:25:04.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.218+0000 7f2f7bfff640 1 -- 172.21.15.60:0/877628799 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f2f7c005020 con 0x7f2f84101e00 2024-03-04T12:25:04.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.219+0000 7f2f7bfff640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2f700a1e10 0x7f2f700a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:04.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.219+0000 7f2f8259c640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2f700a1e10 0x7f2f700a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:04.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.219+0000 7f2f7bfff640 1 -- 172.21.15.60:0/877628799 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2f7c0ce980 con 0x7f2f84101e00 2024-03-04T12:25:04.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.219+0000 7f2f8259c640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2f700a1e10 0x7f2f700a42d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2f7402c7b0 tx=0x7f2f74005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:04.229 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.223+0000 7f2f7bfff640 1 -- 172.21.15.60:0/877628799 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2f7c09b890 con 0x7f2f84101e00 2024-03-04T12:25:04.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:03 smithi086 ceph-mon[36851]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:25:04.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:03 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1378734271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:04.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:03 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2141302832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:04.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:03 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/273688905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:04.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:03 smithi063 ceph-mon[36717]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:25:04.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:03 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1378734271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:04.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:03 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2141302832' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:04.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:03 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/273688905' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:04.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.403+0000 7f2f8897e640 1 -- 172.21.15.60:0/877628799 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2f8419ff20 con 0x7f2f84101e00 2024-03-04T12:25:04.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.404+0000 7f2f7bfff640 1 -- 172.21.15.60:0/877628799 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2f7c09b190 con 0x7f2f84101e00 2024-03-04T12:25:04.410 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:25:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.406+0000 7f2f8897e640 1 -- 172.21.15.60:0/877628799 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2f700a1e10 msgr2=0x7f2f700a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.406+0000 7f2f8897e640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2f700a1e10 0x7f2f700a42d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2f7402c7b0 tx=0x7f2f74005fd0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.406+0000 7f2f8897e640 1 -- 172.21.15.60:0/877628799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f84101e00 msgr2=0x7f2f8419ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.406+0000 7f2f8897e640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f84101e00 0x7f2f8419ec30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c00eb10 tx=0x7f2f7c00c2d0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.406+0000 7f2f8897e640 1 -- 172.21.15.60:0/877628799 shutdown_connections 2024-03-04T12:25:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.406+0000 7f2f8897e640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2f700a1e10 0x7f2f700a42d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.406+0000 7f2f8897e640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2f84101e00 0x7f2f8419ec30 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.412 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.406+0000 7f2f8897e640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84101440 0x7f2f84197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.406+0000 7f2f8897e640 1 --2- 172.21.15.60:0/877628799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f84100250 0x7f2f841976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.406+0000 7f2f8897e640 1 -- 172.21.15.60:0/877628799 >> 172.21.15.60:0/877628799 conn(0x7f2f840fb7c0 msgr2=0x7f2f84068600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:04.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.406+0000 7f2f8897e640 1 -- 172.21.15.60:0/877628799 shutdown_connections 2024-03-04T12:25:04.413 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.406+0000 7f2f8897e640 1 -- 172.21.15.60:0/877628799 wait complete. 2024-03-04T12:25:04.423 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T12:25:04.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3355678452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0104700 msgr2=0x7ff9e0106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 --2- 172.21.15.60:0/3355678452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0104700 0x7ff9e0106af0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0009900 tx=0x7ff9d002bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:04.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3355678452 shutdown_connections 2024-03-04T12:25:04.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 --2- 172.21.15.60:0/3355678452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0107030 0x7ff9e0109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 --2- 172.21.15.60:0/3355678452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0104700 0x7ff9e0106af0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 --2- 172.21.15.60:0/3355678452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e0101dd0 0x7ff9e01041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3355678452 >> 172.21.15.60:0/3355678452 conn(0x7ff9e00fb7c0 msgr2=0x7ff9e00fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:04.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3355678452 shutdown_connections 2024-03-04T12:25:04.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3355678452 wait complete. 2024-03-04T12:25:04.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 Processor -- start 2024-03-04T12:25:04.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 -- start start 2024-03-04T12:25:04.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0101dd0 0x7ff9e0197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:04.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0104700 0x7ff9e0197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:04.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e0107030 0x7ff9e019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:04.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e010cfe0 con 0x7ff9e0104700 2024-03-04T12:25:04.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e010ce60 con 0x7ff9e0101dd0 2024-03-04T12:25:04.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e78cd640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e010d160 con 0x7ff9e0107030 2024-03-04T12:25:04.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.523+0000 7ff9e4e68640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0104700 0x7ff9e0197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:04.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9e4e68640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0104700 0x7ff9e0197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46688/0 (socket says 172.21.15.60:46688) 2024-03-04T12:25:04.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9e4e68640 1 -- 172.21.15.60:0/3520959174 learned_addr learned my addr 172.21.15.60:0/3520959174 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:04.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9e5e6a640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e0107030 0x7ff9e019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:04.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9e5669640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0101dd0 0x7ff9e0197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:04.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9e4e68640 1 -- 172.21.15.60:0/3520959174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e0107030 msgr2=0x7ff9e019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9e4e68640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e0107030 0x7ff9e019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9e4e68640 1 -- 172.21.15.60:0/3520959174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0101dd0 msgr2=0x7ff9e0197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9e4e68640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0101dd0 0x7ff9e0197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9e4e68640 1 -- 172.21.15.60:0/3520959174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e019f2b0 con 0x7ff9e0104700 2024-03-04T12:25:04.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9e4e68640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0104700 0x7ff9e0197bd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0005ab0 tx=0x7ff9d0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:04.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9d67fc640 1 -- 172.21.15.60:0/3520959174 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0041070 con 0x7ff9e0104700 2024-03-04T12:25:04.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9d67fc640 1 -- 172.21.15.60:0/3520959174 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9d00054e0 con 0x7ff9e0104700 2024-03-04T12:25:04.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3520959174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e019f540 con 0x7ff9e0104700 2024-03-04T12:25:04.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9d67fc640 1 -- 172.21.15.60:0/3520959174 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0034440 con 0x7ff9e0104700 2024-03-04T12:25:04.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.524+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3520959174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e019fa30 con 0x7ff9e0104700 2024-03-04T12:25:04.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.525+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3520959174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e00fd2b0 con 0x7ff9e0104700 2024-03-04T12:25:04.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.526+0000 7ff9d67fc640 1 -- 172.21.15.60:0/3520959174 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7ff9d0044070 con 0x7ff9e0104700 2024-03-04T12:25:04.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.527+0000 7ff9d67fc640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff9cc0a1e10 0x7ff9cc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:04.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.527+0000 7ff9e5669640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff9cc0a1e10 0x7ff9cc0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:04.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.527+0000 7ff9d67fc640 1 -- 172.21.15.60:0/3520959174 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff9d00f0080 con 0x7ff9e0104700 2024-03-04T12:25:04.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.527+0000 7ff9e5669640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff9cc0a1e10 0x7ff9cc0a42d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0058c0 tx=0x7ff9dc005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:04.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.530+0000 7ff9d67fc640 1 -- 172.21.15.60:0/3520959174 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff9d00bd0a0 con 0x7ff9e0104700 2024-03-04T12:25:04.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.711+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3520959174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff9e002d040 con 0x7ff9e0104700 2024-03-04T12:25:04.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.711+0000 7ff9d67fc640 1 -- 172.21.15.60:0/3520959174 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff9d00bc9a0 con 0x7ff9e0104700 2024-03-04T12:25:04.718 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.713+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3520959174 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff9cc0a1e10 msgr2=0x7ff9cc0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.713+0000 7ff9e78cd640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff9cc0a1e10 0x7ff9cc0a42d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0058c0 tx=0x7ff9dc005c60 comp rx=0 tx=0).stop 2024-03-04T12:25:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.713+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3520959174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0104700 msgr2=0x7ff9e0197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.713+0000 7ff9e78cd640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0104700 0x7ff9e0197bd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0005ab0 tx=0x7ff9d0005dc0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.713+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3520959174 shutdown_connections 2024-03-04T12:25:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.713+0000 7ff9e78cd640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff9cc0a1e10 0x7ff9cc0a42d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.714+0000 7ff9e78cd640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e0107030 0x7ff9e019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.714+0000 7ff9e78cd640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9e0104700 0x7ff9e0197bd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.714+0000 7ff9e78cd640 1 --2- 172.21.15.60:0/3520959174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0101dd0 0x7ff9e0197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.714+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3520959174 >> 172.21.15.60:0/3520959174 conn(0x7ff9e00fb7c0 msgr2=0x7ff9e01027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:04.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.714+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3520959174 shutdown_connections 2024-03-04T12:25:04.721 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.714+0000 7ff9e78cd640 1 -- 172.21.15.60:0/3520959174 wait complete. 2024-03-04T12:25:04.731 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T12:25:04.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.829+0000 7f58f6281640 1 -- 172.21.15.60:0/4076905873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0101e70 msgr2=0x7f58f0104260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.829+0000 7f58f6281640 1 --2- 172.21.15.60:0/4076905873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0101e70 0x7f58f0104260 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f58e0009900 tx=0x7f58e002bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:04.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.829+0000 7f58f6281640 1 -- 172.21.15.60:0/4076905873 shutdown_connections 2024-03-04T12:25:04.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.829+0000 7f58f6281640 1 --2- 172.21.15.60:0/4076905873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f01070d0 0x7f58f01094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 --2- 172.21.15.60:0/4076905873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58f01047a0 0x7f58f0106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 --2- 172.21.15.60:0/4076905873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0101e70 0x7f58f0104260 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 -- 172.21.15.60:0/4076905873 >> 172.21.15.60:0/4076905873 conn(0x7f58f00fb7c0 msgr2=0x7f58f00fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:04.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 -- 172.21.15.60:0/4076905873 shutdown_connections 2024-03-04T12:25:04.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 -- 172.21.15.60:0/4076905873 wait complete. 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 Processor -- start 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 -- start start 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0101e70 0x7f58f01017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f01047a0 0x7f58f00ffe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58f01070d0 0x7f58f0100360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f010d000 con 0x7f58f0101e70 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f010ce80 con 0x7f58f01047a0 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.830+0000 7f58f6281640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f010d180 con 0x7f58f01070d0 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0101e70 0x7f58f01017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58effff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0101e70 0x7f58f01017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46710/0 (socket says 172.21.15.60:46710) 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58effff640 1 -- 172.21.15.60:0/1464896609 learned_addr learned my addr 172.21.15.60:0/1464896609 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58f481e640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58f01070d0 0x7f58f0100360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:04.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58ef7fe640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f01047a0 0x7f58f00ffe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:04.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58effff640 1 -- 172.21.15.60:0/1464896609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58f01070d0 msgr2=0x7f58f0100360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58effff640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58f01070d0 0x7f58f0100360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58effff640 1 -- 172.21.15.60:0/1464896609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f01047a0 msgr2=0x7f58f00ffe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:04.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58effff640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f01047a0 0x7f58f00ffe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:04.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58effff640 1 -- 172.21.15.60:0/1464896609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f0100bf0 con 0x7f58f0101e70 2024-03-04T12:25:04.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58ef7fe640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f01047a0 0x7f58f00ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:04.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58effff640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0101e70 0x7f58f01017e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f58e0009230 tx=0x7f58e0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:04.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58ed7fa640 1 -- 172.21.15.60:0/1464896609 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e0041070 con 0x7f58f0101e70 2024-03-04T12:25:04.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58ed7fa640 1 -- 172.21.15.60:0/1464896609 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58e00044d0 con 0x7f58f0101e70 2024-03-04T12:25:04.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58f6281640 1 -- 172.21.15.60:0/1464896609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f01a1770 con 0x7f58f0101e70 2024-03-04T12:25:04.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58ed7fa640 1 -- 172.21.15.60:0/1464896609 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e0034440 con 0x7f58f0101e70 2024-03-04T12:25:04.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.831+0000 7f58f6281640 1 -- 172.21.15.60:0/1464896609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f01a1d10 con 0x7f58f0101e70 2024-03-04T12:25:04.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.832+0000 7f58f6281640 1 -- 172.21.15.60:0/1464896609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f00fd2b0 con 0x7f58f0101e70 2024-03-04T12:25:04.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.833+0000 7f58ed7fa640 1 -- 172.21.15.60:0/1464896609 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f58e0044070 con 0x7f58f0101e70 2024-03-04T12:25:04.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.834+0000 7f58ed7fa640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f58d80a1e90 0x7f58d80a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:04.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.834+0000 7f58ed7fa640 1 -- 172.21.15.60:0/1464896609 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f58e00f0200 con 0x7f58f0101e70 2024-03-04T12:25:04.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.834+0000 7f58ef7fe640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f58d80a1e90 0x7f58d80a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:04.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.834+0000 7f58ef7fe640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f58d80a1e90 0x7f58d80a4350 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f58f0101550 tx=0x7f58e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:04.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:04.837+0000 7f58ed7fa640 1 -- 172.21.15.60:0/1464896609 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f58e00bd220 con 0x7f58f0101e70 2024-03-04T12:25:05.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.024+0000 7f58f6281640 1 -- 172.21.15.60:0/1464896609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f58f002d040 con 0x7f58f0101e70 2024-03-04T12:25:05.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.024+0000 7f58ed7fa640 1 -- 172.21.15.60:0/1464896609 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f58e00bcb20 con 0x7f58f0101e70 2024-03-04T12:25:05.031 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-03-04T12:25:05.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.027+0000 7f58f6281640 1 -- 172.21.15.60:0/1464896609 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f58d80a1e90 msgr2=0x7f58d80a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.027+0000 7f58f6281640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f58d80a1e90 0x7f58d80a4350 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f58f0101550 tx=0x7f58e4005f90 comp rx=0 tx=0).stop 2024-03-04T12:25:05.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.027+0000 7f58f6281640 1 -- 172.21.15.60:0/1464896609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0101e70 msgr2=0x7f58f01017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.027+0000 7f58f6281640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0101e70 0x7f58f01017e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f58e0009230 tx=0x7f58e0005d50 comp rx=0 tx=0).stop 2024-03-04T12:25:05.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.027+0000 7f58f6281640 1 -- 172.21.15.60:0/1464896609 shutdown_connections 2024-03-04T12:25:05.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.027+0000 7f58f6281640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f58d80a1e90 0x7f58d80a4350 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.027+0000 7f58f6281640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58f01070d0 0x7f58f0100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.027+0000 7f58f6281640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f01047a0 0x7f58f00ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.027+0000 7f58f6281640 1 --2- 172.21.15.60:0/1464896609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f58f0101e70 0x7f58f01017e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.027+0000 7f58f6281640 1 -- 172.21.15.60:0/1464896609 >> 172.21.15.60:0/1464896609 conn(0x7f58f00fb7c0 msgr2=0x7f58f0102890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:05.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.027+0000 7f58f6281640 1 -- 172.21.15.60:0/1464896609 shutdown_connections 2024-03-04T12:25:05.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.027+0000 7f58f6281640 1 -- 172.21.15.60:0/1464896609 wait complete. 2024-03-04T12:25:05.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:04 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3544735661' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:05.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:04 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/877628799' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:05.043 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:04 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3520959174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:05.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T12:25:05.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 -- 172.21.15.60:0/2906896056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68fc100240 msgr2=0x7f68fc100660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 --2- 172.21.15.60:0/2906896056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68fc100240 0x7f68fc100660 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009900 tx=0x7f68ec02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:05.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 -- 172.21.15.60:0/2906896056 shutdown_connections 2024-03-04T12:25:05.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 --2- 172.21.15.60:0/2906896056 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68fc101df0 0x7f68fc108680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 --2- 172.21.15.60:0/2906896056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68fc101430 0x7f68fc1018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 --2- 172.21.15.60:0/2906896056 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68fc100240 0x7f68fc100660 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 -- 172.21.15.60:0/2906896056 >> 172.21.15.60:0/2906896056 conn(0x7f68fc0fb790 msgr2=0x7f68fc0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 -- 172.21.15.60:0/2906896056 shutdown_connections 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 -- 172.21.15.60:0/2906896056 wait complete. 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 Processor -- start 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 -- start start 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68fc100240 0x7f68fc1041d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.144+0000 7f690264a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68fc101430 0x7f68fc102810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f690264a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68fc101df0 0x7f68fc102d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f690264a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68fc10bf10 con 0x7f68fc101df0 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f690264a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68fc10bd90 con 0x7f68fc100240 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f690264a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68fc10c090 con 0x7f68fc101430 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f6900be7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68fc101df0 0x7f68fc102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f6900be7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68fc101df0 0x7f68fc102d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:46728/0 (socket says 172.21.15.60:46728) 2024-03-04T12:25:05.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f6900be7640 1 -- 172.21.15.60:0/3585638005 learned_addr learned my addr 172.21.15.60:0/3585638005 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f68fbfff640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68fc100240 0x7f68fc1041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f68fb7fe640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68fc101430 0x7f68fc102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f6900be7640 1 -- 172.21.15.60:0/3585638005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68fc101430 msgr2=0x7f68fc102810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f6900be7640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68fc101430 0x7f68fc102810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f6900be7640 1 -- 172.21.15.60:0/3585638005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68fc100240 msgr2=0x7f68fc1041d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f6900be7640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68fc100240 0x7f68fc1041d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f6900be7640 1 -- 172.21.15.60:0/3585638005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68fc1035e0 con 0x7f68fc101df0 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f6900be7640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68fc101df0 0x7f68fc102d50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f68f400cce0 tx=0x7f68f400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f68f97fa640 1 -- 172.21.15.60:0/3585638005 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f400f450 con 0x7f68fc101df0 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f68f97fa640 1 -- 172.21.15.60:0/3585638005 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f68f400fad0 con 0x7f68fc101df0 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f68f97fa640 1 -- 172.21.15.60:0/3585638005 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4018c10 con 0x7f68fc101df0 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f690264a640 1 -- 172.21.15.60:0/3585638005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68fc1a17a0 con 0x7f68fc101df0 2024-03-04T12:25:05.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.145+0000 7f690264a640 1 -- 172.21.15.60:0/3585638005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68fc1a1cf0 con 0x7f68fc101df0 2024-03-04T12:25:05.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.146+0000 7f690264a640 1 -- 172.21.15.60:0/3585638005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68fc1a1fb0 con 0x7f68fc101df0 2024-03-04T12:25:05.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.147+0000 7f68f97fa640 1 -- 172.21.15.60:0/3585638005 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f68f400f5f0 con 0x7f68fc101df0 2024-03-04T12:25:05.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.148+0000 7f68f97fa640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f68e40a1e90 0x7f68e40a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:05.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.148+0000 7f68fbfff640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f68e40a1e90 0x7f68e40a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:05.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.148+0000 7f68f97fa640 1 -- 172.21.15.60:0/3585638005 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f68f40ce980 con 0x7f68fc101df0 2024-03-04T12:25:05.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.148+0000 7f68fbfff640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f68e40a1e90 0x7f68e40a4350 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f68ec000c00 tx=0x7f68ec004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:05.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.152+0000 7f68f97fa640 1 -- 172.21.15.60:0/3585638005 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f68f4014030 con 0x7f68fc101df0 2024-03-04T12:25:05.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.333+0000 7f690264a640 1 -- 172.21.15.60:0/3585638005 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f68fc1a22a0 con 0x7f68fc101df0 2024-03-04T12:25:05.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.333+0000 7f68f97fa640 1 -- 172.21.15.60:0/3585638005 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f68f409b8f0 con 0x7f68fc101df0 2024-03-04T12:25:05.339 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:05.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.335+0000 7f690264a640 1 -- 172.21.15.60:0/3585638005 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f68e40a1e90 msgr2=0x7f68e40a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.335+0000 7f690264a640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f68e40a1e90 0x7f68e40a4350 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f68ec000c00 tx=0x7f68ec004f10 comp rx=0 tx=0).stop 2024-03-04T12:25:05.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.335+0000 7f690264a640 1 -- 172.21.15.60:0/3585638005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68fc101df0 msgr2=0x7f68fc102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.335+0000 7f690264a640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68fc101df0 0x7f68fc102d50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f68f400cce0 tx=0x7f68f400a620 comp rx=0 tx=0).stop 2024-03-04T12:25:05.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.335+0000 7f690264a640 1 -- 172.21.15.60:0/3585638005 shutdown_connections 2024-03-04T12:25:05.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.335+0000 7f690264a640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f68e40a1e90 0x7f68e40a4350 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.335+0000 7f690264a640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f68fc101df0 0x7f68fc102d50 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.335+0000 7f690264a640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f68fc101430 0x7f68fc102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.335+0000 7f690264a640 1 --2- 172.21.15.60:0/3585638005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68fc100240 0x7f68fc1041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.335+0000 7f690264a640 1 -- 172.21.15.60:0/3585638005 >> 172.21.15.60:0/3585638005 conn(0x7f68fc0fb790 msgr2=0x7f68fc0fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.335+0000 7f690264a640 1 -- 172.21.15.60:0/3585638005 shutdown_connections 2024-03-04T12:25:05.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.335+0000 7f690264a640 1 -- 172.21.15.60:0/3585638005 wait complete. 2024-03-04T12:25:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:04 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3544735661' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:04 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/877628799' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:04 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3520959174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:04 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3544735661' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:04 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/877628799' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:04 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3520959174' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:05.352 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:25:05.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.449+0000 7ff955861640 1 -- 172.21.15.60:0/3830834689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9501025b0 msgr2=0x7ff950102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.449+0000 7ff955861640 1 --2- 172.21.15.60:0/3830834689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9501025b0 0x7ff950102a30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff944009900 tx=0x7ff94402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.449+0000 7ff955861640 1 -- 172.21.15.60:0/3830834689 shutdown_connections 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.449+0000 7ff955861640 1 --2- 172.21.15.60:0/3830834689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff950102f70 0x7ff950109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.449+0000 7ff955861640 1 --2- 172.21.15.60:0/3830834689 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9501025b0 0x7ff950102a30 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.449+0000 7ff955861640 1 --2- 172.21.15.60:0/3830834689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9501013c0 0x7ff9501017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.449+0000 7ff955861640 1 -- 172.21.15.60:0/3830834689 >> 172.21.15.60:0/3830834689 conn(0x7ff9500fc950 msgr2=0x7ff9500fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.449+0000 7ff955861640 1 -- 172.21.15.60:0/3830834689 shutdown_connections 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.449+0000 7ff955861640 1 -- 172.21.15.60:0/3830834689 wait complete. 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.449+0000 7ff955861640 1 Processor -- start 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff955861640 1 -- start start 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff955861640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9501013c0 0x7ff9500750e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff955861640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9501025b0 0x7ff950073720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:05.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff955861640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff950102f70 0x7ff950073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff955861640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95010cfe0 con 0x7ff9501025b0 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff955861640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95010ce60 con 0x7ff9501013c0 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff955861640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95010d160 con 0x7ff950102f70 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff94e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9501025b0 0x7ff950073720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff94e7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9501025b0 0x7ff950073720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60358/0 (socket says 172.21.15.60:60358) 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff94e7fc640 1 -- 172.21.15.60:0/1888240286 learned_addr learned my addr 172.21.15.60:0/1888240286 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff94f7fe640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff950102f70 0x7ff950073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff94effd640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9501013c0 0x7ff9500750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff94e7fc640 1 -- 172.21.15.60:0/1888240286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff950102f70 msgr2=0x7ff950073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff94e7fc640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff950102f70 0x7ff950073c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff94e7fc640 1 -- 172.21.15.60:0/1888240286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9501013c0 msgr2=0x7ff9500750e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff94e7fc640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9501013c0 0x7ff9500750e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff94e7fc640 1 -- 172.21.15.60:0/1888240286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9500744f0 con 0x7ff9501025b0 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff94effd640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9501013c0 0x7ff9500750e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:05.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.450+0000 7ff94e7fc640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9501025b0 0x7ff950073720 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff94402ce70 tx=0x7ff944005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.451+0000 7ff95485f640 1 -- 172.21.15.60:0/1888240286 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944041070 con 0x7ff9501025b0 2024-03-04T12:25:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.451+0000 7ff95485f640 1 -- 172.21.15.60:0/1888240286 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9440054e0 con 0x7ff9501025b0 2024-03-04T12:25:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.451+0000 7ff955861640 1 -- 172.21.15.60:0/1888240286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9501a1730 con 0x7ff9501025b0 2024-03-04T12:25:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.451+0000 7ff95485f640 1 -- 172.21.15.60:0/1888240286 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440346a0 con 0x7ff9501025b0 2024-03-04T12:25:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.451+0000 7ff955861640 1 -- 172.21.15.60:0/1888240286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9501a1c80 con 0x7ff9501025b0 2024-03-04T12:25:05.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.452+0000 7ff955861640 1 -- 172.21.15.60:0/1888240286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff950108680 con 0x7ff9501025b0 2024-03-04T12:25:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.453+0000 7ff95485f640 1 -- 172.21.15.60:0/1888240286 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7ff944044070 con 0x7ff9501025b0 2024-03-04T12:25:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.453+0000 7ff95485f640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff9380a1e10 0x7ff9380a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.453+0000 7ff94effd640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff9380a1e10 0x7ff9380a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.453+0000 7ff95485f640 1 -- 172.21.15.60:0/1888240286 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff9440efe60 con 0x7ff9501025b0 2024-03-04T12:25:05.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.454+0000 7ff94effd640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff9380a1e10 0x7ff9380a42d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff940005d00 tx=0x7ff940005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:05.463 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.457+0000 7ff95485f640 1 -- 172.21.15.60:0/1888240286 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff9440bcee0 con 0x7ff9501025b0 2024-03-04T12:25:05.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.676+0000 7ff955861640 1 -- 172.21.15.60:0/1888240286 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff95002ceb0 con 0x7ff9501025b0 2024-03-04T12:25:05.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.677+0000 7ff95485f640 1 -- 172.21.15.60:0/1888240286 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11373 (secure 0 0 0) 0x7ff9440bc7e0 con 0x7ff9501025b0 2024-03-04T12:25:05.683 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:05.683 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:23:41.896895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:25:05.683 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 28 2024-03-04T12:25:05.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.679+0000 7ff955861640 1 -- 172.21.15.60:0/1888240286 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff9380a1e10 msgr2=0x7ff9380a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.679+0000 7ff955861640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff9380a1e10 0x7ff9380a42d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff940005d00 tx=0x7ff940005c90 comp rx=0 tx=0).stop 2024-03-04T12:25:05.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.679+0000 7ff955861640 1 -- 172.21.15.60:0/1888240286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9501025b0 msgr2=0x7ff950073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.679+0000 7ff955861640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9501025b0 0x7ff950073720 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff94402ce70 tx=0x7ff944005e40 comp rx=0 tx=0).stop 2024-03-04T12:25:05.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.679+0000 7ff955861640 1 -- 172.21.15.60:0/1888240286 shutdown_connections 2024-03-04T12:25:05.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.679+0000 7ff955861640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff9380a1e10 0x7ff9380a42d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.679+0000 7ff955861640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff950102f70 0x7ff950073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.679+0000 7ff955861640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff9501025b0 0x7ff950073720 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.679+0000 7ff955861640 1 --2- 172.21.15.60:0/1888240286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9501013c0 0x7ff9500750e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.679+0000 7ff955861640 1 -- 172.21.15.60:0/1888240286 >> 172.21.15.60:0/1888240286 conn(0x7ff9500fc950 msgr2=0x7ff950105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:05.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.679+0000 7ff955861640 1 -- 172.21.15.60:0/1888240286 shutdown_connections 2024-03-04T12:25:05.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.679+0000 7ff955861640 1 -- 172.21.15.60:0/1888240286 wait complete. 2024-03-04T12:25:05.696 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:25:05.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.793+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/1029395875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f78104710 msgr2=0x7f9f78106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.801 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.793+0000 7f9f7f2e8640 1 --2- 172.21.15.60:0/1029395875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f78104710 0x7f9f78106b00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9f68009900 tx=0x7f9f6802bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.793+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/1029395875 shutdown_connections 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.793+0000 7f9f7f2e8640 1 --2- 172.21.15.60:0/1029395875 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f78107040 0x7f9f78109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.793+0000 7f9f7f2e8640 1 --2- 172.21.15.60:0/1029395875 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f78104710 0x7f9f78106b00 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.793+0000 7f9f7f2e8640 1 --2- 172.21.15.60:0/1029395875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f78101de0 0x7f9f781041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.793+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/1029395875 >> 172.21.15.60:0/1029395875 conn(0x7f9f780fb7f0 msgr2=0x7f9f780fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.793+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/1029395875 shutdown_connections 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.793+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/1029395875 wait complete. 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.793+0000 7f9f7f2e8640 1 Processor -- start 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7f2e8640 1 -- start start 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7f2e8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f78101de0 0x7f9f78197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7f2e8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f78104710 0x7f9f78197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7f2e8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f78107040 0x7f9f7819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:05.802 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7f2e8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7810cff0 con 0x7f9f78107040 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7f2e8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7810ce70 con 0x7f9f78104710 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7f2e8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7810d170 con 0x7f9f78101de0 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7d885640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f78107040 0x7f9f7819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7d885640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f78107040 0x7f9f7819ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60374/0 (socket says 172.21.15.60:60374) 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7d885640 1 -- 172.21.15.60:0/466301686 learned_addr learned my addr 172.21.15.60:0/466301686 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7d084640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f78101de0 0x7f9f78197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7c883640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f78104710 0x7f9f78197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7d885640 1 -- 172.21.15.60:0/466301686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f78101de0 msgr2=0x7f9f78197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7d885640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f78101de0 0x7f9f78197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7d885640 1 -- 172.21.15.60:0/466301686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f78104710 msgr2=0x7f9f78197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7d885640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f78104710 0x7f9f78197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7d885640 1 -- 172.21.15.60:0/466301686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7819f230 con 0x7f9f78107040 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7d084640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f78101de0 0x7f9f78197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7c883640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f78104710 0x7f9f78197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7d885640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f78107040 0x7f9f7819ebc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000cce0 tx=0x7f9f7000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f6e7fc640 1 -- 172.21.15.60:0/466301686 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7000f450 con 0x7f9f78107040 2024-03-04T12:25:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f6e7fc640 1 -- 172.21.15.60:0/466301686 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f7000fad0 con 0x7f9f78107040 2024-03-04T12:25:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/466301686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f7819f520 con 0x7f9f78107040 2024-03-04T12:25:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.794+0000 7f9f6e7fc640 1 -- 172.21.15.60:0/466301686 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f70018c10 con 0x7f9f78107040 2024-03-04T12:25:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.795+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/466301686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f7819fa70 con 0x7f9f78107040 2024-03-04T12:25:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.795+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/466301686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f780fd2e0 con 0x7f9f78107040 2024-03-04T12:25:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.797+0000 7f9f6e7fc640 1 -- 172.21.15.60:0/466301686 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 17) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f9f7000f5f0 con 0x7f9f78107040 2024-03-04T12:25:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.797+0000 7f9f6e7fc640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9f640a1e90 0x7f9f640a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.797+0000 7f9f7d084640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9f640a1e90 0x7f9f640a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.797+0000 7f9f6e7fc640 1 -- 172.21.15.60:0/466301686 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9f700ceac0 con 0x7f9f78107040 2024-03-04T12:25:05.805 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.798+0000 7f9f7d084640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9f640a1e90 0x7f9f640a4350 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009c20 tx=0x7f9f74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:05.806 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.801+0000 7f9f6e7fc640 1 -- 172.21.15.60:0/466301686 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9f70014030 con 0x7f9f78107040 2024-03-04T12:25:05.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.941+0000 7f9f6e7fc640 1 -- 172.21.15.60:0/466301686 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f9f70097a50 con 0x7f9f78107040 2024-03-04T12:25:05.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.990+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/466301686 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f78061f30 con 0x7f9f78107040 2024-03-04T12:25:05.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.991+0000 7f9f6e7fc640 1 -- 172.21.15.60:0/466301686 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24562 (secure 0 0 0) 0x7f9f7009bae0 con 0x7f9f78107040 2024-03-04T12:25:05.998 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:05.998 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":92,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:25:05.921266+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:25:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.993+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/466301686 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9f640a1e90 msgr2=0x7f9f640a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.993+0000 7f9f7f2e8640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9f640a1e90 0x7f9f640a4350 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009c20 tx=0x7f9f74005f90 comp rx=0 tx=0).stop 2024-03-04T12:25:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.993+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/466301686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f78107040 msgr2=0x7f9f7819ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.993+0000 7f9f7f2e8640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f78107040 0x7f9f7819ebc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000cce0 tx=0x7f9f7000a620 comp rx=0 tx=0).stop 2024-03-04T12:25:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.993+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/466301686 shutdown_connections 2024-03-04T12:25:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.993+0000 7f9f7f2e8640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9f640a1e90 0x7f9f640a4350 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.993+0000 7f9f7f2e8640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f78107040 0x7f9f7819ebc0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.993+0000 7f9f7f2e8640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f78104710 0x7f9f78197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.993+0000 7f9f7f2e8640 1 --2- 172.21.15.60:0/466301686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f78101de0 0x7f9f78197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.993+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/466301686 >> 172.21.15.60:0/466301686 conn(0x7f9f780fb7f0 msgr2=0x7f9f78102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.993+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/466301686 shutdown_connections 2024-03-04T12:25:06.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:05.994+0000 7f9f7f2e8640 1 -- 172.21.15.60:0/466301686 wait complete. 2024-03-04T12:25:06.009 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:05 smithi060 ceph-mon[32745]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:25:06.010 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-04T12:25:06.010 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:05 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-04T12:25:06.010 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:05 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1464896609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:06.010 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:05 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3585638005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:06.010 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:05 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1888240286' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:06.011 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:25:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.109+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/4350077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa4101380 msgr2=0x7fcfa41017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.109+0000 7fcfaa9e1640 1 --2- 172.21.15.60:0/4350077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa4101380 0x7fcfa41017a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fcf94009900 tx=0x7fcf9402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.109+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/4350077 shutdown_connections 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.109+0000 7fcfaa9e1640 1 --2- 172.21.15.60:0/4350077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfa4102f30 0x7fcfa41097c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.109+0000 7fcfaa9e1640 1 --2- 172.21.15.60:0/4350077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfa4102570 0x7fcfa41029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.109+0000 7fcfaa9e1640 1 --2- 172.21.15.60:0/4350077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa4101380 0x7fcfa41017a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.109+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/4350077 >> 172.21.15.60:0/4350077 conn(0x7fcfa40fc8f0 msgr2=0x7fcfa40fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.109+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/4350077 shutdown_connections 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.109+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/4350077 wait complete. 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.110+0000 7fcfaa9e1640 1 Processor -- start 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.110+0000 7fcfaa9e1640 1 -- start start 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.110+0000 7fcfaa9e1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfa4101380 0x7fcfa4197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.110+0000 7fcfaa9e1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa4102570 0x7fcfa4197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.110+0000 7fcfaa9e1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfa4102f30 0x7fcfa419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.110+0000 7fcfaa9e1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa410d060 con 0x7fcfa4102570 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.110+0000 7fcfaa9e1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa410cee0 con 0x7fcfa4102f30 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.110+0000 7fcfaa9e1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa410d1e0 con 0x7fcfa4101380 2024-03-04T12:25:06.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.110+0000 7fcfa37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa4102570 0x7fcfa4197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.110+0000 7fcfa37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa4102570 0x7fcfa4197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60388/0 (socket says 172.21.15.60:60388) 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.110+0000 7fcfa37fe640 1 -- 172.21.15.60:0/1095837321 learned_addr learned my addr 172.21.15.60:0/1095837321 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfa3fff640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfa4101380 0x7fcfa4197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfa8f7e640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfa4102f30 0x7fcfa419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfa37fe640 1 -- 172.21.15.60:0/1095837321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfa4101380 msgr2=0x7fcfa4197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfa37fe640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfa4101380 0x7fcfa4197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfa37fe640 1 -- 172.21.15.60:0/1095837321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfa4102f30 msgr2=0x7fcfa419ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfa37fe640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfa4102f30 0x7fcfa419ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfa37fe640 1 -- 172.21.15.60:0/1095837321 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa419f280 con 0x7fcfa4102570 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfa37fe640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa4102570 0x7fcfa4197ba0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800e9f0 tx=0x7fcf9800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfa17fa640 1 -- 172.21.15.60:0/1095837321 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800cd80 con 0x7fcfa4102570 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfa17fa640 1 -- 172.21.15.60:0/1095837321 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf98010910 con 0x7fcfa4102570 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/1095837321 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa419f570 con 0x7fcfa4102570 2024-03-04T12:25:06.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfa17fa640 1 -- 172.21.15.60:0/1095837321 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98018990 con 0x7fcfa4102570 2024-03-04T12:25:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.111+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/1095837321 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa419fac0 con 0x7fcfa4102570 2024-03-04T12:25:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.112+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/1095837321 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa4108640 con 0x7fcfa4102570 2024-03-04T12:25:06.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.113+0000 7fcfa17fa640 1 -- 172.21.15.60:0/1095837321 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fcf98010ac0 con 0x7fcfa4102570 2024-03-04T12:25:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.114+0000 7fcfa17fa640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcf8c0a1d80 0x7fcf8c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.114+0000 7fcfa3fff640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcf8c0a1d80 0x7fcf8c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.114+0000 7fcfa17fa640 1 -- 172.21.15.60:0/1095837321 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fcf980ce4b0 con 0x7fcfa4102570 2024-03-04T12:25:06.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.114+0000 7fcfa3fff640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcf8c0a1d80 0x7fcf8c0a4240 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcf9402c170 tx=0x7fcf9402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.118+0000 7fcfa17fa640 1 -- 172.21.15.60:0/1095837321 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcf9809aea0 con 0x7fcfa4102570 2024-03-04T12:25:06.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.337+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/1095837321 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfa402ceb0 con 0x7fcfa4102570 2024-03-04T12:25:06.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.338+0000 7fcfa17fa640 1 -- 172.21.15.60:0/1095837321 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11373 (secure 0 0 0) 0x7fcf9809ef40 con 0x7fcfa4102570 2024-03-04T12:25:06.344 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:06.344 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:23:41.896895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:25:06.345 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 28 2024-03-04T12:25:06.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.340+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/1095837321 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcf8c0a1d80 msgr2=0x7fcf8c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.340+0000 7fcfaa9e1640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcf8c0a1d80 0x7fcf8c0a4240 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcf9402c170 tx=0x7fcf9402df90 comp rx=0 tx=0).stop 2024-03-04T12:25:06.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.340+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/1095837321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa4102570 msgr2=0x7fcfa4197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.340+0000 7fcfaa9e1640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa4102570 0x7fcfa4197ba0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800e9f0 tx=0x7fcf9800eeb0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.340+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/1095837321 shutdown_connections 2024-03-04T12:25:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.340+0000 7fcfaa9e1640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fcf8c0a1d80 0x7fcf8c0a4240 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.340+0000 7fcfaa9e1640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcfa4102f30 0x7fcfa419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.340+0000 7fcfaa9e1640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fcfa4102570 0x7fcfa4197ba0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.340+0000 7fcfaa9e1640 1 --2- 172.21.15.60:0/1095837321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfa4101380 0x7fcfa4197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.340+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/1095837321 >> 172.21.15.60:0/1095837321 conn(0x7fcfa40fc8f0 msgr2=0x7fcfa4105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.340+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/1095837321 shutdown_connections 2024-03-04T12:25:06.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.340+0000 7fcfaa9e1640 1 -- 172.21.15.60:0/1095837321 wait complete. 2024-03-04T12:25:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:05 smithi086 ceph-mon[36851]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:25:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-04T12:25:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:05 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-04T12:25:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:05 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1464896609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:05 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3585638005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:06.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:05 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1888240286' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:05 smithi063 ceph-mon[36717]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:25:06.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-04T12:25:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:05 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-04T12:25:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:05 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1464896609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:05 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3585638005' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:05 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1888240286' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:06.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-04T12:23:12.241967+0000', 'modified': '2024-03-04T12:23:41.896895+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24524, 'mds_1': 24487, 'mds_2': 24479, 'mds_3': 24407, 'mds_4': 24394}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14631': {'gid': 14631, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.60:6839/2180757903', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6838', 'nonce': 2180757903}, {'type': 'v1', 'addr': '172.21.15.60:6839', 'nonce': 2180757903}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14643': {'gid': 14643, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.60:6841/4175833227', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6840', 'nonce': 4175833227}, {'type': 'v1', 'addr': '172.21.15.60:6841', 'nonce': 4175833227}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24394': {'gid': 24394, 'name': 'l', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.86:6833/777854669', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6832', 'nonce': 777854669}, {'type': 'v1', 'addr': '172.21.15.86:6833', 'nonce': 777854669}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24407': {'gid': 24407, 'name': 'k', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.63:6833/2270964219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6832', 'nonce': 2270964219}, {'type': 'v1', 'addr': '172.21.15.63:6833', 'nonce': 2270964219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.63:6835/2453356505', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6834', 'nonce': 2453356505}, {'type': 'v1', 'addr': '172.21.15.63:6835', 'nonce': 2453356505}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.86:6835/2934725061', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2934725061}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2934725061}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.86:6837/472598778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 472598778}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 472598778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.63:6837/3974574804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6836', 'nonce': 3974574804}, {'type': 'v1', 'addr': '172.21.15.63:6837', 'nonce': 3974574804}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.86:6839/3347800322', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6838', 'nonce': 3347800322}, {'type': 'v1', 'addr': '172.21.15.86:6839', 'nonce': 3347800322}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.63:6839/2728229139', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6838', 'nonce': 2728229139}, {'type': 'v1', 'addr': '172.21.15.63:6839', 'nonce': 2728229139}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T12:25:06.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-04T12:25:06.358 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-03-04T12:25:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.455+0000 7f14e2ef7640 1 -- 172.21.15.60:0/1547900641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14dc101440 msgr2=0x7f14dc1018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.455+0000 7f14e2ef7640 1 --2- 172.21.15.60:0/1547900641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14dc101440 0x7f14dc1018c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009900 tx=0x7f14cc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.455+0000 7f14e2ef7640 1 -- 172.21.15.60:0/1547900641 shutdown_connections 2024-03-04T12:25:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.455+0000 7f14e2ef7640 1 --2- 172.21.15.60:0/1547900641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14dc101e00 0x7f14dc108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.455+0000 7f14e2ef7640 1 --2- 172.21.15.60:0/1547900641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14dc101440 0x7f14dc1018c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.455+0000 7f14e2ef7640 1 --2- 172.21.15.60:0/1547900641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14dc100250 0x7f14dc100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.455+0000 7f14e2ef7640 1 -- 172.21.15.60:0/1547900641 >> 172.21.15.60:0/1547900641 conn(0x7f14dc0fb7e0 msgr2=0x7f14dc0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.455+0000 7f14e2ef7640 1 -- 172.21.15.60:0/1547900641 shutdown_connections 2024-03-04T12:25:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.455+0000 7f14e2ef7640 1 -- 172.21.15.60:0/1547900641 wait complete. 2024-03-04T12:25:06.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e2ef7640 1 Processor -- start 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e2ef7640 1 -- start start 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e2ef7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14dc100250 0x7f14dc197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e2ef7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14dc101440 0x7f14dc197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e2ef7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14dc101e00 0x7f14dc19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e2ef7640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc10bee0 con 0x7f14dc101e00 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e2ef7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc10bd60 con 0x7f14dc100250 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e2ef7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc10c060 con 0x7f14dc101440 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e1494640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14dc101e00 0x7f14dc19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e1494640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14dc101e00 0x7f14dc19ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60400/0 (socket says 172.21.15.60:60400) 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e1494640 1 -- 172.21.15.60:0/648298463 learned_addr learned my addr 172.21.15.60:0/648298463 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e0c93640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14dc100250 0x7f14dc197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14d3fff640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14dc101440 0x7f14dc197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e1494640 1 -- 172.21.15.60:0/648298463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14dc101440 msgr2=0x7f14dc197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e1494640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14dc101440 0x7f14dc197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e1494640 1 -- 172.21.15.60:0/648298463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14dc100250 msgr2=0x7f14dc197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e1494640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14dc100250 0x7f14dc197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.456+0000 7f14e1494640 1 -- 172.21.15.60:0/648298463 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc19f240 con 0x7f14dc101e00 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.457+0000 7f14e1494640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14dc101e00 0x7f14dc19ebd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f14d400eaa0 tx=0x7f14d400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.457+0000 7f14d1ffb640 1 -- 172.21.15.60:0/648298463 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d400cdb0 con 0x7f14dc101e00 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.457+0000 7f14e2ef7640 1 -- 172.21.15.60:0/648298463 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14dc19f530 con 0x7f14dc101e00 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.457+0000 7f14d1ffb640 1 -- 172.21.15.60:0/648298463 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14d4005020 con 0x7f14dc101e00 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.457+0000 7f14d1ffb640 1 -- 172.21.15.60:0/648298463 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d40116b0 con 0x7f14dc101e00 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.457+0000 7f14e2ef7640 1 -- 172.21.15.60:0/648298463 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14dc19fa00 con 0x7f14dc101e00 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.459+0000 7f14d1ffb640 1 -- 172.21.15.60:0/648298463 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f14d4005560 con 0x7f14dc101e00 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.459+0000 7f14d1ffb640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f14c40a1d80 0x7f14c40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.459+0000 7f14e5ba8740 1 Processor -- start 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e5ba8740 1 -- start start 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e0c93640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f14c40a1d80 0x7f14c40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14d1ffb640 1 -- 172.21.15.60:0/648298463 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f14d40ce370 con 0x7f14dc101e00 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e5ba8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 0x56131343e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e5ba8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56131343ebf0 0x5613134436f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e5ba8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561313443d60 0x5613134461f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e5ba8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x56131339f7c0 con 0x56131343e200 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e5ba8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x561313348b50 con 0x561313443d60 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e5ba8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56131339b860 con 0x56131343ebf0 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e0c93640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 0x56131343e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e0c93640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 0x56131343e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60412/0 (socket says 172.21.15.60:60412) 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e0c93640 1 -- 172.21.15.60:0/2518863665 learned_addr learned my addr 172.21.15.60:0/2518863665 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e1494640 1 --2- 172.21.15.60:0/2518863665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561313443d60 0x5613134461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14d3fff640 1 --2- 172.21.15.60:0/2518863665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56131343ebf0 0x5613134436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e0c93640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f14c40a1d80 0x7f14c40a4240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f14dc1012a0 tx=0x7f14d800af80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e0c93640 1 -- 172.21.15.60:0/2518863665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56131343ebf0 msgr2=0x5613134436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e0c93640 1 --2- 172.21.15.60:0/2518863665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56131343ebf0 0x5613134436f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e0c93640 1 -- 172.21.15.60:0/2518863665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561313443d60 msgr2=0x5613134461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e0c93640 1 --2- 172.21.15.60:0/2518863665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561313443d60 0x5613134461f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e0c93640 1 -- 172.21.15.60:0/2518863665 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56131335c9a0 con 0x56131343e200 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14e0c93640 1 --2- 172.21.15.60:0/2518863665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 0x56131343e620 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f14d8012450 tx=0x7f14d801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=318ccbf95f1458bf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.460+0000 7f14c2ffd640 1 -- 172.21.15.60:0/2518863665 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8018070 con 0x56131343e200 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.461+0000 7f14c2ffd640 1 -- 172.21.15.60:0/2518863665 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14d8010b10 con 0x56131343e200 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.461+0000 7f14c2ffd640 1 -- 172.21.15.60:0/2518863665 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d801cb30 con 0x56131343e200 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.461+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2518863665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 msgr2=0x56131343e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.461+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2518863665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 0x56131343e620 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f14d8012450 tx=0x7f14d801a000 comp rx=0 tx=0).stop 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.461+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2518863665 shutdown_connections 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.461+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2518863665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561313443d60 0x5613134461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.461+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2518863665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56131343ebf0 0x5613134436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.461+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2518863665 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 0x56131343e620 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.461+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2518863665 >> 172.21.15.60:0/2518863665 conn(0x561313435360 msgr2=0x561313435760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.461+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2518863665 shutdown_connections 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.461+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2518863665 wait complete. 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.461+0000 7f14e5ba8740 1 -- start start 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e5ba8740 1 Processor -- start 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e5ba8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343e200 0x5613134485f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e5ba8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56131343ebf0 0x561313448b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e5ba8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561313443d60 0x56131344fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e5ba8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x561313446cb0 con 0x561313443d60 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e5ba8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x56131335bf90 con 0x56131343e200 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e5ba8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x561313446e30 con 0x56131343ebf0 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e5ba8740 10 client.0 _ll_register_callbacks cb 0x56131337dd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e1494640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561313443d60 0x56131344fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e1494640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561313443d60 0x56131344fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60426/0 (socket says 172.21.15.60:60426) 2024-03-04T12:25:06.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e1494640 1 -- 172.21.15.60:0/1816832791 learned_addr learned my addr 172.21.15.60:0/1816832791 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e0c93640 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343e200 0x5613134485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14d3fff640 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56131343ebf0 0x561313448b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e1494640 1 -- 172.21.15.60:0/1816832791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56131343ebf0 msgr2=0x561313448b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e1494640 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56131343ebf0 0x561313448b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e1494640 1 -- 172.21.15.60:0/1816832791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343e200 msgr2=0x5613134485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e1494640 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343e200 0x5613134485f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14e1494640 1 -- 172.21.15.60:0/1816832791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5613130b42d0 con 0x561313443d60 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.462+0000 7f14d3fff640 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56131343ebf0 0x561313448b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14e1494640 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561313443d60 0x56131344fb70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f14d409acf0 tx=0x7f14d409ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f39256f44ae4901 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14bbfff640 1 -- 172.21.15.60:0/1816832791 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d401f040 con 0x561313443d60 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14bbfff640 1 -- 172.21.15.60:0/1816832791 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f14d40051d0 con 0x561313443d60 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14e5ba8740 1 -- 172.21.15.60:0/1816832791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14d8010b10 con 0x561313443d60 2024-03-04T12:25:06.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14bbfff640 1 -- 172.21.15.60:0/1816832791 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14d40b9170 con 0x561313443d60 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14e5ba8740 10 client.15237 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14e5ba8740 1 -- 172.21.15.60:0/1816832791 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56131317e6b0 con 0x561313443d60 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14bbfff640 1 -- 172.21.15.60:0/1816832791 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14d8010b10 con 0x561313443d60 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14bbfff640 1 -- 172.21.15.60:0/1816832791 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f14d400f040 con 0x561313443d60 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14e5ba8740 10 client.15237 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14e5ba8740 10 client.15237 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14e5ba8740 20 client.15237 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x561313455150 0x5613134541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14e5ba8740 4 client.15237 mds_command: new command op to 24394 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.463+0000 7f14e5ba8740 1 -- 172.21.15.60:0/1816832791 --> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5613133acd00 con 0x561313455150 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.464+0000 7f14e0c93640 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x561313455150 0x5613134541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.464+0000 7f14e0c93640 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x561313455150 0x5613134541f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1c95f3d9ff2169fd server_cookie=3bfeb055abdbbbb3 in_seq=0 out_seq=0 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.464+0000 7f14bbfff640 10 client.15237 ms_handle_connect on v2:172.21.15.86:6832/777854669 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14bbfff640 1 -- 172.21.15.60:0/1816832791 <== mds.4 v2:172.21.15.86:6832/777854669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5613133acd00 con 0x561313455150 2024-03-04T12:25:06.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14bbfff640 10 client.15237 handle_command_reply: tid=0 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14e5ba8740 1 client.15237 shutdown 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14e5ba8740 1 -- 172.21.15.60:0/1816832791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561313443d60 msgr2=0x56131344fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561313443d60 0x56131344fb70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f14d409acf0 tx=0x7f14d409ef40 comp rx=0 tx=0).stop 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14e5ba8740 1 -- 172.21.15.60:0/1816832791 shutdown_connections 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561313443d60 0x56131344fb70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56131343ebf0 0x561313448b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x561313455150 0x5613134541f0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/1816832791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343e200 0x5613134485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14bbfff640 0 client.15237 ms_handle_reset on v2:172.21.15.86:6832/777854669 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14e5ba8740 1 -- 172.21.15.60:0/1816832791 >> 172.21.15.60:0/1816832791 conn(0x561313435360 msgr2=0x561313434640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14e5ba8740 1 -- 172.21.15.60:0/1816832791 shutdown_connections 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14e5ba8740 1 -- 172.21.15.60:0/1816832791 wait complete. 2024-03-04T12:25:06.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.465+0000 7f14e5ba8740 20 client.15237 trim_cache size 0 max 16384 2024-03-04T12:25:06.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.480+0000 7f14e5ba8740 1 Processor -- start 2024-03-04T12:25:06.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.480+0000 7f14e5ba8740 1 -- start start 2024-03-04T12:25:06.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e5ba8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 0x5613134304d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e5ba8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343ebf0 0x56131344fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e5ba8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561313443d60 0x56131343f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e5ba8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x561313454970 con 0x56131343e200 2024-03-04T12:25:06.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e5ba8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x561313454fd0 con 0x56131343ebf0 2024-03-04T12:25:06.492 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e5ba8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x561313455630 con 0x561313443d60 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e0c93640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 0x5613134304d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e0c93640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 0x5613134304d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60440/0 (socket says 172.21.15.60:60440) 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e0c93640 1 -- 172.21.15.60:0/2472270154 learned_addr learned my addr 172.21.15.60:0/2472270154 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14d3fff640 1 --2- 172.21.15.60:0/2472270154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343ebf0 0x56131344fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e0c93640 1 -- 172.21.15.60:0/2472270154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561313443d60 msgr2=0x56131343f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e0c93640 1 --2- 172.21.15.60:0/2472270154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561313443d60 0x56131343f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e0c93640 1 -- 172.21.15.60:0/2472270154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343ebf0 msgr2=0x56131344fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e0c93640 1 --2- 172.21.15.60:0/2472270154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343ebf0 0x56131344fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e0c93640 1 -- 172.21.15.60:0/2472270154 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561313434f00 con 0x56131343e200 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14e0c93640 1 --2- 172.21.15.60:0/2472270154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 0x5613134304d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f14d8012ce0 tx=0x7f14d800ffb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff37079beed0df04 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14c27fc640 1 -- 172.21.15.60:0/2472270154 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8035070 con 0x56131343e200 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14c27fc640 1 -- 172.21.15.60:0/2472270154 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14d801c420 con 0x56131343e200 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.481+0000 7f14c27fc640 1 -- 172.21.15.60:0/2472270154 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8018040 con 0x56131343e200 2024-03-04T12:25:06.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.482+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2472270154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 msgr2=0x5613134304d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.482+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2472270154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 0x5613134304d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f14d8012ce0 tx=0x7f14d800ffb0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.482+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2472270154 shutdown_connections 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.482+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2472270154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561313443d60 0x56131343f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.482+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2472270154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343ebf0 0x56131344fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.482+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2472270154 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343e200 0x5613134304d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.482+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2472270154 >> 172.21.15.60:0/2472270154 conn(0x561313435360 msgr2=0x561313435740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.482+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2472270154 shutdown_connections 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.482+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2472270154 wait complete. 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.482+0000 7f14e5ba8740 1 -- start start 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.482+0000 7f14e5ba8740 1 Processor -- start 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14e5ba8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343e200 0x56131344cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14e5ba8740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343ebf0 0x56131344d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14e5ba8740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561313443d60 0x56131344e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14e5ba8740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613134562f0 con 0x56131343ebf0 2024-03-04T12:25:06.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14e5ba8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x561313455c90 con 0x56131343e200 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14e5ba8740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x561313456950 con 0x561313443d60 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14e5ba8740 10 client.0 _ll_register_callbacks cb 0x7f14d800a810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14d3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343ebf0 0x56131344d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14d3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343ebf0 0x56131344d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60448/0 (socket says 172.21.15.60:60448) 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14d3fff640 1 -- 172.21.15.60:0/2050150830 learned_addr learned my addr 172.21.15.60:0/2050150830 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14e0c93640 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343e200 0x56131344cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14e1494640 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561313443d60 0x56131344e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14d3fff640 1 -- 172.21.15.60:0/2050150830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561313443d60 msgr2=0x56131344e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14d3fff640 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561313443d60 0x56131344e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14d3fff640 1 -- 172.21.15.60:0/2050150830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343e200 msgr2=0x56131344cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14d3fff640 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343e200 0x56131344cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14d3fff640 1 -- 172.21.15.60:0/2050150830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56131344ea30 con 0x56131343ebf0 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14e1494640 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561313443d60 0x56131344e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:06.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14d3fff640 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343ebf0 0x56131344d430 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009900 tx=0x7f14cc005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5becf8dad5d96532 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14bbfff640 1 -- 172.21.15.60:0/2050150830 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc005230 con 0x56131343ebf0 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14bbfff640 1 -- 172.21.15.60:0/2050150830 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f14cc0053d0 con 0x56131343ebf0 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.483+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2050150830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14d801c420 con 0x56131343ebf0 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.484+0000 7f14bbfff640 1 -- 172.21.15.60:0/2050150830 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14cc052050 con 0x56131343ebf0 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.484+0000 7f14e5ba8740 10 client.15243 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.484+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2050150830 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56131344fdd0 con 0x56131343ebf0 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.484+0000 7f14bbfff640 1 -- 172.21.15.60:0/2050150830 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14d801c420 con 0x56131343ebf0 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.484+0000 7f14bbfff640 1 -- 172.21.15.60:0/2050150830 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f14cc04d030 con 0x56131343ebf0 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.484+0000 7f14e5ba8740 10 client.15243 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.484+0000 7f14e5ba8740 10 client.15243 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.484+0000 7f14e5ba8740 20 client.15243 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.484+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x5613134b1280 0x5613134acea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.484+0000 7f14e5ba8740 4 client.15243 mds_command: new command op to 24394 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.484+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2050150830 --> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5613133a7050 con 0x5613134b1280 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.484+0000 7f14e0c93640 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x5613134b1280 0x5613134acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e0c93640 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x5613134b1280 0x5613134acea0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4f2831a1f305969f server_cookie=54e40921ba986781 in_seq=0 out_seq=0 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14bbfff640 10 client.15243 ms_handle_connect on v2:172.21.15.86:6832/777854669 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14bbfff640 1 -- 172.21.15.60:0/2050150830 <== mds.4 v2:172.21.15.86:6832/777854669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5613133a7050 con 0x5613134b1280 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14bbfff640 10 client.15243 handle_command_reply: tid=0 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e5ba8740 1 client.15243 shutdown 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2050150830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343ebf0 msgr2=0x56131344d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343ebf0 0x56131344d430 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009900 tx=0x7f14cc005e80 comp rx=0 tx=0).stop 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2050150830 shutdown_connections 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561313443d60 0x56131344e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56131343ebf0 0x56131344d430 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x5613134b1280 0x5613134acea0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e5ba8740 1 --2- 172.21.15.60:0/2050150830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56131343e200 0x56131344cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14bbfff640 0 client.15243 ms_handle_reset on v2:172.21.15.86:6832/777854669 2024-03-04T12:25:06.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2050150830 >> 172.21.15.60:0/2050150830 conn(0x561313435360 msgr2=0x56131343f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2050150830 shutdown_connections 2024-03-04T12:25:06.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e5ba8740 1 -- 172.21.15.60:0/2050150830 wait complete. 2024-03-04T12:25:06.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.485+0000 7f14e5ba8740 20 client.15243 trim_cache size 0 max 16384 2024-03-04T12:25:06.498 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:25:06.498 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:25:06.498 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 4, 2024-03-04T12:25:06.498 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 24394, 2024-03-04T12:25:06.498 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-03-04T12:25:06.498 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-03-04T12:25:06.499 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-03-04T12:25:06.499 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 85.593124610999993, 2024-03-04T12:25:06.499 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 28, 2024-03-04T12:25:06.499 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 92, 2024-03-04T12:25:06.499 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 89, 2024-03-04T12:25:06.499 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 151.88266149200001 2024-03-04T12:25:06.499 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:25:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.486+0000 7f14e2ef7640 1 -- 172.21.15.60:0/648298463 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f14c40a1d80 msgr2=0x7f14c40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.486+0000 7f14e2ef7640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f14c40a1d80 0x7f14c40a4240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f14dc1012a0 tx=0x7f14d800af80 comp rx=0 tx=0).stop 2024-03-04T12:25:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.486+0000 7f14e2ef7640 1 -- 172.21.15.60:0/648298463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14dc101e00 msgr2=0x7f14dc19ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.486+0000 7f14e2ef7640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14dc101e00 0x7f14dc19ebd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f14d400eaa0 tx=0x7f14d400ef60 comp rx=0 tx=0).stop 2024-03-04T12:25:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.486+0000 7f14e2ef7640 1 -- 172.21.15.60:0/648298463 shutdown_connections 2024-03-04T12:25:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.486+0000 7f14e2ef7640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f14c40a1d80 0x7f14c40a4240 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.486+0000 7f14e2ef7640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f14dc101e00 0x7f14dc19ebd0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.486+0000 7f14e2ef7640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14dc101440 0x7f14dc197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.486+0000 7f14e2ef7640 1 --2- 172.21.15.60:0/648298463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f14dc100250 0x7f14dc197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.486+0000 7f14e2ef7640 1 -- 172.21.15.60:0/648298463 >> 172.21.15.60:0/648298463 conn(0x7f14dc0fb7e0 msgr2=0x7f14dc0699d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.486+0000 7f14e2ef7640 1 -- 172.21.15.60:0/648298463 shutdown_connections 2024-03-04T12:25:06.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.486+0000 7f14e2ef7640 1 -- 172.21.15.60:0/648298463 wait complete. 2024-03-04T12:25:06.503 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-04T12:25:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.600+0000 7f5898efc640 1 -- 172.21.15.60:0/3353726423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5894100250 msgr2=0x7f5894100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.600+0000 7f5898efc640 1 --2- 172.21.15.60:0/3353726423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5894100250 0x7f5894100670 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5884009900 tx=0x7f588402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.600+0000 7f5898efc640 1 -- 172.21.15.60:0/3353726423 shutdown_connections 2024-03-04T12:25:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.600+0000 7f5898efc640 1 --2- 172.21.15.60:0/3353726423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5894101e00 0x7f5894108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.600+0000 7f5898efc640 1 --2- 172.21.15.60:0/3353726423 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5894101440 0x7f58941018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.600+0000 7f5898efc640 1 --2- 172.21.15.60:0/3353726423 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5894100250 0x7f5894100670 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.601+0000 7f5898efc640 1 -- 172.21.15.60:0/3353726423 >> 172.21.15.60:0/3353726423 conn(0x7f58940fb7c0 msgr2=0x7f58940fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.601+0000 7f5898efc640 1 -- 172.21.15.60:0/3353726423 shutdown_connections 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.601+0000 7f5898efc640 1 -- 172.21.15.60:0/3353726423 wait complete. 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.601+0000 7f5898efc640 1 Processor -- start 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.601+0000 7f5898efc640 1 -- start start 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.601+0000 7f5898efc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5894100250 0x7f5894197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.601+0000 7f5898efc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5894101440 0x7f5894197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.601+0000 7f5898efc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5894101e00 0x7f589419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.601+0000 7f5898efc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589410bee0 con 0x7f5894101e00 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.601+0000 7f5898efc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589410bd60 con 0x7f5894101440 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.601+0000 7f5898efc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589410c060 con 0x7f5894100250 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f5892d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5894101e00 0x7f589419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f589259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5894100250 0x7f5894197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f5892d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5894101e00 0x7f589419ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60472/0 (socket says 172.21.15.60:60472) 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f5892d9d640 1 -- 172.21.15.60:0/4066691190 learned_addr learned my addr 172.21.15.60:0/4066691190 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f588bfff640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5894101440 0x7f5894197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f5892d9d640 1 -- 172.21.15.60:0/4066691190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5894100250 msgr2=0x7f5894197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f5892d9d640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5894100250 0x7f5894197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f5892d9d640 1 -- 172.21.15.60:0/4066691190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5894101440 msgr2=0x7f5894197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f5892d9d640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5894101440 0x7f5894197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f5892d9d640 1 -- 172.21.15.60:0/4066691190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589419f330 con 0x7f5894101e00 2024-03-04T12:25:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f588bfff640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5894101440 0x7f5894197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f589259c640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5894100250 0x7f5894197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:25:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f5892d9d640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5894101e00 0x7f589419ecc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f588c00cce0 tx=0x7f588c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f588b7fe640 1 -- 172.21.15.60:0/4066691190 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c00f450 con 0x7f5894101e00 2024-03-04T12:25:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f5898efc640 1 -- 172.21.15.60:0/4066691190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f589419f620 con 0x7f5894101e00 2024-03-04T12:25:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f588b7fe640 1 -- 172.21.15.60:0/4066691190 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f588c00fad0 con 0x7f5894101e00 2024-03-04T12:25:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f588b7fe640 1 -- 172.21.15.60:0/4066691190 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f588c018c10 con 0x7f5894101e00 2024-03-04T12:25:06.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.602+0000 7f5898efc640 1 -- 172.21.15.60:0/4066691190 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f589419fb20 con 0x7f5894101e00 2024-03-04T12:25:06.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.604+0000 7f588b7fe640 1 -- 172.21.15.60:0/4066691190 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f588c00f5f0 con 0x7f5894101e00 2024-03-04T12:25:06.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f589bbb3740 1 Processor -- start 2024-03-04T12:25:06.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f589bbb3740 1 -- start start 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f589bbb3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24ab8df0 0x55bd24ab9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f589bbb3740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24abe2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f589bbb3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24abe950 0x55bd24ac0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f589bbb3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd24a1e7c0 con 0x55bd24ab8df0 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f589bbb3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd249c7b50 con 0x55bd24abe950 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f589bbb3740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd24a1a860 con 0x55bd24ab97e0 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f588bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24ab8df0 0x55bd24ab9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f588bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24ab8df0 0x55bd24ab9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60484/0 (socket says 172.21.15.60:60484) 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f588bfff640 1 -- 172.21.15.60:0/1909211978 learned_addr learned my addr 172.21.15.60:0/1909211978 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f588b7fe640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f58800a1d80 0x7f58800a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f589259c640 1 --2- 172.21.15.60:0/1909211978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24abe2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f5892d9d640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f58800a1d80 0x7f58800a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.605+0000 7f588bfff640 1 --2- 172.21.15.60:0/1909211978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24abe950 0x55bd24ac0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.606+0000 7f588b7fe640 1 -- 172.21.15.60:0/4066691190 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f588c0ce820 con 0x7f5894101e00 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.606+0000 7f588bfff640 1 -- 172.21.15.60:0/1909211978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 msgr2=0x55bd24abe2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.606+0000 7f588bfff640 1 --2- 172.21.15.60:0/1909211978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24abe2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.606+0000 7f588bfff640 1 -- 172.21.15.60:0/1909211978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24abe950 msgr2=0x55bd24ac0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.606+0000 7f588bfff640 1 --2- 172.21.15.60:0/1909211978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24abe950 0x55bd24ac0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.606+0000 7f588bfff640 1 -- 172.21.15.60:0/1909211978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd249db9a0 con 0x55bd24ab8df0 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.606+0000 7f589259c640 1 --2- 172.21.15.60:0/1909211978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24abe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.606+0000 7f588bfff640 1 --2- 172.21.15.60:0/1909211978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24ab8df0 0x55bd24ab9210 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f587c00ceb0 tx=0x7f587c00c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b25c17aeaebfe815 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.606+0000 7f5892d9d640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f58800a1d80 0x7f58800a4240 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f588c09b150 tx=0x7f588c06e280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.606+0000 7f5888ff9640 1 -- 172.21.15.60:0/1909211978 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c01d070 con 0x55bd24ab8df0 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.606+0000 7f5888ff9640 1 -- 172.21.15.60:0/1909211978 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f587c010e40 con 0x55bd24ab8df0 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.606+0000 7f5888ff9640 1 -- 172.21.15.60:0/1909211978 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c0038a0 con 0x55bd24ab8df0 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 -- 172.21.15.60:0/1909211978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24ab8df0 msgr2=0x55bd24ab9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 --2- 172.21.15.60:0/1909211978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24ab8df0 0x55bd24ab9210 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f587c00ceb0 tx=0x7f587c00c640 comp rx=0 tx=0).stop 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 -- 172.21.15.60:0/1909211978 shutdown_connections 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 --2- 172.21.15.60:0/1909211978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24abe950 0x55bd24ac0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 --2- 172.21.15.60:0/1909211978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24abe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 --2- 172.21.15.60:0/1909211978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24ab8df0 0x55bd24ab9210 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 -- 172.21.15.60:0/1909211978 >> 172.21.15.60:0/1909211978 conn(0x55bd24ab4360 msgr2=0x55bd24ab4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 -- 172.21.15.60:0/1909211978 shutdown_connections 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 -- 172.21.15.60:0/1909211978 wait complete. 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 -- start start 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 Processor -- start 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 0x55bd24ac3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24ac3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24aca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd24ac1900 con 0x55bd24abe950 2024-03-04T12:25:06.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd249daf90 con 0x55bd24ab8df0 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd24ac1a80 con 0x55bd24ab97e0 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.607+0000 7f589bbb3740 10 client.0 _ll_register_callbacks cb 0x55bd249fcd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f588bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24aca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f588bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24aca7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60498/0 (socket says 172.21.15.60:60498) 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f588bfff640 1 -- 172.21.15.60:0/3191309815 learned_addr learned my addr 172.21.15.60:0/3191309815 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f589259c640 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24ac3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f588bfff640 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 0x55bd24ac3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f588bfff640 1 -- 172.21.15.60:0/3191309815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 msgr2=0x55bd24ac3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f588bfff640 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24ac3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f588bfff640 1 -- 172.21.15.60:0/3191309815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 msgr2=0x55bd24ac3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f588bfff640 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 0x55bd24ac3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f588bfff640 1 -- 172.21.15.60:0/3191309815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bd247332d0 con 0x55bd24abe950 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f588bfff640 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24aca7c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f587c010bd0 tx=0x7f587c012cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f65035ae3f2ad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f5879ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:06.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f5879ffb640 1 -- 172.21.15.60:0/3191309815 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c01d070 con 0x55bd24abe950 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f5879ffb640 1 -- 172.21.15.60:0/3191309815 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f587c012660 con 0x55bd24abe950 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f589bbb3740 1 -- 172.21.15.60:0/3191309815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bd24a2bd00 con 0x55bd24abe950 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f5879ffb640 1 -- 172.21.15.60:0/3191309815 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f587c012860 con 0x55bd24abe950 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f589bbb3740 10 client.15255 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f589bbb3740 1 -- 172.21.15.60:0/3191309815 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bd247fd6b0 con 0x55bd24abe950 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.608+0000 7f5879ffb640 1 -- 172.21.15.60:0/3191309815 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55bd24a2bd00 con 0x55bd24abe950 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.609+0000 7f5879ffb640 1 -- 172.21.15.60:0/3191309815 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f587c022470 con 0x55bd24abe950 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.609+0000 7f589bbb3740 10 client.15255 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.609+0000 7f589bbb3740 10 client.15255 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.609+0000 7f589bbb3740 20 client.15255 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.609+0000 7f589bbb3740 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bd24acfac0 0x55bd24aced60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.609+0000 7f589bbb3740 4 client.15255 mds_command: new command op to 24407 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.609+0000 7f589bbb3740 1 -- 172.21.15.60:0/3191309815 --> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bd24a26050 con 0x55bd24acfac0 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.609+0000 7f589259c640 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bd24acfac0 0x55bd24aced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.609+0000 7f5879ffb640 10 client.15255 ms_handle_connect on v2:172.21.15.63:6832/2270964219 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.609+0000 7f589259c640 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bd24acfac0 0x55bd24aced60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ceba6269223f91e4 server_cookie=cf5a547aa95a8017 in_seq=0 out_seq=0 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.610+0000 7f5879ffb640 1 -- 172.21.15.60:0/3191309815 <== mds.3 v2:172.21.15.63:6832/2270964219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55bd24a26050 con 0x55bd24acfac0 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.610+0000 7f5879ffb640 10 client.15255 handle_command_reply: tid=0 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.610+0000 7f589bbb3740 1 client.15255 shutdown 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.610+0000 7f589bbb3740 1 -- 172.21.15.60:0/3191309815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 msgr2=0x55bd24aca7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.610+0000 7f589bbb3740 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24aca7c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f587c010bd0 tx=0x7f587c012cc0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.610+0000 7f589bbb3740 1 -- 172.21.15.60:0/3191309815 shutdown_connections 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.610+0000 7f589bbb3740 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bd24acfac0 0x55bd24aced60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.610+0000 7f589bbb3740 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24aca7c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.610+0000 7f589bbb3740 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24ac3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.610+0000 7f5879ffb640 0 client.15255 ms_handle_reset on v2:172.21.15.63:6832/2270964219 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.610+0000 7f589bbb3740 1 --2- 172.21.15.60:0/3191309815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 0x55bd24ac3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.611+0000 7f589bbb3740 1 -- 172.21.15.60:0/3191309815 >> 172.21.15.60:0/3191309815 conn(0x55bd24ab4360 msgr2=0x55bd24ab3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.611+0000 7f589bbb3740 1 -- 172.21.15.60:0/3191309815 shutdown_connections 2024-03-04T12:25:06.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.611+0000 7f589bbb3740 1 -- 172.21.15.60:0/3191309815 wait complete. 2024-03-04T12:25:06.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.611+0000 7f589bbb3740 20 client.15255 trim_cache size 0 max 16384 2024-03-04T12:25:06.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.626+0000 7f589bbb3740 1 Processor -- start 2024-03-04T12:25:06.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.626+0000 7f589bbb3740 1 -- start start 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.626+0000 7f589bbb3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 0x55bd24ac2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.626+0000 7f589bbb3740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24ac1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.626+0000 7f589bbb3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24aca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.626+0000 7f589bbb3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd24acf2e0 con 0x55bd24abe950 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.626+0000 7f589bbb3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd24acf940 con 0x55bd24ab8df0 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.626+0000 7f589bbb3740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd24acffa0 con 0x55bd24ab97e0 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.626+0000 7f588bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24aca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.626+0000 7f588bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24aca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60508/0 (socket says 172.21.15.60:60508) 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.626+0000 7f588bfff640 1 -- 172.21.15.60:0/3646447922 learned_addr learned my addr 172.21.15.60:0/3646447922 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f588bfff640 1 --2- 172.21.15.60:0/3646447922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 0x55bd24ac2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f589259c640 1 --2- 172.21.15.60:0/3646447922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24ac1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f588bfff640 1 -- 172.21.15.60:0/3646447922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 msgr2=0x55bd24ac1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f588bfff640 1 --2- 172.21.15.60:0/3646447922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24ac1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f588bfff640 1 -- 172.21.15.60:0/3646447922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 msgr2=0x55bd24ac2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f588bfff640 1 --2- 172.21.15.60:0/3646447922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 0x55bd24ac2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f588bfff640 1 -- 172.21.15.60:0/3646447922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd24ab9bc0 con 0x55bd24abe950 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f588bfff640 1 --2- 172.21.15.60:0/3646447922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24aca6e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f587c00ceb0 tx=0x7f587c005a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0b2fe67ec12859 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f587bfff640 1 -- 172.21.15.60:0/3646447922 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c01d030 con 0x55bd24abe950 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f587bfff640 1 -- 172.21.15.60:0/3646447922 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f587c012d30 con 0x55bd24abe950 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f587bfff640 1 -- 172.21.15.60:0/3646447922 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c005670 con 0x55bd24abe950 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f589bbb3740 1 -- 172.21.15.60:0/3646447922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 msgr2=0x55bd24aca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f589bbb3740 1 --2- 172.21.15.60:0/3646447922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24aca6e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f587c00ceb0 tx=0x7f587c005a10 comp rx=0 tx=0).stop 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f589bbb3740 1 -- 172.21.15.60:0/3646447922 shutdown_connections 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f589bbb3740 1 --2- 172.21.15.60:0/3646447922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24aca6e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f589bbb3740 1 --2- 172.21.15.60:0/3646447922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24ac1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f589bbb3740 1 --2- 172.21.15.60:0/3646447922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 0x55bd24ac2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.627+0000 7f589bbb3740 1 -- 172.21.15.60:0/3646447922 >> 172.21.15.60:0/3646447922 conn(0x55bd24ab4360 msgr2=0x55bd24ab4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f589bbb3740 1 -- 172.21.15.60:0/3646447922 shutdown_connections 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f589bbb3740 1 -- 172.21.15.60:0/3646447922 wait complete. 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f589bbb3740 1 -- start start 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f589bbb3740 1 Processor -- start 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f589bbb3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 0x55bd24aca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f589bbb3740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24ac6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f589bbb3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24ac73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f589bbb3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd24ad0c60 con 0x55bd24abe950 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f589bbb3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd24ad0600 con 0x55bd24ab8df0 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f589bbb3740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd24ad12c0 con 0x55bd24ab97e0 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f589bbb3740 10 client.0 _ll_register_callbacks cb 0x7f587c00c4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f588bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24ac73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f588bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24ac73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60524/0 (socket says 172.21.15.60:60524) 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f588bfff640 1 -- 172.21.15.60:0/1890932872 learned_addr learned my addr 172.21.15.60:0/1890932872 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f589259c640 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24ac6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.628+0000 7f588bfff640 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 0x55bd24aca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f588bfff640 1 -- 172.21.15.60:0/1890932872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 msgr2=0x55bd24ac6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f588bfff640 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24ac6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f588bfff640 1 -- 172.21.15.60:0/1890932872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 msgr2=0x55bd24aca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f588bfff640 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 0x55bd24aca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f588bfff640 1 -- 172.21.15.60:0/1890932872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bd24acaa00 con 0x55bd24abe950 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f588bfff640 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24ac73b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f587c010cc0 tx=0x7f587c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eddaa95d25e351ff server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f5879ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f5879ffb640 1 -- 172.21.15.60:0/1890932872 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587c043040 con 0x55bd24abe950 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f5879ffb640 1 -- 172.21.15.60:0/1890932872 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f587c005650 con 0x55bd24abe950 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f589bbb3740 1 -- 172.21.15.60:0/1890932872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f587c012d30 con 0x55bd24abe950 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f5879ffb640 1 -- 172.21.15.60:0/1890932872 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f587c01d040 con 0x55bd24abe950 2024-03-04T12:25:06.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f589bbb3740 10 client.15261 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f589bbb3740 1 -- 172.21.15.60:0/1890932872 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bd24b265c0 con 0x55bd24abe950 2024-03-04T12:25:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f5879ffb640 1 -- 172.21.15.60:0/1890932872 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f587c012d30 con 0x55bd24abe950 2024-03-04T12:25:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f5879ffb640 1 -- 172.21.15.60:0/1890932872 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f587c049020 con 0x55bd24abe950 2024-03-04T12:25:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f589bbb3740 10 client.15261 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f589bbb3740 10 client.15261 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-04T12:25:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f589bbb3740 20 client.15261 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f589bbb3740 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bd24b2a880 0x55bd24b26850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f589bbb3740 4 client.15261 mds_command: new command op to 24407 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T12:25:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.629+0000 7f589bbb3740 1 -- 172.21.15.60:0/1890932872 --> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bd249a49a0 con 0x55bd24b2a880 2024-03-04T12:25:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f589259c640 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bd24b2a880 0x55bd24b26850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f589259c640 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bd24b2a880 0x55bd24b26850 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=aef5c6add7b938f4 server_cookie=88291495616acaed in_seq=0 out_seq=0 2024-03-04T12:25:06.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f5879ffb640 10 client.15261 ms_handle_connect on v2:172.21.15.63:6832/2270964219 2024-03-04T12:25:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f5879ffb640 1 -- 172.21.15.60:0/1890932872 <== mds.3 v2:172.21.15.63:6832/2270964219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55bd249a49a0 con 0x55bd24b2a880 2024-03-04T12:25:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f5879ffb640 10 client.15261 handle_command_reply: tid=0 2024-03-04T12:25:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f589bbb3740 1 client.15261 shutdown 2024-03-04T12:25:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f589bbb3740 1 -- 172.21.15.60:0/1890932872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 msgr2=0x55bd24ac73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f589bbb3740 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24ac73b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f587c010cc0 tx=0x7f587c000bc0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f589bbb3740 1 -- 172.21.15.60:0/1890932872 shutdown_connections 2024-03-04T12:25:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f589bbb3740 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bd24b2a880 0x55bd24b26850 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f589bbb3740 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bd24abe950 0x55bd24ac73b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f5879ffb640 0 client.15261 ms_handle_reset on v2:172.21.15.63:6832/2270964219 2024-03-04T12:25:06.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f589bbb3740 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bd24ab97e0 0x55bd24ac6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f589bbb3740 1 --2- 172.21.15.60:0/1890932872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd24ab8df0 0x55bd24aca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f589bbb3740 1 -- 172.21.15.60:0/1890932872 >> 172.21.15.60:0/1890932872 conn(0x55bd24ab4360 msgr2=0x55bd24ab4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.630+0000 7f589bbb3740 1 -- 172.21.15.60:0/1890932872 shutdown_connections 2024-03-04T12:25:06.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f589bbb3740 1 -- 172.21.15.60:0/1890932872 wait complete. 2024-03-04T12:25:06.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f589bbb3740 20 client.15261 trim_cache size 0 max 16384 2024-03-04T12:25:06.644 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:25:06.645 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:25:06.645 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 3, 2024-03-04T12:25:06.645 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 24407, 2024-03-04T12:25:06.645 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-03-04T12:25:06.645 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-03-04T12:25:06.645 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-03-04T12:25:06.645 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 86.744114936000003, 2024-03-04T12:25:06.645 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 28, 2024-03-04T12:25:06.645 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 92, 2024-03-04T12:25:06.645 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 89, 2024-03-04T12:25:06.645 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 149.01076460300001 2024-03-04T12:25:06.645 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:25:06.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f5898efc640 1 -- 172.21.15.60:0/4066691190 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f58800a1d80 msgr2=0x7f58800a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f5898efc640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f58800a1d80 0x7f58800a4240 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f588c09b150 tx=0x7f588c06e280 comp rx=0 tx=0).stop 2024-03-04T12:25:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f5898efc640 1 -- 172.21.15.60:0/4066691190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5894101e00 msgr2=0x7f589419ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f5898efc640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5894101e00 0x7f589419ecc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f588c00cce0 tx=0x7f588c00a620 comp rx=0 tx=0).stop 2024-03-04T12:25:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f5898efc640 1 -- 172.21.15.60:0/4066691190 shutdown_connections 2024-03-04T12:25:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f5898efc640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f58800a1d80 0x7f58800a4240 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f5898efc640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5894101e00 0x7f589419ecc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f5898efc640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5894101440 0x7f5894197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f5898efc640 1 --2- 172.21.15.60:0/4066691190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5894100250 0x7f5894197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f5898efc640 1 -- 172.21.15.60:0/4066691190 >> 172.21.15.60:0/4066691190 conn(0x7f58940fb7c0 msgr2=0x7f58940fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f5898efc640 1 -- 172.21.15.60:0/4066691190 shutdown_connections 2024-03-04T12:25:06.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.631+0000 7f5898efc640 1 -- 172.21.15.60:0/4066691190 wait complete. 2024-03-04T12:25:06.649 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-04T12:25:06.754 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.748+0000 7facd3812640 1 -- 172.21.15.60:0/984076452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faccc102530 msgr2=0x7faccc1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.748+0000 7facd3812640 1 --2- 172.21.15.60:0/984076452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faccc102530 0x7faccc1029b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7facbc009900 tx=0x7facbc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:06.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.748+0000 7facd3812640 1 -- 172.21.15.60:0/984076452 shutdown_connections 2024-03-04T12:25:06.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.748+0000 7facd3812640 1 --2- 172.21.15.60:0/984076452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faccc102ef0 0x7faccc109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.748+0000 7facd3812640 1 --2- 172.21.15.60:0/984076452 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faccc102530 0x7faccc1029b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.748+0000 7facd3812640 1 --2- 172.21.15.60:0/984076452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faccc101340 0x7faccc101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.748+0000 7facd3812640 1 -- 172.21.15.60:0/984076452 >> 172.21.15.60:0/984076452 conn(0x7faccc0fc8b0 msgr2=0x7faccc0fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.748+0000 7facd3812640 1 -- 172.21.15.60:0/984076452 shutdown_connections 2024-03-04T12:25:06.755 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.748+0000 7facd3812640 1 -- 172.21.15.60:0/984076452 wait complete. 2024-03-04T12:25:06.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd3812640 1 Processor -- start 2024-03-04T12:25:06.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd3812640 1 -- start start 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd3812640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faccc101340 0x7faccc197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd3812640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faccc102530 0x7faccc197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd3812640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faccc102ef0 0x7faccc19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd3812640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc10cfd0 con 0x7faccc102ef0 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd3812640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc10ce50 con 0x7faccc102530 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd3812640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faccc10d150 con 0x7faccc101340 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd1daf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faccc102ef0 0x7faccc19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd1daf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faccc102ef0 0x7faccc19ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60548/0 (socket says 172.21.15.60:60548) 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd1daf640 1 -- 172.21.15.60:0/143440122 learned_addr learned my addr 172.21.15.60:0/143440122 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd15ae640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faccc101340 0x7faccc197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd0dad640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faccc102530 0x7faccc197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd1daf640 1 -- 172.21.15.60:0/143440122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faccc101340 msgr2=0x7faccc197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd1daf640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faccc101340 0x7faccc197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd1daf640 1 -- 172.21.15.60:0/143440122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faccc102530 msgr2=0x7faccc197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd1daf640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faccc102530 0x7faccc197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd1daf640 1 -- 172.21.15.60:0/143440122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faccc19f2a0 con 0x7faccc102ef0 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd0dad640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faccc102530 0x7faccc197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.749+0000 7facd1daf640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faccc102ef0 0x7faccc19ec30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7facc400cad0 tx=0x7facc400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.750+0000 7facc27fc640 1 -- 172.21.15.60:0/143440122 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc400ae20 con 0x7faccc102ef0 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.750+0000 7facd3812640 1 -- 172.21.15.60:0/143440122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faccc19f590 con 0x7faccc102ef0 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.750+0000 7facc27fc640 1 -- 172.21.15.60:0/143440122 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facc4005020 con 0x7faccc102ef0 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.750+0000 7facc27fc640 1 -- 172.21.15.60:0/143440122 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc400f6f0 con 0x7faccc102ef0 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.750+0000 7facd3812640 1 -- 172.21.15.60:0/143440122 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faccc19fa60 con 0x7faccc102ef0 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.752+0000 7facc27fc640 1 -- 172.21.15.60:0/143440122 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7facc40051d0 con 0x7faccc102ef0 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.752+0000 7facd63fc740 1 Processor -- start 2024-03-04T12:25:06.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.752+0000 7facd63fc740 1 -- start start 2024-03-04T12:25:06.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.752+0000 7facd63fc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33a2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.752+0000 7facd63fc740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a27e0 0x5586c33a72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.752+0000 7facd63fc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a7950 0x5586c33a9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.752+0000 7facd63fc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586c33077c0 con 0x5586c33a1df0 2024-03-04T12:25:06.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.752+0000 7facd63fc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586c32b0b50 con 0x5586c33a7950 2024-03-04T12:25:06.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.752+0000 7facd63fc740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586c3303860 con 0x5586c33a27e0 2024-03-04T12:25:06.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.752+0000 7facd0dad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.752+0000 7facc27fc640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7facb80a1d80 0x7facb80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.752+0000 7facd0dad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33a2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60554/0 (socket says 172.21.15.60:60554) 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facd0dad640 1 -- 172.21.15.60:0/1192331967 learned_addr learned my addr 172.21.15.60:0/1192331967 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facd1daf640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7facb80a1d80 0x7facb80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facd15ae640 1 --2- 172.21.15.60:0/1192331967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a27e0 0x5586c33a72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facd0dad640 1 --2- 172.21.15.60:0/1192331967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a7950 0x5586c33a9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facc27fc640 1 -- 172.21.15.60:0/143440122 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7facc40ce190 con 0x7faccc102ef0 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facd0dad640 1 -- 172.21.15.60:0/1192331967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a27e0 msgr2=0x5586c33a72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facd0dad640 1 --2- 172.21.15.60:0/1192331967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a27e0 0x5586c33a72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facd0dad640 1 -- 172.21.15.60:0/1192331967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a7950 msgr2=0x5586c33a9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facd0dad640 1 --2- 172.21.15.60:0/1192331967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a7950 0x5586c33a9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facd0dad640 1 -- 172.21.15.60:0/1192331967 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586c32c49a0 con 0x5586c33a1df0 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facd15ae640 1 --2- 172.21.15.60:0/1192331967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a27e0 0x5586c33a72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facd0dad640 1 --2- 172.21.15.60:0/1192331967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33a2210 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7facbc02cfd0 tx=0x7facbc0058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7180b9e6ae586ccc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facd1daf640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7facb80a1d80 0x7facb80a4240 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7facc409a830 tx=0x7facc4065c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facb77fe640 1 -- 172.21.15.60:0/1192331967 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc004230 con 0x5586c33a1df0 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facb77fe640 1 -- 172.21.15.60:0/1192331967 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facbc004ba0 con 0x5586c33a1df0 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.753+0000 7facb77fe640 1 -- 172.21.15.60:0/1192331967 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc0055b0 con 0x5586c33a1df0 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.754+0000 7facd63fc740 1 -- 172.21.15.60:0/1192331967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 msgr2=0x5586c33a2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.754+0000 7facd63fc740 1 --2- 172.21.15.60:0/1192331967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33a2210 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7facbc02cfd0 tx=0x7facbc0058b0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.754+0000 7facd63fc740 1 -- 172.21.15.60:0/1192331967 shutdown_connections 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.754+0000 7facd63fc740 1 --2- 172.21.15.60:0/1192331967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a7950 0x5586c33a9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.754+0000 7facd63fc740 1 --2- 172.21.15.60:0/1192331967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a27e0 0x5586c33a72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.754+0000 7facd63fc740 1 --2- 172.21.15.60:0/1192331967 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33a2210 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.754+0000 7facd63fc740 1 -- 172.21.15.60:0/1192331967 >> 172.21.15.60:0/1192331967 conn(0x5586c339d360 msgr2=0x5586c339d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.754+0000 7facd63fc740 1 -- 172.21.15.60:0/1192331967 shutdown_connections 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.754+0000 7facd63fc740 1 -- 172.21.15.60:0/1192331967 wait complete. 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.754+0000 7facd63fc740 1 -- start start 2024-03-04T12:25:06.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.754+0000 7facd63fc740 1 Processor -- start 2024-03-04T12:25:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd63fc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33ac210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd63fc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a27e0 0x5586c33ac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd63fc740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a7950 0x5586c33b3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd63fc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586c33aa840 con 0x5586c33a1df0 2024-03-04T12:25:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd63fc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586c32c3f90 con 0x5586c33a27e0 2024-03-04T12:25:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd63fc740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586c33aa9c0 con 0x5586c33a7950 2024-03-04T12:25:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd63fc740 10 client.0 _ll_register_callbacks cb 0x5586c32e5d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd0dad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd0dad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33ac210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60556/0 (socket says 172.21.15.60:60556) 2024-03-04T12:25:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd0dad640 1 -- 172.21.15.60:0/3290845840 learned_addr learned my addr 172.21.15.60:0/3290845840 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd15ae640 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a27e0 0x5586c33ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd0dad640 1 -- 172.21.15.60:0/3290845840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a7950 msgr2=0x5586c33b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd0dad640 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a7950 0x5586c33b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd0dad640 1 -- 172.21.15.60:0/3290845840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a27e0 msgr2=0x5586c33ac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd0dad640 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a27e0 0x5586c33ac750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd0dad640 1 -- 172.21.15.60:0/3290845840 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5586c301c2d0 con 0x5586c33a1df0 2024-03-04T12:25:06.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd15ae640 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a27e0 0x5586c33ac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facd0dad640 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33ac210 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7facbc02c170 tx=0x7facbc004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb31e4b6a4788f1b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.755+0000 7facb4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facb4ff9640 1 -- 172.21.15.60:0/3290845840 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc041660 con 0x5586c33a1df0 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facb4ff9640 1 -- 172.21.15.60:0/3290845840 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7facbc041800 con 0x5586c33a1df0 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facd63fc740 1 -- 172.21.15.60:0/3290845840 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7facbc004ba0 con 0x5586c33a1df0 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facb4ff9640 1 -- 172.21.15.60:0/3290845840 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7facbc041a00 con 0x5586c33a1df0 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facd63fc740 10 client.15273 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facd63fc740 1 -- 172.21.15.60:0/3290845840 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5586c30e66b0 con 0x5586c33a1df0 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facb4ff9640 1 -- 172.21.15.60:0/3290845840 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facbc004ba0 con 0x5586c33a1df0 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facb4ff9640 1 -- 172.21.15.60:0/3290845840 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7facbc043ba0 con 0x5586c33a1df0 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facd63fc740 10 client.15273 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facd63fc740 10 client.15273 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facd63fc740 20 client.15273 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:06.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facd63fc740 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x5586c33b7c80 0x5586c33bcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facd63fc740 4 client.15273 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.756+0000 7facd63fc740 1 -- 172.21.15.60:0/3290845840 --> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5586c3314d00 con 0x5586c33b7c80 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.757+0000 7facd15ae640 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x5586c33b7c80 0x5586c33bcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.757+0000 7facb4ff9640 10 client.15273 ms_handle_connect on v2:172.21.15.63:6836/3974574804 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.757+0000 7facd15ae640 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x5586c33b7c80 0x5586c33bcd50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bf40abfce0b6ff80 server_cookie=d3a3589643900895 in_seq=0 out_seq=0 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facb4ff9640 1 -- 172.21.15.60:0/3290845840 <== mds.2 v2:172.21.15.63:6836/3974574804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5586c3314d00 con 0x5586c33b7c80 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facb4ff9640 10 client.15273 handle_command_reply: tid=0 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facd63fc740 1 client.15273 shutdown 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facd63fc740 1 -- 172.21.15.60:0/3290845840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 msgr2=0x5586c33ac210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facd63fc740 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33ac210 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7facbc02c170 tx=0x7facbc004d50 comp rx=0 tx=0).stop 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facd63fc740 1 -- 172.21.15.60:0/3290845840 shutdown_connections 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facd63fc740 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x5586c33b7c80 0x5586c33bcd50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facd63fc740 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a7950 0x5586c33b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facd63fc740 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a27e0 0x5586c33ac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facb4ff9640 0 client.15273 ms_handle_reset on v2:172.21.15.63:6836/3974574804 2024-03-04T12:25:06.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facd63fc740 1 --2- 172.21.15.60:0/3290845840 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33ac210 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facd63fc740 1 -- 172.21.15.60:0/3290845840 >> 172.21.15.60:0/3290845840 conn(0x5586c339d360 msgr2=0x5586c339c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facd63fc740 1 -- 172.21.15.60:0/3290845840 shutdown_connections 2024-03-04T12:25:06.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facd63fc740 1 -- 172.21.15.60:0/3290845840 wait complete. 2024-03-04T12:25:06.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.758+0000 7facd63fc740 20 client.15273 trim_cache size 0 max 16384 2024-03-04T12:25:06.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd63fc740 1 Processor -- start 2024-03-04T12:25:06.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd63fc740 1 -- start start 2024-03-04T12:25:06.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd63fc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33a9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd63fc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a27e0 0x5586c33c1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd63fc740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a7950 0x5586c339bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd63fc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586c33b80f0 con 0x5586c33a1df0 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd63fc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586c33b8750 con 0x5586c33a27e0 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd63fc740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586c33b8db0 con 0x5586c33a7950 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd0dad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33a9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd0dad640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33a9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60564/0 (socket says 172.21.15.60:60564) 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd0dad640 1 -- 172.21.15.60:0/295707661 learned_addr learned my addr 172.21.15.60:0/295707661 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd0dad640 1 -- 172.21.15.60:0/295707661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a7950 msgr2=0x5586c339bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd0dad640 1 --2- 172.21.15.60:0/295707661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a7950 0x5586c339bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd0dad640 1 -- 172.21.15.60:0/295707661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a27e0 msgr2=0x5586c33c1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd0dad640 1 --2- 172.21.15.60:0/295707661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a27e0 0x5586c33c1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.774+0000 7facd0dad640 1 -- 172.21.15.60:0/295707661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586c3398dc0 con 0x5586c33a1df0 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facd0dad640 1 --2- 172.21.15.60:0/295707661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33a9d40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7facbc02cfd0 tx=0x7facbc02cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7520e069ae58dd08 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facb6ffd640 1 -- 172.21.15.60:0/295707661 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc062070 con 0x5586c33a1df0 2024-03-04T12:25:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facb6ffd640 1 -- 172.21.15.60:0/295707661 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facbc050070 con 0x5586c33a1df0 2024-03-04T12:25:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facb6ffd640 1 -- 172.21.15.60:0/295707661 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc041ba0 con 0x5586c33a1df0 2024-03-04T12:25:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facd63fc740 1 -- 172.21.15.60:0/295707661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 msgr2=0x5586c33a9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facd63fc740 1 --2- 172.21.15.60:0/295707661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33a9d40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7facbc02cfd0 tx=0x7facbc02cb60 comp rx=0 tx=0).stop 2024-03-04T12:25:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facd63fc740 1 -- 172.21.15.60:0/295707661 shutdown_connections 2024-03-04T12:25:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facd63fc740 1 --2- 172.21.15.60:0/295707661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a7950 0x5586c339bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facd63fc740 1 --2- 172.21.15.60:0/295707661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a27e0 0x5586c33c1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facd63fc740 1 --2- 172.21.15.60:0/295707661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a1df0 0x5586c33a9d40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facd63fc740 1 -- 172.21.15.60:0/295707661 >> 172.21.15.60:0/295707661 conn(0x5586c339d360 msgr2=0x5586c33c0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facd63fc740 1 -- 172.21.15.60:0/295707661 shutdown_connections 2024-03-04T12:25:06.785 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.775+0000 7facd63fc740 1 -- 172.21.15.60:0/295707661 wait complete. 2024-03-04T12:25:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd63fc740 1 -- start start 2024-03-04T12:25:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd63fc740 1 Processor -- start 2024-03-04T12:25:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd63fc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a1df0 0x5586c340b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd63fc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a27e0 0x5586c340ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd63fc740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a7950 0x5586c3412a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd63fc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586c33b9a70 con 0x5586c33a27e0 2024-03-04T12:25:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd63fc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586c33b9410 con 0x5586c33a1df0 2024-03-04T12:25:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd63fc740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586c33ba0d0 con 0x5586c33a7950 2024-03-04T12:25:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd63fc740 10 client.0 _ll_register_callbacks cb 0x7facbc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd15ae640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a27e0 0x5586c340ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd15ae640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a27e0 0x5586c340ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60578/0 (socket says 172.21.15.60:60578) 2024-03-04T12:25:06.786 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd15ae640 1 -- 172.21.15.60:0/4196685598 learned_addr learned my addr 172.21.15.60:0/4196685598 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.776+0000 7facd15ae640 1 -- 172.21.15.60:0/4196685598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a7950 msgr2=0x5586c3412a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.789 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd15ae640 1 --2- 172.21.15.60:0/4196685598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a7950 0x5586c3412a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd15ae640 1 -- 172.21.15.60:0/4196685598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a1df0 msgr2=0x5586c340b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd0dad640 1 --2- 172.21.15.60:0/4196685598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a1df0 0x5586c340b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd15ae640 1 --2- 172.21.15.60:0/4196685598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a1df0 0x5586c340b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd15ae640 1 -- 172.21.15.60:0/4196685598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5586c3413050 con 0x5586c33a27e0 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd15ae640 1 --2- 172.21.15.60:0/4196685598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a27e0 0x5586c340ba20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7facc800abc0 tx=0x7facc8013b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae59e4f26659072 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facb4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facb4ff9640 1 -- 172.21.15.60:0/4196685598 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8004030 con 0x5586c33a27e0 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd63fc740 1 -- 172.21.15.60:0/4196685598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7facbc050070 con 0x5586c33a27e0 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facb4ff9640 1 -- 172.21.15.60:0/4196685598 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7facc80041d0 con 0x5586c33a27e0 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facb4ff9640 1 -- 172.21.15.60:0/4196685598 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7facc802a050 con 0x5586c33a27e0 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd63fc740 10 client.15279 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd63fc740 1 -- 172.21.15.60:0/4196685598 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5586c3414450 con 0x5586c33a27e0 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facb4ff9640 1 -- 172.21.15.60:0/4196685598 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facbc050070 con 0x5586c33a27e0 2024-03-04T12:25:06.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facb4ff9640 1 -- 172.21.15.60:0/4196685598 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7facc800fb70 con 0x5586c33a27e0 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd63fc740 10 client.15279 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd63fc740 10 client.15279 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd63fc740 20 client.15279 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd63fc740 1 --2- 172.21.15.60:0/4196685598 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x5586c3418b70 0x5586c3414740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd63fc740 4 client.15279 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.777+0000 7facd63fc740 1 -- 172.21.15.60:0/4196685598 --> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5586c330f050 con 0x5586c3418b70 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facd0dad640 1 --2- 172.21.15.60:0/4196685598 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x5586c3418b70 0x5586c3414740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facb4ff9640 10 client.15279 ms_handle_connect on v2:172.21.15.63:6836/3974574804 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facd0dad640 1 --2- 172.21.15.60:0/4196685598 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x5586c3418b70 0x5586c3414740 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d5b75d6f9bd47c70 server_cookie=ac11499315a83b2e in_seq=0 out_seq=0 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facb4ff9640 1 -- 172.21.15.60:0/4196685598 <== mds.2 v2:172.21.15.63:6836/3974574804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5586c330f050 con 0x5586c3418b70 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facb4ff9640 10 client.15279 handle_command_reply: tid=0 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facd63fc740 1 client.15279 shutdown 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facd63fc740 1 -- 172.21.15.60:0/4196685598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a27e0 msgr2=0x5586c340ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facd63fc740 1 --2- 172.21.15.60:0/4196685598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a27e0 0x5586c340ba20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7facc800abc0 tx=0x7facc8013b20 comp rx=0 tx=0).stop 2024-03-04T12:25:06.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facd63fc740 1 -- 172.21.15.60:0/4196685598 shutdown_connections 2024-03-04T12:25:06.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facd63fc740 1 --2- 172.21.15.60:0/4196685598 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x5586c3418b70 0x5586c3414740 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facd63fc740 1 --2- 172.21.15.60:0/4196685598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5586c33a7950 0x5586c3412a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facb4ff9640 0 client.15279 ms_handle_reset on v2:172.21.15.63:6836/3974574804 2024-03-04T12:25:06.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facd63fc740 1 --2- 172.21.15.60:0/4196685598 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5586c33a27e0 0x5586c340ba20 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facd63fc740 1 --2- 172.21.15.60:0/4196685598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586c33a1df0 0x5586c340b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.778+0000 7facd63fc740 1 -- 172.21.15.60:0/4196685598 >> 172.21.15.60:0/4196685598 conn(0x5586c339d360 msgr2=0x5586c33c1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd63fc740 1 -- 172.21.15.60:0/4196685598 shutdown_connections 2024-03-04T12:25:06.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd63fc740 1 -- 172.21.15.60:0/4196685598 wait complete. 2024-03-04T12:25:06.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd63fc740 20 client.15279 trim_cache size 0 max 16384 2024-03-04T12:25:06.792 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:25:06.793 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:25:06.793 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 2, 2024-03-04T12:25:06.793 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 24479, 2024-03-04T12:25:06.793 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-03-04T12:25:06.793 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-03-04T12:25:06.793 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-03-04T12:25:06.793 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 87.771765235999993, 2024-03-04T12:25:06.793 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 28, 2024-03-04T12:25:06.793 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 92, 2024-03-04T12:25:06.793 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 89, 2024-03-04T12:25:06.793 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 131.71578757099999 2024-03-04T12:25:06.793 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:25:06.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd3812640 1 -- 172.21.15.60:0/143440122 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7facb80a1d80 msgr2=0x7facb80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd3812640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7facb80a1d80 0x7facb80a4240 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7facc409a830 tx=0x7facc4065c80 comp rx=0 tx=0).stop 2024-03-04T12:25:06.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd3812640 1 -- 172.21.15.60:0/143440122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faccc102ef0 msgr2=0x7faccc19ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd3812640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faccc102ef0 0x7faccc19ec30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7facc400cad0 tx=0x7facc400cf90 comp rx=0 tx=0).stop 2024-03-04T12:25:06.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd3812640 1 -- 172.21.15.60:0/143440122 shutdown_connections 2024-03-04T12:25:06.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd3812640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7facb80a1d80 0x7facb80a4240 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd3812640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7faccc102ef0 0x7faccc19ec30 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd3812640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faccc102530 0x7faccc197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd3812640 1 --2- 172.21.15.60:0/143440122 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faccc101340 0x7faccc197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd3812640 1 -- 172.21.15.60:0/143440122 >> 172.21.15.60:0/143440122 conn(0x7faccc0fc8b0 msgr2=0x7faccc1058d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd3812640 1 -- 172.21.15.60:0/143440122 shutdown_connections 2024-03-04T12:25:06.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.779+0000 7facd3812640 1 -- 172.21.15.60:0/143440122 wait complete. 2024-03-04T12:25:06.796 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-04T12:25:06.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.897+0000 7f9aa1812640 1 -- 172.21.15.60:0/2633257651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a9c0ff640 msgr2=0x7f9a9c0ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.897+0000 7f9aa1812640 1 --2- 172.21.15.60:0/2633257651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a9c0ff640 0x7f9a9c0ffa60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009930 tx=0x7f9a8c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:06.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.897+0000 7f9aa1812640 1 -- 172.21.15.60:0/2633257651 shutdown_connections 2024-03-04T12:25:06.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.897+0000 7f9aa1812640 1 --2- 172.21.15.60:0/2633257651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c103070 0x7f9a9c109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.897+0000 7f9aa1812640 1 --2- 172.21.15.60:0/2633257651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a9c1026b0 0x7f9a9c102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.897+0000 7f9aa1812640 1 --2- 172.21.15.60:0/2633257651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a9c0ff640 0x7f9a9c0ffa60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.897+0000 7f9aa1812640 1 -- 172.21.15.60:0/2633257651 >> 172.21.15.60:0/2633257651 conn(0x7f9a9c0faf50 msgr2=0x7f9a9c0fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.897+0000 7f9aa1812640 1 -- 172.21.15.60:0/2633257651 shutdown_connections 2024-03-04T12:25:06.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.897+0000 7f9aa1812640 1 -- 172.21.15.60:0/2633257651 wait complete. 2024-03-04T12:25:06.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9aa1812640 1 Processor -- start 2024-03-04T12:25:06.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9aa1812640 1 -- start start 2024-03-04T12:25:06.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9aa1812640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a9c0ff640 0x7f9a9c197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9aa1812640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a9c1026b0 0x7f9a9c197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9aa1812640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c103070 0x7f9a9c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9aa1812640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c10cfc0 con 0x7f9a9c1026b0 2024-03-04T12:25:06.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9aa1812640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c10ce40 con 0x7f9a9c0ff640 2024-03-04T12:25:06.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9aa1812640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c10d140 con 0x7f9a9c103070 2024-03-04T12:25:06.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9a93fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a9c1026b0 0x7f9a9c197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9a93fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a9c1026b0 0x7f9a9c197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60588/0 (socket says 172.21.15.60:60588) 2024-03-04T12:25:06.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9a93fff640 1 -- 172.21.15.60:0/1888679673 learned_addr learned my addr 172.21.15.60:0/1888679673 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9a9affd640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a9c0ff640 0x7f9a9c197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9a9b7fe640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c103070 0x7f9a9c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9a93fff640 1 -- 172.21.15.60:0/1888679673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c103070 msgr2=0x7f9a9c19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9a93fff640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c103070 0x7f9a9c19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9a93fff640 1 -- 172.21.15.60:0/1888679673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a9c0ff640 msgr2=0x7f9a9c197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9a93fff640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a9c0ff640 0x7f9a9c197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9a93fff640 1 -- 172.21.15.60:0/1888679673 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c19f2b0 con 0x7f9a9c1026b0 2024-03-04T12:25:06.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.898+0000 7f9a9b7fe640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c103070 0x7f9a9c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:06.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.899+0000 7f9a93fff640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a9c1026b0 0x7f9a9c197bd0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400e9f0 tx=0x7f9a8400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.899+0000 7f9aa0810640 1 -- 172.21.15.60:0/1888679673 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8400cd80 con 0x7f9a9c1026b0 2024-03-04T12:25:06.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.899+0000 7f9aa0810640 1 -- 172.21.15.60:0/1888679673 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a84010910 con 0x7f9a9c1026b0 2024-03-04T12:25:06.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.899+0000 7f9aa1812640 1 -- 172.21.15.60:0/1888679673 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9c19f5a0 con 0x7f9a9c1026b0 2024-03-04T12:25:06.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.899+0000 7f9aa0810640 1 -- 172.21.15.60:0/1888679673 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a84018990 con 0x7f9a9c1026b0 2024-03-04T12:25:06.908 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.899+0000 7f9aa1812640 1 -- 172.21.15.60:0/1888679673 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9c19faf0 con 0x7f9a9c1026b0 2024-03-04T12:25:06.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9aa0810640 1 -- 172.21.15.60:0/1888679673 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f9a84010ac0 con 0x7f9a9c1026b0 2024-03-04T12:25:06.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9aa43f5740 1 Processor -- start 2024-03-04T12:25:06.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9aa43f5740 1 -- start start 2024-03-04T12:25:06.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9aa43f5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459200 0x55eef0459620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9aa43f5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 0x55eef045e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9aa43f5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef045ed60 0x55eef04611f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9aa43f5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef03ba7c0 con 0x55eef0459bf0 2024-03-04T12:25:06.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9aa43f5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef0363b50 con 0x55eef045ed60 2024-03-04T12:25:06.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9aa43f5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef03b6860 con 0x55eef0459200 2024-03-04T12:25:06.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9a9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 0x55eef045e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9a9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 0x55eef045e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60598/0 (socket says 172.21.15.60:60598) 2024-03-04T12:25:06.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9a9affd640 1 -- 172.21.15.60:0/1300301245 learned_addr learned my addr 172.21.15.60:0/1300301245 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9a93fff640 1 --2- 172.21.15.60:0/1300301245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef045ed60 0x55eef04611f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9a9b7fe640 1 --2- 172.21.15.60:0/1300301245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459200 0x55eef0459620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9aa0810640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9a880a1d80 0x7f9a880a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9a9affd640 1 -- 172.21.15.60:0/1300301245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459200 msgr2=0x55eef0459620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9a9affd640 1 --2- 172.21.15.60:0/1300301245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459200 0x55eef0459620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9a9affd640 1 -- 172.21.15.60:0/1300301245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef045ed60 msgr2=0x55eef04611f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9a9affd640 1 --2- 172.21.15.60:0/1300301245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef045ed60 0x55eef04611f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9a9affd640 1 -- 172.21.15.60:0/1300301245 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eef03779a0 con 0x55eef0459bf0 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.902+0000 7f9aa0810640 1 -- 172.21.15.60:0/1888679673 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f9a840ce4d0 con 0x7f9a9c1026b0 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.903+0000 7f9a9b7fe640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9a880a1d80 0x7f9a880a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.903+0000 7f9a9affd640 1 --2- 172.21.15.60:0/1300301245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 0x55eef045e6f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009900 tx=0x7f9a8c0095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5461f9a187112a56 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.903+0000 7f9a91ffb640 1 -- 172.21.15.60:0/1300301245 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c02cb80 con 0x55eef0459bf0 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.903+0000 7f9a91ffb640 1 -- 172.21.15.60:0/1300301245 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a8c005040 con 0x55eef0459bf0 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.903+0000 7f9a91ffb640 1 -- 172.21.15.60:0/1300301245 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c004a30 con 0x55eef0459bf0 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.903+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1300301245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 msgr2=0x55eef045e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.903+0000 7f9a9b7fe640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9a880a1d80 0x7f9a880a4240 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55eef045a450 tx=0x7f9a94004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.903+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/1300301245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 0x55eef045e6f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009900 tx=0x7f9a8c0095e0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.903+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1300301245 shutdown_connections 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/1300301245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef045ed60 0x55eef04611f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/1300301245 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 0x55eef045e6f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/1300301245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459200 0x55eef0459620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1300301245 >> 172.21.15.60:0/1300301245 conn(0x55eef0450360 msgr2=0x55eef0450760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1300301245 shutdown_connections 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1300301245 wait complete. 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 -- start start 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 Processor -- start 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459200 0x55eef04636b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459bf0 0x55eef0463bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef045ed60 0x55eef046ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef0461cb0 con 0x55eef0459200 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef0376f90 con 0x55eef045ed60 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef0461e30 con 0x55eef0459bf0 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9aa43f5740 10 client.0 _ll_register_callbacks cb 0x55eef0398d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9a9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459200 0x55eef04636b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9a9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459200 0x55eef04636b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60612/0 (socket says 172.21.15.60:60612) 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.904+0000 7f9a9affd640 1 -- 172.21.15.60:0/1337398318 learned_addr learned my addr 172.21.15.60:0/1337398318 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a93fff640 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459bf0 0x55eef0463bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a9affd640 1 -- 172.21.15.60:0/1337398318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459bf0 msgr2=0x55eef0463bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a9affd640 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459bf0 0x55eef0463bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a9b7fe640 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef045ed60 0x55eef046ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a9affd640 1 -- 172.21.15.60:0/1337398318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef045ed60 msgr2=0x55eef046ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a9affd640 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef045ed60 0x55eef046ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a9affd640 1 -- 172.21.15.60:0/1337398318 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eef00cf2d0 con 0x55eef0459200 2024-03-04T12:25:06.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a9affd640 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459200 0x55eef04636b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c036040 tx=0x7f9a8c02ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=296f3628e2dd50a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a7affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a7affd640 1 -- 172.21.15.60:0/1337398318 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c004b40 con 0x55eef0459200 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a7affd640 1 -- 172.21.15.60:0/1337398318 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f9a8c004ce0 con 0x55eef0459200 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1337398318 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a8c005040 con 0x55eef0459200 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a7affd640 1 -- 172.21.15.60:0/1337398318 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a8c057050 con 0x55eef0459200 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9aa43f5740 10 client.15291 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1337398318 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55eef01996b0 con 0x55eef0459200 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.905+0000 7f9a7affd640 1 -- 172.21.15.60:0/1337398318 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a8c005040 con 0x55eef0459200 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.906+0000 7f9a7affd640 1 -- 172.21.15.60:0/1337398318 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9a8c034440 con 0x55eef0459200 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.906+0000 7f9aa43f5740 10 client.15291 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.906+0000 7f9aa43f5740 10 client.15291 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.906+0000 7f9aa43f5740 20 client.15291 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.906+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55eef046fc80 0x55eef0474100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.906+0000 7f9aa43f5740 4 client.15291 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.906+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1337398318 --> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eef03c7d00 con 0x55eef046fc80 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.906+0000 7f9a93fff640 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55eef046fc80 0x55eef0474100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.906+0000 7f9a7affd640 10 client.15291 ms_handle_connect on v2:172.21.15.86:6838/3347800322 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.906+0000 7f9a93fff640 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55eef046fc80 0x55eef0474100 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=12042988d67570bc server_cookie=9067a836c90ff6c in_seq=0 out_seq=0 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9a7affd640 1 -- 172.21.15.60:0/1337398318 <== mds.1 v2:172.21.15.86:6838/3347800322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55eef03c7d00 con 0x55eef046fc80 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9a7affd640 10 client.15291 handle_command_reply: tid=0 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9aa43f5740 1 client.15291 shutdown 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1337398318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459200 msgr2=0x55eef04636b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459200 0x55eef04636b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c036040 tx=0x7f9a8c02ccb0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1337398318 shutdown_connections 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef045ed60 0x55eef046ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459bf0 0x55eef0463bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55eef046fc80 0x55eef0474100 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/1337398318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459200 0x55eef04636b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9a7affd640 0 client.15291 ms_handle_reset on v2:172.21.15.86:6838/3347800322 2024-03-04T12:25:06.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1337398318 >> 172.21.15.60:0/1337398318 conn(0x55eef0450360 msgr2=0x55eef044f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1337398318 shutdown_connections 2024-03-04T12:25:06.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9aa43f5740 1 -- 172.21.15.60:0/1337398318 wait complete. 2024-03-04T12:25:06.925 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.907+0000 7f9aa43f5740 20 client.15291 trim_cache size 0 max 16384 2024-03-04T12:25:06.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.923+0000 7f9aa43f5740 1 Processor -- start 2024-03-04T12:25:06.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.923+0000 7f9aa43f5740 1 -- start start 2024-03-04T12:25:06.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.923+0000 7f9aa43f5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0450360 0x55eef0461150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.923+0000 7f9aa43f5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459200 0x55eef046ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.923+0000 7f9aa43f5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef0459bf0 0x55eef0478830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.923+0000 7f9aa43f5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef046f4a0 con 0x55eef0450360 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.923+0000 7f9aa43f5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef046fb00 con 0x55eef0459bf0 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.923+0000 7f9aa43f5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef0470160 con 0x55eef0459200 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.923+0000 7f9a9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0450360 0x55eef0461150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.923+0000 7f9a9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0450360 0x55eef0461150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60620/0 (socket says 172.21.15.60:60620) 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.923+0000 7f9a9affd640 1 -- 172.21.15.60:0/3000058281 learned_addr learned my addr 172.21.15.60:0/3000058281 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.924+0000 7f9a9b7fe640 1 --2- 172.21.15.60:0/3000058281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef0459bf0 0x55eef0478830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.924+0000 7f9a9affd640 1 -- 172.21.15.60:0/3000058281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459200 msgr2=0x55eef046ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.924+0000 7f9a9affd640 1 --2- 172.21.15.60:0/3000058281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459200 0x55eef046ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.924+0000 7f9a9affd640 1 -- 172.21.15.60:0/3000058281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef0459bf0 msgr2=0x55eef0478830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.924+0000 7f9a9affd640 1 --2- 172.21.15.60:0/3000058281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef0459bf0 0x55eef0478830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.924+0000 7f9a9affd640 1 -- 172.21.15.60:0/3000058281 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eef045e350 con 0x55eef0450360 2024-03-04T12:25:06.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.924+0000 7f9a9b7fe640 1 --2- 172.21.15.60:0/3000058281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef0459bf0 0x55eef0478830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.924+0000 7f9a9affd640 1 --2- 172.21.15.60:0/3000058281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0450360 0x55eef0461150 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c02c710 tx=0x7f9a8c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b910a0ac81098262 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.924+0000 7f9a917fa640 1 -- 172.21.15.60:0/3000058281 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c009520 con 0x55eef0450360 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.924+0000 7f9a917fa640 1 -- 172.21.15.60:0/3000058281 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a8c049070 con 0x55eef0450360 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.924+0000 7f9a917fa640 1 -- 172.21.15.60:0/3000058281 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c034bf0 con 0x55eef0450360 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 -- 172.21.15.60:0/3000058281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0450360 msgr2=0x55eef0461150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/3000058281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0450360 0x55eef0461150 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c02c710 tx=0x7f9a8c005fd0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 -- 172.21.15.60:0/3000058281 shutdown_connections 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/3000058281 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef0459bf0 0x55eef0478830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/3000058281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0459200 0x55eef046ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/3000058281 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0450360 0x55eef0461150 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 -- 172.21.15.60:0/3000058281 >> 172.21.15.60:0/3000058281 conn(0x55eef045a9d0 msgr2=0x55eef0478120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 -- 172.21.15.60:0/3000058281 shutdown_connections 2024-03-04T12:25:06.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 -- 172.21.15.60:0/3000058281 wait complete. 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 -- start start 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 Processor -- start 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0450360 0x55eef0467cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef0459200 0x55eef0468230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 0x55eef04691a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef0470e20 con 0x55eef0459bf0 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef04707c0 con 0x55eef0459200 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef0471480 con 0x55eef0450360 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9aa43f5740 10 client.0 _ll_register_callbacks cb 0x7f9a8c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9a9b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 0x55eef04691a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9a9b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 0x55eef04691a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60622/0 (socket says 172.21.15.60:60622) 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.925+0000 7f9a9b7fe640 1 -- 172.21.15.60:0/2115550478 learned_addr learned my addr 172.21.15.60:0/2115550478 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a93fff640 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef0459200 0x55eef0468230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a9affd640 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0450360 0x55eef0467cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a9b7fe640 1 -- 172.21.15.60:0/2115550478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0450360 msgr2=0x55eef0467cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a9b7fe640 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0450360 0x55eef0467cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a9b7fe640 1 -- 172.21.15.60:0/2115550478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef0459200 msgr2=0x55eef0468230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a9b7fe640 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef0459200 0x55eef0468230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a9b7fe640 1 -- 172.21.15.60:0/2115550478 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eef0469790 con 0x55eef0459bf0 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a9b7fe640 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 0x55eef04691a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9a94010ab0 tx=0x7f9a94015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563f1e623acc8f0a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a7affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a7affd640 1 -- 172.21.15.60:0/2115550478 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9401bc70 con 0x55eef0459bf0 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a7affd640 1 -- 172.21.15.60:0/2115550478 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f9a94026410 con 0x55eef0459bf0 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9aa43f5740 1 -- 172.21.15.60:0/2115550478 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a8c049070 con 0x55eef0459bf0 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a7affd640 1 -- 172.21.15.60:0/2115550478 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a94034050 con 0x55eef0459bf0 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9aa43f5740 10 client.15297 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9aa43f5740 1 -- 172.21.15.60:0/2115550478 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55eef046aa70 con 0x55eef0459bf0 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.926+0000 7f9a7affd640 1 -- 172.21.15.60:0/2115550478 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a8c049070 con 0x55eef0459bf0 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9a7affd640 1 -- 172.21.15.60:0/2115550478 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9a9400f040 con 0x55eef0459bf0 2024-03-04T12:25:06.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9aa43f5740 10 client.15297 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9aa43f5740 10 client.15297 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9aa43f5740 20 client.15297 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55eef045ed60 0x55eef04c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9aa43f5740 4 client.15297 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9aa43f5740 1 -- 172.21.15.60:0/2115550478 --> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55eef03c2050 con 0x55eef045ed60 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9a9affd640 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55eef045ed60 0x55eef04c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9a9affd640 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55eef045ed60 0x55eef04c7dd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e51053c8919b8e8e server_cookie=9bfc5935389a9583 in_seq=0 out_seq=0 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9a7affd640 10 client.15297 ms_handle_connect on v2:172.21.15.86:6838/3347800322 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9a7affd640 1 -- 172.21.15.60:0/2115550478 <== mds.1 v2:172.21.15.86:6838/3347800322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55eef03c2050 con 0x55eef045ed60 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9a7affd640 10 client.15297 handle_command_reply: tid=0 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.927+0000 7f9aa43f5740 1 client.15297 shutdown 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa43f5740 1 -- 172.21.15.60:0/2115550478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 msgr2=0x55eef04691a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 0x55eef04691a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9a94010ab0 tx=0x7f9a94015040 comp rx=0 tx=0).stop 2024-03-04T12:25:06.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa43f5740 1 -- 172.21.15.60:0/2115550478 shutdown_connections 2024-03-04T12:25:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55eef045ed60 0x55eef04c7dd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55eef0459bf0 0x55eef04691a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9a7affd640 0 client.15297 ms_handle_reset on v2:172.21.15.86:6838/3347800322 2024-03-04T12:25:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eef0459200 0x55eef0468230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa43f5740 1 --2- 172.21.15.60:0/2115550478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eef0450360 0x55eef0467cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa43f5740 1 -- 172.21.15.60:0/2115550478 >> 172.21.15.60:0/2115550478 conn(0x55eef045a9d0 msgr2=0x55eef04789b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa43f5740 1 -- 172.21.15.60:0/2115550478 shutdown_connections 2024-03-04T12:25:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa43f5740 1 -- 172.21.15.60:0/2115550478 wait complete. 2024-03-04T12:25:06.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa43f5740 20 client.15297 trim_cache size 0 max 16384 2024-03-04T12:25:06.940 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:25:06.940 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:25:06.941 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 1, 2024-03-04T12:25:06.941 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 24487, 2024-03-04T12:25:06.941 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-03-04T12:25:06.941 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-03-04T12:25:06.941 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-03-04T12:25:06.941 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 88.931474207999997, 2024-03-04T12:25:06.941 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 28, 2024-03-04T12:25:06.941 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 92, 2024-03-04T12:25:06.941 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 89, 2024-03-04T12:25:06.941 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 129.11745333299999 2024-03-04T12:25:06.941 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:25:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa1812640 1 -- 172.21.15.60:0/1888679673 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9a880a1d80 msgr2=0x7f9a880a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa1812640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9a880a1d80 0x7f9a880a4240 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55eef045a450 tx=0x7f9a94004a40 comp rx=0 tx=0).stop 2024-03-04T12:25:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa1812640 1 -- 172.21.15.60:0/1888679673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a9c1026b0 msgr2=0x7f9a9c197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa1812640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a9c1026b0 0x7f9a9c197bd0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400e9f0 tx=0x7f9a8400eeb0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa1812640 1 -- 172.21.15.60:0/1888679673 shutdown_connections 2024-03-04T12:25:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa1812640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f9a880a1d80 0x7f9a880a4240 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa1812640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c103070 0x7f9a9c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa1812640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9a9c1026b0 0x7f9a9c197bd0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa1812640 1 --2- 172.21.15.60:0/1888679673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a9c0ff640 0x7f9a9c197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:06.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa1812640 1 -- 172.21.15.60:0/1888679673 >> 172.21.15.60:0/1888679673 conn(0x7f9a9c0faf50 msgr2=0x7f9a9c105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:06.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.928+0000 7f9aa1812640 1 -- 172.21.15.60:0/1888679673 shutdown_connections 2024-03-04T12:25:06.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:06.929+0000 7f9aa1812640 1 -- 172.21.15.60:0/1888679673 wait complete. 2024-03-04T12:25:06.950 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-04T12:25:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.048+0000 7f8f448a4640 1 -- 172.21.15.60:0/2819627033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40102f10 msgr2=0x7f8f401097a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.048+0000 7f8f448a4640 1 --2- 172.21.15.60:0/2819627033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40102f10 0x7f8f401097a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8f30009ca0 tx=0x7f8f3002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.048+0000 7f8f448a4640 1 -- 172.21.15.60:0/2819627033 shutdown_connections 2024-03-04T12:25:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.048+0000 7f8f448a4640 1 --2- 172.21.15.60:0/2819627033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40102f10 0x7f8f401097a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.048+0000 7f8f448a4640 1 --2- 172.21.15.60:0/2819627033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f40102550 0x7f8f401029d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.048+0000 7f8f448a4640 1 --2- 172.21.15.60:0/2819627033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40101360 0x7f8f40101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.054 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.048+0000 7f8f448a4640 1 -- 172.21.15.60:0/2819627033 >> 172.21.15.60:0/2819627033 conn(0x7f8f400fc8f0 msgr2=0x7f8f400fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:07.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.048+0000 7f8f448a4640 1 -- 172.21.15.60:0/2819627033 shutdown_connections 2024-03-04T12:25:07.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.048+0000 7f8f448a4640 1 -- 172.21.15.60:0/2819627033 wait complete. 2024-03-04T12:25:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.048+0000 7f8f448a4640 1 Processor -- start 2024-03-04T12:25:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f448a4640 1 -- start start 2024-03-04T12:25:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f448a4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f40101360 0x7f8f401975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f448a4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40102550 0x7f8f40197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f448a4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40102f10 0x7f8f4019eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f448a4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4010cff0 con 0x7f8f40102550 2024-03-04T12:25:07.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f448a4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4010ce70 con 0x7f8f40102f10 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f448a4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4010d170 con 0x7f8f40101360 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f3dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40102550 0x7f8f40197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f3dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40102550 0x7f8f40197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60638/0 (socket says 172.21.15.60:60638) 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/3066890220 learned_addr learned my addr 172.21.15.60:0/3066890220 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f40101360 0x7f8f401975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40102f10 0x7f8f4019eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/3066890220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f40101360 msgr2=0x7f8f401975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f40101360 0x7f8f401975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/3066890220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40102f10 msgr2=0x7f8f4019eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40102f10 0x7f8f4019eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/3066890220 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4019f200 con 0x7f8f40102550 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40102f10 0x7f8f4019eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40102550 0x7f8f40197b20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8f2800e9f0 tx=0x7f8f2800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.049+0000 7f8f377fe640 1 -- 172.21.15.60:0/3066890220 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2800cd80 con 0x7f8f40102550 2024-03-04T12:25:07.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.050+0000 7f8f448a4640 1 -- 172.21.15.60:0/3066890220 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4019f4f0 con 0x7f8f40102550 2024-03-04T12:25:07.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.050+0000 7f8f377fe640 1 -- 172.21.15.60:0/3066890220 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f28005500 con 0x7f8f40102550 2024-03-04T12:25:07.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.050+0000 7f8f377fe640 1 -- 172.21.15.60:0/3066890220 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28010690 con 0x7f8f40102550 2024-03-04T12:25:07.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.050+0000 7f8f448a4640 1 -- 172.21.15.60:0/3066890220 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4019fa40 con 0x7f8f40102550 2024-03-04T12:25:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.052+0000 7f8f377fe640 1 -- 172.21.15.60:0/3066890220 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f8f280046c0 con 0x7f8f40102550 2024-03-04T12:25:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.052+0000 7f8f47567740 1 Processor -- start 2024-03-04T12:25:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.052+0000 7f8f47567740 1 -- start start 2024-03-04T12:25:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.052+0000 7f8f47567740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 0x564c75c97210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.052+0000 7f8f47567740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c977e0 0x564c75c9c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.052+0000 7f8f47567740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 0x564c75c9ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.052+0000 7f8f47567740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c75bfc7c0 con 0x564c75c9c950 2024-03-04T12:25:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.052+0000 7f8f47567740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c75ba5b50 con 0x564c75c977e0 2024-03-04T12:25:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.052+0000 7f8f47567740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c75bf8860 con 0x564c75c96df0 2024-03-04T12:25:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.052+0000 7f8f3dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 0x564c75c9ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.052+0000 7f8f3dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 0x564c75c9ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60652/0 (socket says 172.21.15.60:60652) 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/4041766413 learned_addr learned my addr 172.21.15.60:0/4041766413 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f377fe640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8f2c0a1d80 0x7f8f2c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/4041766413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c977e0 0x564c75c9c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/4041766413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 0x564c75c97210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8f2c0a1d80 0x7f8f2c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f377fe640 1 -- 172.21.15.60:0/3066890220 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f8f280ce5e0 con 0x7f8f40102550 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/4041766413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 msgr2=0x564c75c97210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/4041766413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 0x564c75c97210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/4041766413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c977e0 msgr2=0x564c75c9c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/4041766413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c977e0 0x564c75c9c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/4041766413 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c75bb99a0 con 0x564c75c9c950 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/4041766413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c977e0 0x564c75c9c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/4041766413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 0x564c75c9ede0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8f2809ac80 tx=0x7f8f28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67ae25cc15e2051 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8f2c0a1d80 0x7f8f2c0a4240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x564c75c98040 tx=0x7f8f300095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:07.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f34ff9640 1 -- 172.21.15.60:0/4041766413 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2809a8e0 con 0x564c75c9c950 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f34ff9640 1 -- 172.21.15.60:0/4041766413 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f280d1cc0 con 0x564c75c9c950 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.053+0000 7f8f34ff9640 1 -- 172.21.15.60:0/4041766413 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280ce310 con 0x564c75c9c950 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.054+0000 7f8f47567740 1 -- 172.21.15.60:0/4041766413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 msgr2=0x564c75c9ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.054+0000 7f8f47567740 1 --2- 172.21.15.60:0/4041766413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 0x564c75c9ede0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8f2809ac80 tx=0x7f8f28004a40 comp rx=0 tx=0).stop 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.054+0000 7f8f47567740 1 -- 172.21.15.60:0/4041766413 shutdown_connections 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.054+0000 7f8f47567740 1 --2- 172.21.15.60:0/4041766413 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 0x564c75c9ede0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.054+0000 7f8f47567740 1 --2- 172.21.15.60:0/4041766413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c977e0 0x564c75c9c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.054+0000 7f8f47567740 1 --2- 172.21.15.60:0/4041766413 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 0x564c75c97210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.054+0000 7f8f47567740 1 -- 172.21.15.60:0/4041766413 >> 172.21.15.60:0/4041766413 conn(0x564c75c92360 msgr2=0x564c75c92760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.054+0000 7f8f47567740 1 -- 172.21.15.60:0/4041766413 shutdown_connections 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.054+0000 7f8f47567740 1 -- 172.21.15.60:0/4041766413 wait complete. 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.054+0000 7f8f47567740 1 -- start start 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.054+0000 7f8f47567740 1 Processor -- start 2024-03-04T12:25:07.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f47567740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c96df0 0x564c75ca1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f47567740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c977e0 0x564c75ca17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f47567740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 0x564c75ca87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f47567740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c75c9f840 con 0x564c75c9c950 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f47567740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c75bb8f90 con 0x564c75c96df0 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f47567740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c75c9f9c0 con 0x564c75c977e0 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f47567740 10 client.0 _ll_register_callbacks cb 0x564c75bdad90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f3ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 0x564c75ca87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f3ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 0x564c75ca87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60660/0 (socket says 172.21.15.60:60660) 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f3ed9d640 1 -- 172.21.15.60:0/3622989938 learned_addr learned my addr 172.21.15.60:0/3622989938 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c96df0 0x564c75ca1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c977e0 0x564c75ca17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f3ed9d640 1 -- 172.21.15.60:0/3622989938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c977e0 msgr2=0x564c75ca17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c977e0 0x564c75ca17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f3ed9d640 1 -- 172.21.15.60:0/3622989938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c96df0 msgr2=0x564c75ca1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:07.069 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c96df0 0x564c75ca1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f3ed9d640 1 -- 172.21.15.60:0/3622989938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c759112d0 con 0x564c75c9c950 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 0x564c75ca87f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8f30036000 tx=0x7f8f3002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f33ea19360bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f25ffb640 1 -- 172.21.15.60:0/3622989938 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f30052060 con 0x564c75c9c950 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f25ffb640 1 -- 172.21.15.60:0/3622989938 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f8f300044c0 con 0x564c75c9c950 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.055+0000 7f8f47567740 1 -- 172.21.15.60:0/3622989938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564c75c09d00 con 0x564c75c9c950 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.056+0000 7f8f25ffb640 1 -- 172.21.15.60:0/3622989938 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f30059050 con 0x564c75c9c950 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.056+0000 7f8f47567740 10 client.15309 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.056+0000 7f8f47567740 1 -- 172.21.15.60:0/3622989938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564c759db6b0 con 0x564c75c9c950 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.056+0000 7f8f25ffb640 1 -- 172.21.15.60:0/3622989938 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564c75c09d00 con 0x564c75c9c950 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.056+0000 7f8f25ffb640 1 -- 172.21.15.60:0/3622989938 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8f3004dc80 con 0x564c75c9c950 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.056+0000 7f8f47567740 10 client.15309 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.056+0000 7f8f47567740 10 client.15309 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.056+0000 7f8f47567740 20 client.15309 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:07.070 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.056+0000 7f8f47567740 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564c75cade10 0x564c75cacec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.056+0000 7f8f47567740 4 client.15309 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.056+0000 7f8f47567740 1 -- 172.21.15.60:0/3622989938 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c75c04050 con 0x564c75cade10 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.056+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564c75cade10 0x564c75cacec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.057+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564c75cade10 0x564c75cacec0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76975133364c9547 server_cookie=e663870986e2a3b4 in_seq=0 out_seq=0 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.057+0000 7f8f25ffb640 10 client.15309 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.057+0000 7f8f25ffb640 1 -- 172.21.15.60:0/3622989938 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564c75c04050 con 0x564c75cade10 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.057+0000 7f8f25ffb640 10 client.15309 handle_command_reply: tid=0 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.057+0000 7f8f47567740 1 client.15309 shutdown 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.058+0000 7f8f47567740 1 -- 172.21.15.60:0/3622989938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 msgr2=0x564c75ca87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.058+0000 7f8f47567740 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 0x564c75ca87f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8f30036000 tx=0x7f8f3002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.058+0000 7f8f47567740 1 -- 172.21.15.60:0/3622989938 shutdown_connections 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.058+0000 7f8f47567740 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564c75cade10 0x564c75cacec0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.058+0000 7f8f47567740 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c9c950 0x564c75ca87f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.058+0000 7f8f47567740 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c977e0 0x564c75ca17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.071 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.058+0000 7f8f47567740 1 --2- 172.21.15.60:0/3622989938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c96df0 0x564c75ca1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.058+0000 7f8f25ffb640 0 client.15309 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.058+0000 7f8f47567740 1 -- 172.21.15.60:0/3622989938 >> 172.21.15.60:0/3622989938 conn(0x564c75c92360 msgr2=0x564c75c91640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.058+0000 7f8f47567740 1 -- 172.21.15.60:0/3622989938 shutdown_connections 2024-03-04T12:25:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.058+0000 7f8f47567740 1 -- 172.21.15.60:0/3622989938 wait complete. 2024-03-04T12:25:07.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.058+0000 7f8f47567740 20 client.15309 trim_cache size 0 max 16384 2024-03-04T12:25:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.073+0000 7f8f47567740 1 Processor -- start 2024-03-04T12:25:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.073+0000 7f8f47567740 1 -- start start 2024-03-04T12:25:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.073+0000 7f8f47567740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 0x564c75ca0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.073+0000 7f8f47567740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 0x564c75ca13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.073+0000 7f8f47567740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c9c950 0x564c75c90db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.073+0000 7f8f47567740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c75cad630 con 0x564c75c977e0 2024-03-04T12:25:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.073+0000 7f8f47567740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c75cadc90 con 0x564c75c9c950 2024-03-04T12:25:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.073+0000 7f8f47567740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c75cae2f0 con 0x564c75c96df0 2024-03-04T12:25:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f3dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 0x564c75ca13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f3dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 0x564c75ca13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60672/0 (socket says 172.21.15.60:60672) 2024-03-04T12:25:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/788149865 learned_addr learned my addr 172.21.15.60:0/788149865 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:07.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/788149865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c9c950 0x564c75c90db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/788149865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 msgr2=0x564c75ca0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/788149865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 0x564c75ca0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/788149865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c9c950 msgr2=0x564c75c90db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/788149865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c9c950 0x564c75c90db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/788149865 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c75c9bf70 con 0x564c75c977e0 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/788149865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 0x564c75ca13c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8f28097950 tx=0x7f8f280ce8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188665b2dce4598d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f27fff640 1 -- 172.21.15.60:0/788149865 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280b3070 con 0x564c75c977e0 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f27fff640 1 -- 172.21.15.60:0/788149865 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f2809af80 con 0x564c75c977e0 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.074+0000 7f8f27fff640 1 -- 172.21.15.60:0/788149865 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280ce310 con 0x564c75c977e0 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 -- 172.21.15.60:0/788149865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 msgr2=0x564c75ca13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 --2- 172.21.15.60:0/788149865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 0x564c75ca13c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8f28097950 tx=0x7f8f280ce8b0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 -- 172.21.15.60:0/788149865 shutdown_connections 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 --2- 172.21.15.60:0/788149865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c9c950 0x564c75c90db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.086 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 --2- 172.21.15.60:0/788149865 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 0x564c75ca13c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 --2- 172.21.15.60:0/788149865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 0x564c75ca0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 -- 172.21.15.60:0/788149865 >> 172.21.15.60:0/788149865 conn(0x564c75c92360 msgr2=0x564c75c98ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 -- 172.21.15.60:0/788149865 shutdown_connections 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 -- 172.21.15.60:0/788149865 wait complete. 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 -- start start 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 Processor -- start 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 0x564c75d008d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 0x564c75d00e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c9c950 0x564c75d07e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c75caefb0 con 0x564c75c977e0 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c75cae950 con 0x564c75c9c950 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c75caf610 con 0x564c75c96df0 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f47567740 10 client.0 _ll_register_callbacks cb 0x7f8f28097220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:07.087 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f3dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 0x564c75d00e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f3dd9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 0x564c75d00e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60676/0 (socket says 172.21.15.60:60676) 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/317108197 learned_addr learned my addr 172.21.15.60:0/317108197 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.075+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c9c950 0x564c75d07e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 0x564c75d008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/317108197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 msgr2=0x564c75d008d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 0x564c75d008d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/317108197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c9c950 msgr2=0x564c75d07e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c9c950 0x564c75d07e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f3dd9b640 1 -- 172.21.15.60:0/317108197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c75d08440 con 0x564c75c977e0 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 0x564c75d008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f3dd9b640 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 0x564c75d00e10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8f28095a20 tx=0x7f8f2809a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e623e642e53854ec server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:07.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f25ffb640 1 -- 172.21.15.60:0/317108197 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280b3050 con 0x564c75c977e0 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f25ffb640 1 -- 172.21.15.60:0/317108197 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f8f280a3290 con 0x564c75c977e0 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f47567740 1 -- 172.21.15.60:0/317108197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564c75b829a0 con 0x564c75c977e0 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f25ffb640 1 -- 172.21.15.60:0/317108197 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f2809b680 con 0x564c75c977e0 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f47567740 10 client.15315 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f47567740 1 -- 172.21.15.60:0/317108197 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564c75d09740 con 0x564c75c977e0 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f25ffb640 1 -- 172.21.15.60:0/317108197 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564c75b829a0 con 0x564c75c977e0 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f25ffb640 1 -- 172.21.15.60:0/317108197 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8f28096df0 con 0x564c75c977e0 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.076+0000 7f8f47567740 10 client.15315 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 10 client.15315 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 20 client.15315 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564c75d0d730 0x564c75d09910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 4 client.15315 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 1 -- 172.21.15.60:0/317108197 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564c75bce760 con 0x564c75d0d730 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564c75d0d730 0x564c75d09910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:07.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564c75d0d730 0x564c75d09910 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6694a7cbb90997f server_cookie=82e9ce667c366ef5 in_seq=0 out_seq=0 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f25ffb640 10 client.15315 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f25ffb640 1 -- 172.21.15.60:0/317108197 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x564c75bce760 con 0x564c75d0d730 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f25ffb640 10 client.15315 handle_command_reply: tid=0 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 1 client.15315 shutdown 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 1 -- 172.21.15.60:0/317108197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 msgr2=0x564c75d00e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 0x564c75d00e10 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f8f28095a20 tx=0x7f8f2809a8e0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 1 -- 172.21.15.60:0/317108197 shutdown_connections 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564c75d0d730 0x564c75d09910 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564c75c9c950 0x564c75d07e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f25ffb640 0 client.15315 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564c75c977e0 0x564c75d00e10 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.077+0000 7f8f47567740 1 --2- 172.21.15.60:0/317108197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564c75c96df0 0x564c75d008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f47567740 1 -- 172.21.15.60:0/317108197 >> 172.21.15.60:0/317108197 conn(0x564c75c92360 msgr2=0x564c75ca0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f47567740 1 -- 172.21.15.60:0/317108197 shutdown_connections 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f47567740 1 -- 172.21.15.60:0/317108197 wait complete. 2024-03-04T12:25:07.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f47567740 20 client.15315 trim_cache size 0 max 16384 2024-03-04T12:25:07.091 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:25:07.091 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:25:07.091 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2024-03-04T12:25:07.091 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 24524, 2024-03-04T12:25:07.091 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-03-04T12:25:07.091 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-03-04T12:25:07.091 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-03-04T12:25:07.091 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 114.17375366900001, 2024-03-04T12:25:07.091 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 28, 2024-03-04T12:25:07.091 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 93, 2024-03-04T12:25:07.091 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 89, 2024-03-04T12:25:07.092 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 120.475099503 2024-03-04T12:25:07.092 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:25:07.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f448a4640 1 -- 172.21.15.60:0/3066890220 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8f2c0a1d80 msgr2=0x7f8f2c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f448a4640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8f2c0a1d80 0x7f8f2c0a4240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x564c75c98040 tx=0x7f8f300095c0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f448a4640 1 -- 172.21.15.60:0/3066890220 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40102550 msgr2=0x7f8f40197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f448a4640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40102550 0x7f8f40197b20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8f2800e9f0 tx=0x7f8f2800edf0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f448a4640 1 -- 172.21.15.60:0/3066890220 shutdown_connections 2024-03-04T12:25:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f448a4640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8f2c0a1d80 0x7f8f2c0a4240 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f448a4640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40102f10 0x7f8f4019eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f448a4640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40102550 0x7f8f40197b20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f448a4640 1 --2- 172.21.15.60:0/3066890220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f40101360 0x7f8f401975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f448a4640 1 -- 172.21.15.60:0/3066890220 >> 172.21.15.60:0/3066890220 conn(0x7f8f400fc8f0 msgr2=0x7f8f40105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f448a4640 1 -- 172.21.15.60:0/3066890220 shutdown_connections 2024-03-04T12:25:07.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:07.078+0000 7f8f448a4640 1 -- 172.21.15.60:0/3066890220 wait complete. 2024-03-04T12:25:07.095 INFO:tasks.cephadm:Setup complete, yielding 2024-03-04T12:25:07.096 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T12:25:07.106 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi060.front.sepia.ceph.com 2024-03-04T12:25:07.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- bash -c 'ceph orch status' 2024-03-04T12:25:07.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:06 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-04T12:25:07.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:06 smithi086 ceph-mon[36851]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-04T12:25:07.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:06 smithi086 ceph-mon[36851]: osdmap e92: 12 total, 12 up, 12 in 2024-03-04T12:25:07.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:06 smithi086 ceph-mon[36851]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-04T12:25:07.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:06 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/466301686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:07.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:06 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1095837321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:07.350 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:25:07.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:06 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-04T12:25:07.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:06 smithi063 ceph-mon[36717]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-04T12:25:07.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:06 smithi063 ceph-mon[36717]: osdmap e92: 12 total, 12 up, 12 in 2024-03-04T12:25:07.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:06 smithi063 ceph-mon[36717]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-04T12:25:07.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:06 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/466301686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:07.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:06 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1095837321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:07.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:06 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-04T12:25:07.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:06 smithi060 ceph-mon[32745]: from='mgr.14152 172.21.15.60:0/3626831200' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-04T12:25:07.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:06 smithi060 ceph-mon[32745]: osdmap e92: 12 total, 12 up, 12 in 2024-03-04T12:25:07.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:06 smithi060 ceph-mon[32745]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-04T12:25:07.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:06 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/466301686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:07.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:06 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1095837321' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:08.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:07 smithi060 ceph-mon[32745]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T12:25:08.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:07 smithi060 ceph-mon[32745]: osdmap e93: 12 total, 12 up, 12 in 2024-03-04T12:25:08.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:07 smithi086 ceph-mon[36851]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T12:25:08.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:07 smithi086 ceph-mon[36851]: osdmap e93: 12 total, 12 up, 12 in 2024-03-04T12:25:08.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:07 smithi063 ceph-mon[36717]: pgmap v353: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T12:25:08.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:07 smithi063 ceph-mon[36717]: osdmap e93: 12 total, 12 up, 12 in 2024-03-04T12:25:09.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.195+0000 7f890e1c0700 1 -- 172.21.15.60:0/2894406131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8908100110 msgr2=0x7f8908100530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:09.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.195+0000 7f890e1c0700 1 --2- 172.21.15.60:0/2894406131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8908100110 0x7f8908100530 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009990 tx=0x7f88f8009ca0 comp rx=0 tx=0).stop 2024-03-04T12:25:09.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.195+0000 7f890e1c0700 1 -- 172.21.15.60:0/2894406131 shutdown_connections 2024-03-04T12:25:09.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.195+0000 7f890e1c0700 1 --2- 172.21.15.60:0/2894406131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8908103a90 0x7f890810a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:09.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.195+0000 7f890e1c0700 1 --2- 172.21.15.60:0/2894406131 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8908100110 0x7f8908100530 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:09.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.195+0000 7f890e1c0700 1 --2- 172.21.15.60:0/2894406131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89081030f0 0x7f8908103550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:09.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.195+0000 7f890e1c0700 1 -- 172.21.15.60:0/2894406131 >> 172.21.15.60:0/2894406131 conn(0x7f89080fba30 msgr2=0x7f89080fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:09.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.195+0000 7f890e1c0700 1 -- 172.21.15.60:0/2894406131 shutdown_connections 2024-03-04T12:25:09.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.196+0000 7f890e1c0700 1 -- 172.21.15.60:0/2894406131 wait complete. 2024-03-04T12:25:09.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.196+0000 7f890e1c0700 1 Processor -- start 2024-03-04T12:25:09.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.196+0000 7f890e1c0700 1 -- start start 2024-03-04T12:25:09.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.196+0000 7f890e1c0700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8908100110 0x7f8908198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:09.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.196+0000 7f890e1c0700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89081030f0 0x7f89081985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.196+0000 7f890e1c0700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8908103a90 0x7f890819f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.196+0000 7f890e1c0700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890810da80 con 0x7f8908103a90 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.196+0000 7f890e1c0700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890810d900 con 0x7f89081030f0 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.196+0000 7f890e1c0700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890810dc00 con 0x7f8908100110 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8907fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8908103a90 0x7f890819f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8907fff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8908103a90 0x7f890819f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60694/0 (socket says 172.21.15.60:60694) 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8907fff700 1 -- 172.21.15.60:0/4103236921 learned_addr learned my addr 172.21.15.60:0/4103236921 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f89077fe700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8908100110 0x7f8908198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8906ffd700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89081030f0 0x7f89081985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8907fff700 1 -- 172.21.15.60:0/4103236921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8908100110 msgr2=0x7f8908198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8907fff700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8908100110 0x7f8908198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8907fff700 1 -- 172.21.15.60:0/4103236921 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89081030f0 msgr2=0x7f89081985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8907fff700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89081030f0 0x7f89081985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8907fff700 1 -- 172.21.15.60:0/4103236921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890819fcc0 con 0x7f8908103a90 2024-03-04T12:25:09.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8906ffd700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89081030f0 0x7f89081985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:09.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8907fff700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8908103a90 0x7f890819f620 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8900007da0 tx=0x7f890000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:09.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8904ff9700 1 -- 172.21.15.60:0/4103236921 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900016020 con 0x7f8908103a90 2024-03-04T12:25:09.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f890e1c0700 1 -- 172.21.15.60:0/4103236921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890819ffb0 con 0x7f8908103a90 2024-03-04T12:25:09.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8904ff9700 1 -- 172.21.15.60:0/4103236921 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f890000ad00 con 0x7f8908103a90 2024-03-04T12:25:09.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f8904ff9700 1 -- 172.21.15.60:0/4103236921 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89000116b0 con 0x7f8908103a90 2024-03-04T12:25:09.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.197+0000 7f890e1c0700 1 -- 172.21.15.60:0/4103236921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89081a04e0 con 0x7f8908103a90 2024-03-04T12:25:09.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.198+0000 7f890e1c0700 1 -- 172.21.15.60:0/4103236921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8908068ae0 con 0x7f8908103a90 2024-03-04T12:25:09.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.200+0000 7f8904ff9700 1 -- 172.21.15.60:0/4103236921 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f8900019020 con 0x7f8908103a90 2024-03-04T12:25:09.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.200+0000 7f8904ff9700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f88f00a1e10 0x7f88f00a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:09.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.200+0000 7f89077fe700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f88f00a1e10 0x7f88f00a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:09.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.201+0000 7f8904ff9700 1 -- 172.21.15.60:0/4103236921 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f89000ce580 con 0x7f8908103a90 2024-03-04T12:25:09.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.201+0000 7f89077fe700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f88f00a1e10 0x7f88f00a42d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f88f80039c0 tx=0x7f88f8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:09.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.204+0000 7f8904ff9700 1 -- 172.21.15.60:0/4103236921 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8900020020 con 0x7f8908103a90 2024-03-04T12:25:09.431 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.425+0000 7f890e1c0700 1 -- 172.21.15.60:0/4103236921 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8908062260 con 0x7f88f00a1e10 2024-03-04T12:25:09.432 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.426+0000 7f8904ff9700 1 -- 172.21.15.60:0/4103236921 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8908062260 con 0x7f88f00a1e10 2024-03-04T12:25:09.432 INFO:teuthology.orchestra.run.smithi060.stdout:Backend: cephadm 2024-03-04T12:25:09.432 INFO:teuthology.orchestra.run.smithi060.stdout:Available: Yes 2024-03-04T12:25:09.432 INFO:teuthology.orchestra.run.smithi060.stdout:Paused: No 2024-03-04T12:25:09.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.428+0000 7f890e1c0700 1 -- 172.21.15.60:0/4103236921 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f88f00a1e10 msgr2=0x7f88f00a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.428+0000 7f890e1c0700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f88f00a1e10 0x7f88f00a42d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f88f80039c0 tx=0x7f88f8005fb0 comp rx=0 tx=0).stop 2024-03-04T12:25:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.428+0000 7f890e1c0700 1 -- 172.21.15.60:0/4103236921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8908103a90 msgr2=0x7f890819f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.428+0000 7f890e1c0700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8908103a90 0x7f890819f620 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f8900007da0 tx=0x7f890000a340 comp rx=0 tx=0).stop 2024-03-04T12:25:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.428+0000 7f890e1c0700 1 -- 172.21.15.60:0/4103236921 shutdown_connections 2024-03-04T12:25:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.428+0000 7f890e1c0700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f88f00a1e10 0x7f88f00a42d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.428+0000 7f890e1c0700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8908103a90 0x7f890819f620 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.428+0000 7f890e1c0700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8908100110 0x7f8908198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.428+0000 7f890e1c0700 1 --2- 172.21.15.60:0/4103236921 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89081030f0 0x7f89081985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.428+0000 7f890e1c0700 1 -- 172.21.15.60:0/4103236921 >> 172.21.15.60:0/4103236921 conn(0x7f89080fba30 msgr2=0x7f89081063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.428+0000 7f890e1c0700 1 -- 172.21.15.60:0/4103236921 shutdown_connections 2024-03-04T12:25:09.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:09.428+0000 7f890e1c0700 1 -- 172.21.15.60:0/4103236921 wait complete. 2024-03-04T12:25:10.075 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:09 smithi060 ceph-mon[32745]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-04T12:25:10.105 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-04T12:25:10.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:09 smithi086 ceph-mon[36851]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-04T12:25:10.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:09 smithi063 ceph-mon[36717]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-04T12:25:10.353 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:25:11.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:10 smithi060 ceph-mon[32745]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:11.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:10 smithi086 ceph-mon[36851]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:11.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:10 smithi063 ceph-mon[36717]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:12.114 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:11 smithi060 ceph-mon[32745]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:12.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.116+0000 7f6777a72700 1 -- 172.21.15.60:0/1370102847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67701029f0 msgr2=0x7f6770104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:12.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.116+0000 7f6777a72700 1 --2- 172.21.15.60:0/1370102847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67701029f0 0x7f6770104de0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f676c00b0a0 tx=0x7f676c00b3b0 comp rx=0 tx=0).stop 2024-03-04T12:25:12.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.116+0000 7f6777a72700 1 -- 172.21.15.60:0/1370102847 shutdown_connections 2024-03-04T12:25:12.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.116+0000 7f6777a72700 1 --2- 172.21.15.60:0/1370102847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6770107c50 0x7f677010a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.116+0000 7f6777a72700 1 --2- 172.21.15.60:0/1370102847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67701029f0 0x7f6770104de0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.116+0000 7f6777a72700 1 --2- 172.21.15.60:0/1370102847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6770105320 0x7f6770107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.116+0000 7f6777a72700 1 -- 172.21.15.60:0/1370102847 >> 172.21.15.60:0/1370102847 conn(0x7f67700fc380 msgr2=0x7f67700fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.116+0000 7f6777a72700 1 -- 172.21.15.60:0/1370102847 shutdown_connections 2024-03-04T12:25:12.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.116+0000 7f6777a72700 1 -- 172.21.15.60:0/1370102847 wait complete. 2024-03-04T12:25:12.125 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f6777a72700 1 Processor -- start 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f6777a72700 1 -- start start 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f6777a72700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67701029f0 0x7f6770198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f6777a72700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6770105320 0x7f67701985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f6777a72700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6770107c50 0x7f677019f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f6777a72700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677010da50 con 0x7f67701029f0 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f6777a72700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677010d8d0 con 0x7f6770107c50 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f6777a72700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677010dbd0 con 0x7f6770105320 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f677580e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67701029f0 0x7f6770198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f677580e700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67701029f0 0x7f6770198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60718/0 (socket says 172.21.15.60:60718) 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f677580e700 1 -- 172.21.15.60:0/2941495766 learned_addr learned my addr 172.21.15.60:0/2941495766 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f677500d700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6770105320 0x7f67701985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.117+0000 7f677600f700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6770107c50 0x7f677019f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.118+0000 7f677580e700 1 -- 172.21.15.60:0/2941495766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6770105320 msgr2=0x7f67701985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.118+0000 7f677580e700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6770105320 0x7f67701985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:12.126 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.118+0000 7f677580e700 1 -- 172.21.15.60:0/2941495766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6770107c50 msgr2=0x7f677019f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.118+0000 7f677580e700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6770107c50 0x7f677019f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.118+0000 7f677580e700 1 -- 172.21.15.60:0/2941495766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677019fcf0 con 0x7f67701029f0 2024-03-04T12:25:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.118+0000 7f677600f700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6770107c50 0x7f677019f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.118+0000 7f677580e700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67701029f0 0x7f6770198090 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f676c004d00 tx=0x7f676c004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.118+0000 7f6766ffd700 1 -- 172.21.15.60:0/2941495766 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c021070 con 0x7f67701029f0 2024-03-04T12:25:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.118+0000 7f6766ffd700 1 -- 172.21.15.60:0/2941495766 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f676c007b80 con 0x7f67701029f0 2024-03-04T12:25:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.118+0000 7f6766ffd700 1 -- 172.21.15.60:0/2941495766 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c005400 con 0x7f67701029f0 2024-03-04T12:25:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.118+0000 7f6777a72700 1 -- 172.21.15.60:0/2941495766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677019ff80 con 0x7f67701029f0 2024-03-04T12:25:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.118+0000 7f6777a72700 1 -- 172.21.15.60:0/2941495766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67701a0320 con 0x7f67701029f0 2024-03-04T12:25:12.127 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.119+0000 7f6777a72700 1 -- 172.21.15.60:0/2941495766 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6770068ae0 con 0x7f67701029f0 2024-03-04T12:25:12.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.120+0000 7f6766ffd700 1 -- 172.21.15.60:0/2941495766 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f676c024070 con 0x7f67701029f0 2024-03-04T12:25:12.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.121+0000 7f6766ffd700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f675c0a1e90 0x7f675c0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:12.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.121+0000 7f677500d700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f675c0a1e90 0x7f675c0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:12.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.121+0000 7f6766ffd700 1 -- 172.21.15.60:0/2941495766 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f676c00f020 con 0x7f67701029f0 2024-03-04T12:25:12.128 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.122+0000 7f677500d700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f675c0a1e90 0x7f675c0a4350 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6760009c60 tx=0x7f6760009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:12.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.125+0000 7f6766ffd700 1 -- 172.21.15.60:0/2941495766 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f676c02b020 con 0x7f67701029f0 2024-03-04T12:25:12.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:11 smithi086 ceph-mon[36851]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:12.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:11 smithi063 ceph-mon[36717]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:12.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.357+0000 7f6777a72700 1 -- 172.21.15.60:0/2941495766 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f6770062320 con 0x7f675c0a1e90 2024-03-04T12:25:12.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.371+0000 7f6766ffd700 1 -- 172.21.15.60:0/2941495766 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4092 (secure 0 0 0) 0x7f6770062320 con 0x7f675c0a1e90 2024-03-04T12:25:12.378 INFO:teuthology.orchestra.run.smithi060.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T12:25:12.378 INFO:teuthology.orchestra.run.smithi060.stdout:mds.a smithi060 running (2m) 20s ago 2m 16.2M - 19.0.0-1845-g717ce3cc d71b839abdf8 5b43f6f536a3 2024-03-04T12:25:12.378 INFO:teuthology.orchestra.run.smithi060.stdout:mds.b smithi063 running (2m) 59s ago 2m 25.9M - 19.0.0-1845-g717ce3cc d71b839abdf8 e613f71e49fc 2024-03-04T12:25:12.378 INFO:teuthology.orchestra.run.smithi060.stdout:mds.c smithi086 running (2m) 59s ago 2m 18.4M - 19.0.0-1845-g717ce3cc d71b839abdf8 b34586918c54 2024-03-04T12:25:12.378 INFO:teuthology.orchestra.run.smithi060.stdout:mds.d smithi060 running (2m) 20s ago 2m 15.3M - 19.0.0-1845-g717ce3cc d71b839abdf8 06f28728ec76 2024-03-04T12:25:12.378 INFO:teuthology.orchestra.run.smithi060.stdout:mds.e smithi063 running (2m) 59s ago 2m 16.1M - 19.0.0-1845-g717ce3cc d71b839abdf8 9856d0e13ea0 2024-03-04T12:25:12.378 INFO:teuthology.orchestra.run.smithi060.stdout:mds.f smithi086 running (2m) 59s ago 2m 18.4M - 19.0.0-1845-g717ce3cc d71b839abdf8 089cf647a6e0 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:mds.g smithi060 running (2m) 20s ago 2m 16.4M - 19.0.0-1845-g717ce3cc d71b839abdf8 fb7bf3da2f1e 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:mds.h smithi063 running (2m) 59s ago 2m 17.2M - 19.0.0-1845-g717ce3cc d71b839abdf8 6f810dfaa3da 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:mds.i smithi086 running (2m) 59s ago 2m 19.1M - 19.0.0-1845-g717ce3cc d71b839abdf8 2f83800fdca4 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:mds.j smithi060 running (2m) 20s ago 2m 15.4M - 19.0.0-1845-g717ce3cc d71b839abdf8 1364bf2fcc86 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:mds.k smithi063 running (2m) 59s ago 2m 18.2M - 19.0.0-1845-g717ce3cc d71b839abdf8 ee27880ef103 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:mds.l smithi086 running (2m) 59s ago 2m 19.4M - 19.0.0-1845-g717ce3cc d71b839abdf8 20368d8c0c29 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.x smithi060 *:9283,8765 running (9m) 20s ago 9m 513M - 19.0.0-1845-g717ce3cc d71b839abdf8 fbd5b7e7c4ab 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.y smithi063 *:8443,8765 running (7m) 59s ago 7m 453M - 19.0.0-1845-g717ce3cc d71b839abdf8 41d77cc52e14 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:mgr.z smithi086 *:8443,8765 running (7m) 59s ago 7m 451M - 19.0.0-1845-g717ce3cc d71b839abdf8 d4c9ad43a3c1 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:mon.a smithi060 running (9m) 20s ago 10m 52.1M 2048M 19.0.0-1845-g717ce3cc d71b839abdf8 421225a59508 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:mon.b smithi063 running (8m) 59s ago 8m 44.7M 2048M 19.0.0-1845-g717ce3cc d71b839abdf8 85077ac207da 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:mon.c smithi086 running (8m) 59s ago 8m 40.3M 2048M 19.0.0-1845-g717ce3cc d71b839abdf8 4bc60bafa884 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 smithi060 running (7m) 20s ago 7m 57.1M 4273M 19.0.0-1845-g717ce3cc d71b839abdf8 861c1e5941d2 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 smithi063 running (7m) 59s ago 7m 59.0M 4270M 19.0.0-1845-g717ce3cc d71b839abdf8 1af520c2fa6e 2024-03-04T12:25:12.379 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 smithi086 running (6m) 59s ago 6m 57.5M 4270M 19.0.0-1845-g717ce3cc d71b839abdf8 318ed0e9fe6c 2024-03-04T12:25:12.380 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 smithi060 running (6m) 20s ago 6m 44.1M 4273M 19.0.0-1845-g717ce3cc d71b839abdf8 0249575dc6f2 2024-03-04T12:25:12.380 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 smithi063 running (6m) 59s ago 6m 54.3M 4270M 19.0.0-1845-g717ce3cc d71b839abdf8 6bbacaaa55c9 2024-03-04T12:25:12.380 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 smithi086 running (5m) 59s ago 5m 54.9M 4270M 19.0.0-1845-g717ce3cc d71b839abdf8 fec9b645cfbf 2024-03-04T12:25:12.380 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 smithi060 running (5m) 20s ago 5m 41.9M 4273M 19.0.0-1845-g717ce3cc d71b839abdf8 033afb30cd3c 2024-03-04T12:25:12.380 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 smithi063 running (4m) 59s ago 4m 44.6M 4270M 19.0.0-1845-g717ce3cc d71b839abdf8 f3ad8b5be01e 2024-03-04T12:25:12.380 INFO:teuthology.orchestra.run.smithi060.stdout:osd.8 smithi086 running (4m) 59s ago 4m 56.4M 4270M 19.0.0-1845-g717ce3cc d71b839abdf8 a5f24ffd47af 2024-03-04T12:25:12.380 INFO:teuthology.orchestra.run.smithi060.stdout:osd.9 smithi060 running (3m) 20s ago 3m 53.9M 4273M 19.0.0-1845-g717ce3cc d71b839abdf8 e2eed35f4b31 2024-03-04T12:25:12.380 INFO:teuthology.orchestra.run.smithi060.stdout:osd.10 smithi063 running (3m) 59s ago 3m 43.2M 4270M 19.0.0-1845-g717ce3cc d71b839abdf8 2fa9efedd416 2024-03-04T12:25:12.380 INFO:teuthology.orchestra.run.smithi060.stdout:osd.11 smithi086 running (3m) 59s ago 3m 52.2M 4270M 19.0.0-1845-g717ce3cc d71b839abdf8 06a14f562bca 2024-03-04T12:25:12.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.374+0000 7f6777a72700 1 -- 172.21.15.60:0/2941495766 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f675c0a1e90 msgr2=0x7f675c0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:12.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.374+0000 7f6777a72700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f675c0a1e90 0x7f675c0a4350 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6760009c60 tx=0x7f6760009260 comp rx=0 tx=0).stop 2024-03-04T12:25:12.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.374+0000 7f6777a72700 1 -- 172.21.15.60:0/2941495766 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67701029f0 msgr2=0x7f6770198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:12.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.374+0000 7f6777a72700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67701029f0 0x7f6770198090 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f676c004d00 tx=0x7f676c004d30 comp rx=0 tx=0).stop 2024-03-04T12:25:12.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.374+0000 7f6777a72700 1 -- 172.21.15.60:0/2941495766 shutdown_connections 2024-03-04T12:25:12.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.374+0000 7f6777a72700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6770107c50 0x7f677019f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:12.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.374+0000 7f6777a72700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f675c0a1e90 0x7f675c0a4350 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:12.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.374+0000 7f6777a72700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67701029f0 0x7f6770198090 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:12.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.374+0000 7f6777a72700 1 --2- 172.21.15.60:0/2941495766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6770105320 0x7f67701985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:12.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.374+0000 7f6777a72700 1 -- 172.21.15.60:0/2941495766 >> 172.21.15.60:0/2941495766 conn(0x7f67700fc380 msgr2=0x7f6770103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:12.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.374+0000 7f6777a72700 1 -- 172.21.15.60:0/2941495766 shutdown_connections 2024-03-04T12:25:12.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:12.375+0000 7f6777a72700 1 -- 172.21.15.60:0/2941495766 wait complete. 2024-03-04T12:25:13.064 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-04T12:25:13.310 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:25:14.176 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:13 smithi060 ceph-mon[32745]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 895 B/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:14.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:13 smithi060 ceph-mon[32745]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:13 smithi086 ceph-mon[36851]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 895 B/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:14.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:13 smithi086 ceph-mon[36851]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:14.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:13 smithi063 ceph-mon[36717]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 895 B/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:14.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:13 smithi063 ceph-mon[36717]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:15.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.041+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/3900487989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3a4102fb0 msgr2=0x7ff3a4103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:15.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.041+0000 7ff3ab8b4700 1 --2- 172.21.15.60:0/3900487989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3a4102fb0 0x7ff3a4103430 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff394009990 tx=0x7ff394009ca0 comp rx=0 tx=0).stop 2024-03-04T12:25:15.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.041+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/3900487989 shutdown_connections 2024-03-04T12:25:15.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.041+0000 7ff3ab8b4700 1 --2- 172.21.15.60:0/3900487989 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4103970 0x7ff3a410a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:15.057 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.041+0000 7ff3ab8b4700 1 --2- 172.21.15.60:0/3900487989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3a4101e50 0x7ff3a4102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.041+0000 7ff3ab8b4700 1 --2- 172.21.15.60:0/3900487989 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3a4102fb0 0x7ff3a4103430 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.041+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/3900487989 >> 172.21.15.60:0/3900487989 conn(0x7ff3a40fd430 msgr2=0x7ff3a40ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.042+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/3900487989 shutdown_connections 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.042+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/3900487989 wait complete. 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.042+0000 7ff3ab8b4700 1 Processor -- start 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.042+0000 7ff3ab8b4700 1 -- start start 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.042+0000 7ff3ab8b4700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3a4101e50 0x7ff3a41980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.042+0000 7ff3ab8b4700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4102fb0 0x7ff3a41985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.042+0000 7ff3ab8b4700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3a4103970 0x7ff3a419f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.042+0000 7ff3ab8b4700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a410daa0 con 0x7ff3a4103970 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.042+0000 7ff3ab8b4700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a410d920 con 0x7ff3a4102fb0 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.042+0000 7ff3ab8b4700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a410dc20 con 0x7ff3a4101e50 2024-03-04T12:25:15.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3a9e51700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3a4103970 0x7ff3a419f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3a9e51700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3a4103970 0x7ff3a419f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:60732/0 (socket says 172.21.15.60:60732) 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3a9e51700 1 -- 172.21.15.60:0/193593500 learned_addr learned my addr 172.21.15.60:0/193593500 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3a9650700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3a4101e50 0x7ff3a41980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3a8e4f700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4102fb0 0x7ff3a41985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3a9e51700 1 -- 172.21.15.60:0/193593500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3a4101e50 msgr2=0x7ff3a41980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3a9e51700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3a4101e50 0x7ff3a41980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3a9e51700 1 -- 172.21.15.60:0/193593500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4102fb0 msgr2=0x7ff3a41985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3a9e51700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4102fb0 0x7ff3a41985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3a9e51700 1 -- 172.21.15.60:0/193593500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a419fd10 con 0x7ff3a4103970 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3a8e4f700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4102fb0 0x7ff3a41985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3a9e51700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3a4103970 0x7ff3a419f670 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff39c00b9a0 tx=0x7ff39c00bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff39a7fc700 1 -- 172.21.15.60:0/193593500 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c019070 con 0x7ff3a4103970 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff39a7fc700 1 -- 172.21.15.60:0/193593500 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff39c00ce10 con 0x7ff3a4103970 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/193593500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a419ffa0 con 0x7ff3a4103970 2024-03-04T12:25:15.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff39a7fc700 1 -- 172.21.15.60:0/193593500 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c0073d0 con 0x7ff3a4103970 2024-03-04T12:25:15.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.043+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/193593500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a41a0460 con 0x7ff3a4103970 2024-03-04T12:25:15.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.044+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/193593500 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a4068ae0 con 0x7ff3a4103970 2024-03-04T12:25:15.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.045+0000 7ff39a7fc700 1 -- 172.21.15.60:0/193593500 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7ff39c01c070 con 0x7ff3a4103970 2024-03-04T12:25:15.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.046+0000 7ff39a7fc700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff3900a1e10 0x7ff3900a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:15.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.046+0000 7ff3a9650700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff3900a1e10 0x7ff3900a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:15.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.046+0000 7ff39a7fc700 1 -- 172.21.15.60:0/193593500 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7ff39c0cd490 con 0x7ff3a4103970 2024-03-04T12:25:15.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.047+0000 7ff3a9650700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff3900a1e10 0x7ff3900a42d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0007c60 tx=0x7ff3a00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:15.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.051+0000 7ff39a7fc700 1 -- 172.21.15.60:0/193593500 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff39c09a530 con 0x7ff3a4103970 2024-03-04T12:25:15.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.264+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/193593500 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3a4062260 con 0x7ff3900a1e10 2024-03-04T12:25:15.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.270+0000 7ff39a7fc700 1 -- 172.21.15.60:0/193593500 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7ff3a4062260 con 0x7ff3900a1e10 2024-03-04T12:25:15.276 INFO:teuthology.orchestra.run.smithi060.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-04T12:25:15.276 INFO:teuthology.orchestra.run.smithi060.stdout:mds.all 12/12 62s ago 2m smithi060=a;smithi060=d;smithi060=g;smithi060=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi086=c;smithi086=f;smithi086=i;smithi086=l;count:12 2024-03-04T12:25:15.276 INFO:teuthology.orchestra.run.smithi060.stdout:mgr 3/3 62s ago 8m smithi060=x;smithi063=y;smithi086=z;count:3 2024-03-04T12:25:15.277 INFO:teuthology.orchestra.run.smithi060.stdout:mon 3/3 62s ago 8m smithi060:172.21.15.60=a;smithi063:172.21.15.63=b;smithi086:172.21.15.86=c;count:3 2024-03-04T12:25:15.277 INFO:teuthology.orchestra.run.smithi060.stdout:osd 12 62s ago - 2024-03-04T12:25:15.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.272+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/193593500 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff3900a1e10 msgr2=0x7ff3900a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:15.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.272+0000 7ff3ab8b4700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff3900a1e10 0x7ff3900a42d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0007c60 tx=0x7ff3a00073d0 comp rx=0 tx=0).stop 2024-03-04T12:25:15.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.272+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/193593500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3a4103970 msgr2=0x7ff3a419f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:15.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.272+0000 7ff3ab8b4700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3a4103970 0x7ff3a419f670 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff39c00b9a0 tx=0x7ff39c00bcb0 comp rx=0 tx=0).stop 2024-03-04T12:25:15.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.272+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/193593500 shutdown_connections 2024-03-04T12:25:15.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.272+0000 7ff3ab8b4700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff3900a1e10 0x7ff3900a42d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:15.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.272+0000 7ff3ab8b4700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff3a4103970 0x7ff3a419f670 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:15.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.272+0000 7ff3ab8b4700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3a4101e50 0x7ff3a41980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:15.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.272+0000 7ff3ab8b4700 1 --2- 172.21.15.60:0/193593500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff3a4102fb0 0x7ff3a41985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:15.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.272+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/193593500 >> 172.21.15.60:0/193593500 conn(0x7ff3a40fd430 msgr2=0x7ff3a4106340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:15.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.272+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/193593500 shutdown_connections 2024-03-04T12:25:15.279 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:15.272+0000 7ff3ab8b4700 1 -- 172.21.15.60:0/193593500 wait complete. 2024-03-04T12:25:15.992 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-04T12:25:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:25:16.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:15 smithi086 ceph-mon[36851]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 868 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:16.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:15 smithi063 ceph-mon[36717]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 868 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:16.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:15 smithi060 ceph-mon[32745]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 868 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:17.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:16 smithi086 ceph-mon[36851]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:17.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:16 smithi063 ceph-mon[36717]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:17.376 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:16 smithi060 ceph-mon[32745]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:18.010 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:17 smithi060 ceph-mon[32745]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:18.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.013+0000 7f003c8ba700 1 -- 172.21.15.60:0/13107377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0038100150 msgr2=0x7f0038100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:18.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.013+0000 7f003c8ba700 1 --2- 172.21.15.60:0/13107377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0038100150 0x7f0038100570 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f00280099c0 tx=0x7f0028009cd0 comp rx=0 tx=0).stop 2024-03-04T12:25:18.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.013+0000 7f003c8ba700 1 -- 172.21.15.60:0/13107377 shutdown_connections 2024-03-04T12:25:18.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.013+0000 7f003c8ba700 1 --2- 172.21.15.60:0/13107377 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0038103b70 0x7f003810a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:18.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.013+0000 7f003c8ba700 1 --2- 172.21.15.60:0/13107377 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0038100150 0x7f0038100570 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:18.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.013+0000 7f003c8ba700 1 --2- 172.21.15.60:0/13107377 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00381031d0 0x7f0038103630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:18.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.013+0000 7f003c8ba700 1 -- 172.21.15.60:0/13107377 >> 172.21.15.60:0/13107377 conn(0x7f00380fba70 msgr2=0x7f00380fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:18.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.013+0000 7f003c8ba700 1 -- 172.21.15.60:0/13107377 shutdown_connections 2024-03-04T12:25:18.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.013+0000 7f003c8ba700 1 -- 172.21.15.60:0/13107377 wait complete. 2024-03-04T12:25:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f003c8ba700 1 Processor -- start 2024-03-04T12:25:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f003c8ba700 1 -- start start 2024-03-04T12:25:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f003c8ba700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0038100150 0x7f0038198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f003c8ba700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00381031d0 0x7f00381985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f003c8ba700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0038103b70 0x7f003819f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f003c8ba700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003810da30 con 0x7f00381031d0 2024-03-04T12:25:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f003c8ba700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003810d8b0 con 0x7f0038103b70 2024-03-04T12:25:18.022 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f003c8ba700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003810dbb0 con 0x7f0038100150 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f0035d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00381031d0 0x7f00381985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f0035d9b700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00381031d0 0x7f00381985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49126/0 (socket says 172.21.15.60:49126) 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f0035d9b700 1 -- 172.21.15.60:0/2668768690 learned_addr learned my addr 172.21.15.60:0/2668768690 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f003659c700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0038100150 0x7f0038198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f0036d9d700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0038103b70 0x7f003819f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f0035d9b700 1 -- 172.21.15.60:0/2668768690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0038100150 msgr2=0x7f0038198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f0035d9b700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0038100150 0x7f0038198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f0035d9b700 1 -- 172.21.15.60:0/2668768690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0038103b70 msgr2=0x7f003819f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f0035d9b700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0038103b70 0x7f003819f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f0035d9b700 1 -- 172.21.15.60:0/2668768690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003819fcd0 con 0x7f00381031d0 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f003659c700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0038100150 0x7f0038198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.014+0000 7f0036d9d700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0038103b70 0x7f003819f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.015+0000 7f0035d9b700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00381031d0 0x7f00381985b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f002c00d980 tx=0x7f002c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:18.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.015+0000 7f00277fe700 1 -- 172.21.15.60:0/2668768690 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c016070 con 0x7f00381031d0 2024-03-04T12:25:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.015+0000 7f003c8ba700 1 -- 172.21.15.60:0/2668768690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003819ffc0 con 0x7f00381031d0 2024-03-04T12:25:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.015+0000 7f00277fe700 1 -- 172.21.15.60:0/2668768690 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f002c009b40 con 0x7f00381031d0 2024-03-04T12:25:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.015+0000 7f00277fe700 1 -- 172.21.15.60:0/2668768690 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c00f620 con 0x7f00381031d0 2024-03-04T12:25:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.015+0000 7f003c8ba700 1 -- 172.21.15.60:0/2668768690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00381a0470 con 0x7f00381031d0 2024-03-04T12:25:18.024 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.016+0000 7f003c8ba700 1 -- 172.21.15.60:0/2668768690 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0038068ae0 con 0x7f00381031d0 2024-03-04T12:25:18.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.017+0000 7f00277fe700 1 -- 172.21.15.60:0/2668768690 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f002c019070 con 0x7f00381031d0 2024-03-04T12:25:18.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.018+0000 7f00277fe700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f00200a1e10 0x7f00200a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:18.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.018+0000 7f003659c700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f00200a1e10 0x7f00200a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:18.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.018+0000 7f00277fe700 1 -- 172.21.15.60:0/2668768690 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f002c0cdd60 con 0x7f00381031d0 2024-03-04T12:25:18.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.018+0000 7f003659c700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f00200a1e10 0x7f00200a42d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0028006010 tx=0x7f0028005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.022+0000 7f00277fe700 1 -- 172.21.15.60:0/2668768690 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f002c09ada0 con 0x7f00381031d0 2024-03-04T12:25:18.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.232+0000 7f003c8ba700 1 -- 172.21.15.60:0/2668768690 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0038062260 con 0x7f00200a1e10 2024-03-04T12:25:18.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.233+0000 7f00277fe700 1 -- 172.21.15.60:0/2668768690 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+187 (secure 0 0 0) 0x7f0038062260 con 0x7f00200a1e10 2024-03-04T12:25:18.239 INFO:teuthology.orchestra.run.smithi060.stdout:HOST ADDR LABELS STATUS 2024-03-04T12:25:18.239 INFO:teuthology.orchestra.run.smithi060.stdout:smithi060 172.21.15.60 2024-03-04T12:25:18.239 INFO:teuthology.orchestra.run.smithi060.stdout:smithi063 172.21.15.63 2024-03-04T12:25:18.239 INFO:teuthology.orchestra.run.smithi060.stdout:smithi086 172.21.15.86 2024-03-04T12:25:18.240 INFO:teuthology.orchestra.run.smithi060.stdout:3 hosts in cluster 2024-03-04T12:25:18.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.235+0000 7f003c8ba700 1 -- 172.21.15.60:0/2668768690 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f00200a1e10 msgr2=0x7f00200a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:18.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.235+0000 7f003c8ba700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f00200a1e10 0x7f00200a42d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0028006010 tx=0x7f0028005c20 comp rx=0 tx=0).stop 2024-03-04T12:25:18.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.235+0000 7f003c8ba700 1 -- 172.21.15.60:0/2668768690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00381031d0 msgr2=0x7f00381985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:18.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.235+0000 7f003c8ba700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00381031d0 0x7f00381985b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f002c00d980 tx=0x7f002c00dd40 comp rx=0 tx=0).stop 2024-03-04T12:25:18.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.235+0000 7f003c8ba700 1 -- 172.21.15.60:0/2668768690 shutdown_connections 2024-03-04T12:25:18.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.235+0000 7f003c8ba700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0038103b70 0x7f003819f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:18.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.235+0000 7f003c8ba700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0038100150 0x7f0038198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:18.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.235+0000 7f003c8ba700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f00200a1e10 0x7f00200a42d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:18.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.235+0000 7f003c8ba700 1 --2- 172.21.15.60:0/2668768690 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f00381031d0 0x7f00381985b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:18.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.235+0000 7f003c8ba700 1 -- 172.21.15.60:0/2668768690 >> 172.21.15.60:0/2668768690 conn(0x7f00380fba70 msgr2=0x7f00381064f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:18.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.235+0000 7f003c8ba700 1 -- 172.21.15.60:0/2668768690 shutdown_connections 2024-03-04T12:25:18.242 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:18.235+0000 7f003c8ba700 1 -- 172.21.15.60:0/2668768690 wait complete. 2024-03-04T12:25:18.253 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:17 smithi063 ceph-mon[36717]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:18.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:17 smithi086 ceph-mon[36851]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 716 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:18.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-04T12:25:19.204 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:25:20.324 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:19 smithi060 ceph-mon[32745]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 636 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:20.324 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:19 smithi060 ceph-mon[32745]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:20.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:19 smithi086 ceph-mon[36851]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 636 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:20.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:19 smithi086 ceph-mon[36851]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:20.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:19 smithi063 ceph-mon[36717]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 636 B/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:20.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:19 smithi063 ceph-mon[36717]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.886+0000 7f8ffa39c700 1 -- 172.21.15.60:0/1008463557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff4101e50 msgr2=0x7f8ff4102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.886+0000 7f8ffa39c700 1 --2- 172.21.15.60:0/1008463557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff4101e50 0x7f8ff4102270 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009990 tx=0x7f8fe4009ca0 comp rx=0 tx=0).stop 2024-03-04T12:25:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.886+0000 7f8ffa39c700 1 -- 172.21.15.60:0/1008463557 shutdown_connections 2024-03-04T12:25:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.886+0000 7f8ffa39c700 1 --2- 172.21.15.60:0/1008463557 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ff4103970 0x7f8ff410a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.886+0000 7f8ffa39c700 1 --2- 172.21.15.60:0/1008463557 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff4101e50 0x7f8ff4102270 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.886+0000 7f8ffa39c700 1 --2- 172.21.15.60:0/1008463557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ff4102fb0 0x7f8ff4103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:20.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.886+0000 7f8ffa39c700 1 -- 172.21.15.60:0/1008463557 >> 172.21.15.60:0/1008463557 conn(0x7f8ff40fd3d0 msgr2=0x7f8ff40ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:20.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.886+0000 7f8ffa39c700 1 -- 172.21.15.60:0/1008463557 shutdown_connections 2024-03-04T12:25:20.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.886+0000 7f8ffa39c700 1 -- 172.21.15.60:0/1008463557 wait complete. 2024-03-04T12:25:20.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.887+0000 7f8ffa39c700 1 Processor -- start 2024-03-04T12:25:20.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.887+0000 7f8ffa39c700 1 -- start start 2024-03-04T12:25:20.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.887+0000 7f8ffa39c700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ff4101e50 0x7f8ff4198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:20.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.887+0000 7f8ffa39c700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ff4102fb0 0x7f8ff41985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:20.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.887+0000 7f8ffa39c700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff4103970 0x7f8ff419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:20.895 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.887+0000 7f8ffa39c700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff410da20 con 0x7f8ff4103970 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.887+0000 7f8ffa39c700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff410d8a0 con 0x7f8ff4102fb0 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.887+0000 7f8ffa39c700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff410dba0 con 0x7f8ff4101e50 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.887+0000 7f8ff8939700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff4103970 0x7f8ff419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.887+0000 7f8ff8939700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff4103970 0x7f8ff419f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49148/0 (socket says 172.21.15.60:49148) 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.887+0000 7f8ff8939700 1 -- 172.21.15.60:0/329271392 learned_addr learned my addr 172.21.15.60:0/329271392 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.887+0000 7f8ff3fff700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ff4101e50 0x7f8ff4198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ff37fe700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ff4102fb0 0x7f8ff41985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ff8939700 1 -- 172.21.15.60:0/329271392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ff4101e50 msgr2=0x7f8ff4198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ff8939700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ff4101e50 0x7f8ff4198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ff8939700 1 -- 172.21.15.60:0/329271392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ff4102fb0 msgr2=0x7f8ff41985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ff8939700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ff4102fb0 0x7f8ff41985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ff8939700 1 -- 172.21.15.60:0/329271392 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff419fcd0 con 0x7f8ff4103970 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ff37fe700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ff4102fb0 0x7f8ff41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:20.896 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ff3fff700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ff4101e50 0x7f8ff4198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:25:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ff8939700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff4103970 0x7f8ff419f630 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8fec007f10 tx=0x7f8fec00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ff17fa700 1 -- 172.21.15.60:0/329271392 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec00ddd0 con 0x7f8ff4103970 2024-03-04T12:25:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ff17fa700 1 -- 172.21.15.60:0/329271392 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8fec004020 con 0x7f8ff4103970 2024-03-04T12:25:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ffa39c700 1 -- 172.21.15.60:0/329271392 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff419ffc0 con 0x7f8ff4103970 2024-03-04T12:25:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ff17fa700 1 -- 172.21.15.60:0/329271392 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec015660 con 0x7f8ff4103970 2024-03-04T12:25:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.888+0000 7f8ffa39c700 1 -- 172.21.15.60:0/329271392 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff41a0470 con 0x7f8ff4103970 2024-03-04T12:25:20.897 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.889+0000 7f8ffa39c700 1 -- 172.21.15.60:0/329271392 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff4068ae0 con 0x7f8ff4103970 2024-03-04T12:25:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.890+0000 7f8ff17fa700 1 -- 172.21.15.60:0/329271392 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f8fec0041d0 con 0x7f8ff4103970 2024-03-04T12:25:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.891+0000 7f8ff17fa700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8fdc0a1e90 0x7f8fdc0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.891+0000 7f8ff3fff700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8fdc0a1e90 0x7f8fdc0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.891+0000 7f8ff17fa700 1 -- 172.21.15.60:0/329271392 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f8fec0ce330 con 0x7f8ff4103970 2024-03-04T12:25:20.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.891+0000 7f8ff3fff700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8fdc0a1e90 0x7f8fdc0a4350 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400b2e0 tx=0x7f8fe400b820 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:20.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:20.894+0000 7f8ff17fa700 1 -- 172.21.15.60:0/329271392 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8fec010070 con 0x7f8ff4103970 2024-03-04T12:25:21.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.111+0000 7f8ffa39c700 1 -- 172.21.15.60:0/329271392 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ff4062260 con 0x7f8fdc0a1e90 2024-03-04T12:25:21.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.113+0000 7f8ff17fa700 1 -- 172.21.15.60:0/329271392 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f8ff4062260 con 0x7f8fdc0a1e90 2024-03-04T12:25:21.119 INFO:teuthology.orchestra.run.smithi060.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-04T12:25:21.119 INFO:teuthology.orchestra.run.smithi060.stdout:smithi060 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476008A480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-04T12:25:21.119 INFO:teuthology.orchestra.run.smithi060.stdout:smithi063 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5344005B400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-04T12:25:21.119 INFO:teuthology.orchestra.run.smithi060.stdout:smithi086 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UL400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-04T12:25:21.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.115+0000 7f8ffa39c700 1 -- 172.21.15.60:0/329271392 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8fdc0a1e90 msgr2=0x7f8fdc0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:21.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.115+0000 7f8ffa39c700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8fdc0a1e90 0x7f8fdc0a4350 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8fe400b2e0 tx=0x7f8fe400b820 comp rx=0 tx=0).stop 2024-03-04T12:25:21.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.115+0000 7f8ffa39c700 1 -- 172.21.15.60:0/329271392 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff4103970 msgr2=0x7f8ff419f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:21.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.115+0000 7f8ffa39c700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff4103970 0x7f8ff419f630 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8fec007f10 tx=0x7f8fec00d3b0 comp rx=0 tx=0).stop 2024-03-04T12:25:21.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.115+0000 7f8ffa39c700 1 -- 172.21.15.60:0/329271392 shutdown_connections 2024-03-04T12:25:21.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.115+0000 7f8ffa39c700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8fdc0a1e90 0x7f8fdc0a4350 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:21.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.115+0000 7f8ffa39c700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ff4103970 0x7f8ff419f630 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:21.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.115+0000 7f8ffa39c700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ff4101e50 0x7f8ff4198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:21.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.115+0000 7f8ffa39c700 1 --2- 172.21.15.60:0/329271392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ff4102fb0 0x7f8ff41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:21.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.115+0000 7f8ffa39c700 1 -- 172.21.15.60:0/329271392 >> 172.21.15.60:0/329271392 conn(0x7f8ff40fd3d0 msgr2=0x7f8ff41063c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:21.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.115+0000 7f8ffa39c700 1 -- 172.21.15.60:0/329271392 shutdown_connections 2024-03-04T12:25:21.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:21.115+0000 7f8ffa39c700 1 -- 172.21.15.60:0/329271392 wait complete. 2024-03-04T12:25:21.821 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T12:25:21.830 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi060.front.sepia.ceph.com 2024-03-04T12:25:21.830 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-04T12:25:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:25:22.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:21 smithi060 ceph-mon[32745]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:22.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:21 smithi060 ceph-mon[32745]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:22.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:21 smithi086 ceph-mon[36851]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:22.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:21 smithi086 ceph-mon[36851]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:22.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:21 smithi063 ceph-mon[36717]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:22.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:21 smithi063 ceph-mon[36717]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:23.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.769+0000 7f6ca2dac700 1 -- 172.21.15.60:0/458180672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c9c101f40 msgr2=0x7f6c9c1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:23.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.769+0000 7f6ca2dac700 1 --2- 172.21.15.60:0/458180672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c9c101f40 0x7f6c9c1023a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6c840099c0 tx=0x7f6c84009cd0 comp rx=0 tx=0).stop 2024-03-04T12:25:23.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.769+0000 7f6ca2dac700 1 -- 172.21.15.60:0/458180672 shutdown_connections 2024-03-04T12:25:23.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.769+0000 7f6ca2dac700 1 --2- 172.21.15.60:0/458180672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c1028e0 0x7f6c9c109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:23.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.769+0000 7f6ca2dac700 1 --2- 172.21.15.60:0/458180672 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c9c100d40 0x7f6c9c101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:23.777 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.769+0000 7f6ca2dac700 1 --2- 172.21.15.60:0/458180672 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c9c101f40 0x7f6c9c1023a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.769+0000 7f6ca2dac700 1 -- 172.21.15.60:0/458180672 >> 172.21.15.60:0/458180672 conn(0x7f6c9c0fc2e0 msgr2=0x7f6c9c0fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.769+0000 7f6ca2dac700 1 -- 172.21.15.60:0/458180672 shutdown_connections 2024-03-04T12:25:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.769+0000 7f6ca2dac700 1 -- 172.21.15.60:0/458180672 wait complete. 2024-03-04T12:25:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6ca2dac700 1 Processor -- start 2024-03-04T12:25:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6ca2dac700 1 -- start start 2024-03-04T12:25:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6ca2dac700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c100d40 0x7f6c9c104cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6ca2dac700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c9c101f40 0x7f6c9c103320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6ca2dac700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c9c1028e0 0x7f6c9c103860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6ca2dac700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c10c930 con 0x7f6c9c101f40 2024-03-04T12:25:23.778 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6ca2dac700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c10c7b0 con 0x7f6c9c1028e0 2024-03-04T12:25:23.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6ca2dac700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c10cab0 con 0x7f6c9c100d40 2024-03-04T12:25:23.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6c9bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c9c101f40 0x7f6c9c103320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:23.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6c9bfff700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c9c101f40 0x7f6c9c103320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49162/0 (socket says 172.21.15.60:49162) 2024-03-04T12:25:23.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6c9bfff700 1 -- 172.21.15.60:0/1769721387 learned_addr learned my addr 172.21.15.60:0/1769721387 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:23.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6ca1349700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c9c1028e0 0x7f6c9c103860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:23.779 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6ca0b48700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c100d40 0x7f6c9c104cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:23.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6c9bfff700 1 -- 172.21.15.60:0/1769721387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c100d40 msgr2=0x7f6c9c104cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:23.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6c9bfff700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c100d40 0x7f6c9c104cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:23.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6c9bfff700 1 -- 172.21.15.60:0/1769721387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c9c1028e0 msgr2=0x7f6c9c103860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:23.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6c9bfff700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c9c1028e0 0x7f6c9c103860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:23.780 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6c9bfff700 1 -- 172.21.15.60:0/1769721387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c104110 con 0x7f6c9c101f40 2024-03-04T12:25:23.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6ca0b48700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c100d40 0x7f6c9c104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:23.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.770+0000 7f6c9bfff700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c9c101f40 0x7f6c9c103320 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6c84003890 tx=0x7f6c840047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:23.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.771+0000 7f6c99ffb700 1 -- 172.21.15.60:0/1769721387 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84027070 con 0x7f6c9c101f40 2024-03-04T12:25:23.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.771+0000 7f6c99ffb700 1 -- 172.21.15.60:0/1769721387 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c840041d0 con 0x7f6c9c101f40 2024-03-04T12:25:23.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.771+0000 7f6c99ffb700 1 -- 172.21.15.60:0/1769721387 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8400f460 con 0x7f6c9c101f40 2024-03-04T12:25:23.781 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.771+0000 7f6ca2dac700 1 -- 172.21.15.60:0/1769721387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9c1a2110 con 0x7f6c9c101f40 2024-03-04T12:25:23.782 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.771+0000 7f6ca2dac700 1 -- 172.21.15.60:0/1769721387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c1a2510 con 0x7f6c9c101f40 2024-03-04T12:25:23.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.772+0000 7f6ca2dac700 1 -- 172.21.15.60:0/1769721387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9c068ae0 con 0x7f6c9c101f40 2024-03-04T12:25:23.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.774+0000 7f6c99ffb700 1 -- 172.21.15.60:0/1769721387 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f6c8402a070 con 0x7f6c9c101f40 2024-03-04T12:25:23.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.774+0000 7f6c99ffb700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6c880a1e10 0x7f6c880a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:23.783 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.774+0000 7f6ca0b48700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6c880a1e10 0x7f6c880a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:23.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.774+0000 7f6c99ffb700 1 -- 172.21.15.60:0/1769721387 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f6c840cf550 con 0x7f6c9c101f40 2024-03-04T12:25:23.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.775+0000 7f6ca0b48700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6c880a1e10 0x7f6c880a42d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c101da0 tx=0x7f6c8c009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:23.784 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:23.778+0000 7f6c99ffb700 1 -- 172.21.15.60:0/1769721387 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6c84013070 con 0x7f6c9c101f40 2024-03-04T12:25:24.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.026+0000 7f6ca2dac700 1 -- 172.21.15.60:0/1769721387 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f6c9c1a2900 con 0x7f6c9c101f40 2024-03-04T12:25:24.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.027+0000 7f6c99ffb700 1 -- 172.21.15.60:0/1769721387 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+2959 (secure 0 0 0) 0x7f6c8409c680 con 0x7f6c9c101f40 2024-03-04T12:25:24.033 INFO:teuthology.orchestra.run.smithi060.stdout:e28 2024-03-04T12:25:24.033 INFO:teuthology.orchestra.run.smithi060.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T12:25:24.033 INFO:teuthology.orchestra.run.smithi060.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T12:25:24.033 INFO:teuthology.orchestra.run.smithi060.stdout:legacy client fscid: 1 2024-03-04T12:25:24.033 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:24.033 INFO:teuthology.orchestra.run.smithi060.stdout:Filesystem 'cephfs' (1) 2024-03-04T12:25:24.033 INFO:teuthology.orchestra.run.smithi060.stdout:fs_name cephfs 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 28 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:created 2024-03-04T12:23:12.241967+0000 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:modified 2024-03-04T12:23:41.896895+0000 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:tableserver 0 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:root 0 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:session_timeout 300 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:session_autoclose 300 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:max_file_size 1099511627776 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:max_xattr_size 65536 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:required_client_features {} 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:last_failure 0 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:last_failure_osd_epoch 0 2024-03-04T12:25:24.034 INFO:teuthology.orchestra.run.smithi060.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:max_mds 5 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:in 0,1,2,3,4 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:up {0=24524,1=24487,2=24479,3=24407,4=24394} 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:failed 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:damaged 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:stopped 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:data_pools [3,4] 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:metadata_pool 2 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:inline_data disabled 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:balancer 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:bal_rank_mask -1 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:standby_count_wanted 0 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.b{0:24524} state up:active seq 3 addr [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.c{0:24463} state up:standby-replay seq 1 addr [v2:172.21.15.86:6836/472598778,v1:172.21.15.86:6837/472598778] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.i{1:24487} state up:active seq 12 addr [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T12:25:24.035 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.f{1:24451} state up:standby-replay seq 1 addr [v2:172.21.15.86:6834/2934725061,v1:172.21.15.86:6835/2934725061] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T12:25:24.036 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.h{2:24479} state up:active seq 12 addr [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T12:25:24.036 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.e{2:24443} state up:standby-replay seq 1 addr [v2:172.21.15.63:6834/2453356505,v1:172.21.15.63:6835/2453356505] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T12:25:24.036 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.k{3:24407} state up:active seq 17 addr [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T12:25:24.036 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.a{3:14643} state up:standby-replay seq 1 addr [v2:172.21.15.60:6840/4175833227,v1:172.21.15.60:6841/4175833227] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T12:25:24.036 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.l{4:24394} state up:active seq 18 addr [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T12:25:24.036 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.g{4:14631} state up:standby-replay seq 1 addr [v2:172.21.15.60:6838/2180757903,v1:172.21.15.60:6839/2180757903] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T12:25:24.036 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:24.036 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:24.036 INFO:teuthology.orchestra.run.smithi060.stdout:Standby daemons: 2024-03-04T12:25:24.036 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:24.036 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.j{-1:14559} state up:standby seq 1 addr [v2:172.21.15.60:6834/49412012,v1:172.21.15.60:6835/49412012] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T12:25:24.036 INFO:teuthology.orchestra.run.smithi060.stdout:[mds.d{-1:14571} state up:standby seq 1 addr [v2:172.21.15.60:6836/3896783798,v1:172.21.15.60:6837/3896783798] compat {c=[1],r=[1],i=[fff]}] 2024-03-04T12:25:24.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.029+0000 7f6ca2dac700 1 -- 172.21.15.60:0/1769721387 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6c880a1e10 msgr2=0x7f6c880a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:24.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.029+0000 7f6ca2dac700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6c880a1e10 0x7f6c880a42d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c101da0 tx=0x7f6c8c009040 comp rx=0 tx=0).stop 2024-03-04T12:25:24.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.029+0000 7f6ca2dac700 1 -- 172.21.15.60:0/1769721387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c9c101f40 msgr2=0x7f6c9c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:24.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.029+0000 7f6ca2dac700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c9c101f40 0x7f6c9c103320 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6c84003890 tx=0x7f6c840047b0 comp rx=0 tx=0).stop 2024-03-04T12:25:24.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.029+0000 7f6ca2dac700 1 -- 172.21.15.60:0/1769721387 shutdown_connections 2024-03-04T12:25:24.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.029+0000 7f6ca2dac700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c9c1028e0 0x7f6c9c103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:24.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.029+0000 7f6ca2dac700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c100d40 0x7f6c9c104cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:24.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.029+0000 7f6ca2dac700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6c880a1e10 0x7f6c880a42d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:24.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.029+0000 7f6ca2dac700 1 --2- 172.21.15.60:0/1769721387 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6c9c101f40 0x7f6c9c103320 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:24.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.029+0000 7f6ca2dac700 1 -- 172.21.15.60:0/1769721387 >> 172.21.15.60:0/1769721387 conn(0x7f6c9c0fc2e0 msgr2=0x7f6c9c0fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:24.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.029+0000 7f6ca2dac700 1 -- 172.21.15.60:0/1769721387 shutdown_connections 2024-03-04T12:25:24.038 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:24.029+0000 7f6ca2dac700 1 -- 172.21.15.60:0/1769721387 wait complete. 2024-03-04T12:25:24.038 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 28 2024-03-04T12:25:24.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:23 smithi086 ceph-mon[36851]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:24.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:23 smithi063 ceph-mon[36717]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:24.410 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:23 smithi060 ceph-mon[32745]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T12:25:24.766 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-04T12:25:25.018 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:25:25.139 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:24 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1769721387' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T12:25:25.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:24 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1769721387' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T12:25:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:24 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1769721387' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-04T12:25:26.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:25 smithi086 ceph-mon[36851]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:25:26.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:25 smithi063 ceph-mon[36717]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:25:26.382 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:25 smithi060 ceph-mon[32745]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:25:26.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.806+0000 7f5a88176700 1 -- 172.21.15.60:0/3149710618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a80103970 msgr2=0x7f5a8010a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:26.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.806+0000 7f5a88176700 1 --2- 172.21.15.60:0/3149710618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a80103970 0x7f5a8010a290 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5a7800b0a0 tx=0x7f5a7800b3b0 comp rx=0 tx=0).stop 2024-03-04T12:25:26.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.806+0000 7f5a88176700 1 -- 172.21.15.60:0/3149710618 shutdown_connections 2024-03-04T12:25:26.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.806+0000 7f5a88176700 1 --2- 172.21.15.60:0/3149710618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a80103970 0x7f5a8010a290 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:26.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.806+0000 7f5a88176700 1 --2- 172.21.15.60:0/3149710618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a80101e50 0x7f5a80102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:26.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.806+0000 7f5a88176700 1 --2- 172.21.15.60:0/3149710618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a80102fb0 0x7f5a80103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:26.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.806+0000 7f5a88176700 1 -- 172.21.15.60:0/3149710618 >> 172.21.15.60:0/3149710618 conn(0x7f5a800fd430 msgr2=0x7f5a800ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:26.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.806+0000 7f5a88176700 1 -- 172.21.15.60:0/3149710618 shutdown_connections 2024-03-04T12:25:26.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.806+0000 7f5a88176700 1 -- 172.21.15.60:0/3149710618 wait complete. 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a88176700 1 Processor -- start 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a88176700 1 -- start start 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a88176700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a80101e50 0x7f5a80197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a88176700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a80102fb0 0x7f5a80198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a88176700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a80103970 0x7f5a8019f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a88176700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8010da20 con 0x7f5a80103970 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a88176700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8010d8a0 con 0x7f5a80102fb0 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a88176700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8010dba0 con 0x7f5a80101e50 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a86713700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a80103970 0x7f5a8019f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a86713700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a80103970 0x7f5a8019f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48384/0 (socket says 172.21.15.60:48384) 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a86713700 1 -- 172.21.15.60:0/3868423992 learned_addr learned my addr 172.21.15.60:0/3868423992 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a85f12700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a80101e50 0x7f5a80197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.807+0000 7f5a85711700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a80102fb0 0x7f5a80198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:26.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.808+0000 7f5a86713700 1 -- 172.21.15.60:0/3868423992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a80101e50 msgr2=0x7f5a80197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:26.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.808+0000 7f5a86713700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a80101e50 0x7f5a80197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:26.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.808+0000 7f5a86713700 1 -- 172.21.15.60:0/3868423992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a80102fb0 msgr2=0x7f5a80198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:26.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.808+0000 7f5a86713700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a80102fb0 0x7f5a80198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:26.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.808+0000 7f5a86713700 1 -- 172.21.15.60:0/3868423992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8019fba0 con 0x7f5a80103970 2024-03-04T12:25:26.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.808+0000 7f5a86713700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a80103970 0x7f5a8019f500 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5a78007dc0 tx=0x7f5a78007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:26.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.808+0000 7f5a7effd700 1 -- 172.21.15.60:0/3868423992 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78021020 con 0x7f5a80103970 2024-03-04T12:25:26.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.808+0000 7f5a7effd700 1 -- 172.21.15.60:0/3868423992 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a78004e40 con 0x7f5a80103970 2024-03-04T12:25:26.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.808+0000 7f5a7effd700 1 -- 172.21.15.60:0/3868423992 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a78005510 con 0x7f5a80103970 2024-03-04T12:25:26.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.808+0000 7f5a88176700 1 -- 172.21.15.60:0/3868423992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8019fe30 con 0x7f5a80103970 2024-03-04T12:25:26.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.808+0000 7f5a88176700 1 -- 172.21.15.60:0/3868423992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a801a0290 con 0x7f5a80103970 2024-03-04T12:25:26.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.809+0000 7f5a88176700 1 -- 172.21.15.60:0/3868423992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a80068ae0 con 0x7f5a80103970 2024-03-04T12:25:26.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.811+0000 7f5a7effd700 1 -- 172.21.15.60:0/3868423992 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f5a78024020 con 0x7f5a80103970 2024-03-04T12:25:26.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.811+0000 7f5a7effd700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5a680a1e90 0x7f5a680a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:26.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.812+0000 7f5a85f12700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5a680a1e90 0x7f5a680a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:26.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.812+0000 7f5a7effd700 1 -- 172.21.15.60:0/3868423992 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f5a7800f020 con 0x7f5a80103970 2024-03-04T12:25:26.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.812+0000 7f5a85f12700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5a680a1e90 0x7f5a680a4350 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5a7000ac50 tx=0x7f5a7000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:26.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:26.815+0000 7f5a7effd700 1 -- 172.21.15.60:0/3868423992 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5a7809ccd0 con 0x7f5a80103970 2024-03-04T12:25:27.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.020+0000 7f5a88176700 1 -- 172.21.15.60:0/3868423992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f5a8002d870 con 0x7f5a80103970 2024-03-04T12:25:27.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.021+0000 7f5a7effd700 1 -- 172.21.15.60:0/3868423992 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) v1 ==== 56+0+5611 (secure 0 0 0) 0x7f5a8002d870 con 0x7f5a80103970 2024-03-04T12:25:27.027 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 93 2024-03-04T12:25:27.027 INFO:teuthology.orchestra.run.smithi060.stdout:fsid a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:25:27.027 INFO:teuthology.orchestra.run.smithi060.stdout:created 2024-03-04T12:15:08.418686+0000 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:modified 2024-03-04T12:25:06.929248+0000 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:crush_version 27 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:full_ratio 0.95 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:backfillfull_ratio 0.9 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:nearfull_ratio 0.85 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:require_min_compat_client luminous 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:min_compat_client luminous 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:require_osd_release squid 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:stretch_mode_enabled false 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-03-04T12:25:27.028 INFO:teuthology.orchestra.run.smithi060.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:max_osd 12 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6802/4258653647,v1:172.21.15.60:6803/4258653647] [v2:172.21.15.60:6804/4258653647,v1:172.21.15.60:6805/4258653647] exists,up c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.1 up in weight 1 up_from 13 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6800/1936533350,v1:172.21.15.63:6801/1936533350] [v2:172.21.15.63:6802/1936533350,v1:172.21.15.63:6803/1936533350] exists,up 59fe4466-e274-4543-a2a4-f95588613bf4 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6800/3773043222,v1:172.21.15.86:6801/3773043222] [v2:172.21.15.86:6802/3773043222,v1:172.21.15.86:6803/3773043222] exists,up d05a35c6-2586-47fa-8172-6597e7925861 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.3 up in weight 1 up_from 25 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6810/1173200157,v1:172.21.15.60:6811/1173200157] [v2:172.21.15.60:6812/1173200157,v1:172.21.15.60:6813/1173200157] exists,up a823b593-a7b5-45ff-8765-cac26276dedd 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.4 up in weight 1 up_from 33 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6808/793955,v1:172.21.15.63:6809/793955] [v2:172.21.15.63:6810/793955,v1:172.21.15.63:6811/793955] exists,up 75da5356-1007-4b12-89dc-5e91ec454df8 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6808/1744734157,v1:172.21.15.86:6809/1744734157] [v2:172.21.15.86:6810/1744734157,v1:172.21.15.86:6811/1744734157] exists,up 6d29eae4-5669-4e1e-a94f-ddb37bcc1a72 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.6 up in weight 1 up_from 46 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6818/285200459,v1:172.21.15.60:6819/285200459] [v2:172.21.15.60:6820/285200459,v1:172.21.15.60:6821/285200459] exists,up 4794189a-f45f-4859-99de-d7578ff20141 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6816/942963290,v1:172.21.15.63:6817/942963290] [v2:172.21.15.63:6818/942963290,v1:172.21.15.63:6819/942963290] exists,up 86f9c999-69b0-4c33-b39c-5c307ec78e58 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6816/1013701944,v1:172.21.15.86:6817/1013701944] [v2:172.21.15.86:6818/1013701944,v1:172.21.15.86:6819/1013701944] exists,up 5848de7e-e01b-4ae8-a060-ead59bf03d90 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.9 up in weight 1 up_from 67 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.60:6826/1552052384,v1:172.21.15.60:6827/1552052384] [v2:172.21.15.60:6828/1552052384,v1:172.21.15.60:6829/1552052384] exists,up 92ab3932-3338-42a8-ae91-6d348af0c507 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6824/4239140227,v1:172.21.15.63:6825/4239140227] [v2:172.21.15.63:6826/4239140227,v1:172.21.15.63:6827/4239140227] exists,up 9fa85ae1-3566-4486-8586-05edcecc7848 2024-03-04T12:25:27.029 INFO:teuthology.orchestra.run.smithi060.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6824/2397477201,v1:172.21.15.86:6825/2397477201] [v2:172.21.15.86:6826/2397477201,v1:172.21.15.86:6827/2397477201] exists,up 344b7daf-897d-4217-8cba-192c507c09e1 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_upmap_items 2.6 [8,11] 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_upmap_items 2.8 [2,11] 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_upmap_items 3.e [5,9] 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_upmap_items 3.1e [3,9] 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_upmap_items 3.27 [4,7] 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_upmap_items 3.31 [4,8] 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_upmap_items 3.39 [5,6] 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_upmap_items 3.3c [4,9] 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_upmap_items 4.8 [5,8] 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_upmap_items 4.12 [9,10] 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_upmap_items 4.14 [9,8] 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:pg_upmap_items 4.1c [9,8] 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:blocklist 172.21.15.60:0/2852203003 expires 2024-03-05T12:16:04.073665+0000 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:blocklist 172.21.15.60:0/3868523955 expires 2024-03-05T12:16:04.073665+0000 2024-03-04T12:25:27.030 INFO:teuthology.orchestra.run.smithi060.stdout:blocklist 172.21.15.60:6801/2246601804 expires 2024-03-05T12:16:04.073665+0000 2024-03-04T12:25:27.031 INFO:teuthology.orchestra.run.smithi060.stdout:blocklist 172.21.15.60:6800/2246601804 expires 2024-03-05T12:16:04.073665+0000 2024-03-04T12:25:27.031 INFO:teuthology.orchestra.run.smithi060.stdout:blocklist 172.21.15.60:0/758894979 expires 2024-03-05T12:15:37.282929+0000 2024-03-04T12:25:27.031 INFO:teuthology.orchestra.run.smithi060.stdout:blocklist 172.21.15.60:6801/2061680026 expires 2024-03-05T12:15:37.282929+0000 2024-03-04T12:25:27.031 INFO:teuthology.orchestra.run.smithi060.stdout:blocklist 172.21.15.60:0/3401854556 expires 2024-03-05T12:15:37.282929+0000 2024-03-04T12:25:27.031 INFO:teuthology.orchestra.run.smithi060.stdout:blocklist 172.21.15.60:6800/2061680026 expires 2024-03-05T12:15:37.282929+0000 2024-03-04T12:25:27.031 INFO:teuthology.orchestra.run.smithi060.stdout:blocklist 172.21.15.60:0/3539227088 expires 2024-03-05T12:15:37.282929+0000 2024-03-04T12:25:27.031 INFO:teuthology.orchestra.run.smithi060.stdout:blocklist 172.21.15.60:0/2028967685 expires 2024-03-05T12:16:04.073665+0000 2024-03-04T12:25:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.023+0000 7f5a88176700 1 -- 172.21.15.60:0/3868423992 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5a680a1e90 msgr2=0x7f5a680a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.023+0000 7f5a88176700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5a680a1e90 0x7f5a680a4350 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5a7000ac50 tx=0x7f5a7000a380 comp rx=0 tx=0).stop 2024-03-04T12:25:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.023+0000 7f5a88176700 1 -- 172.21.15.60:0/3868423992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a80103970 msgr2=0x7f5a8019f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.023+0000 7f5a88176700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a80103970 0x7f5a8019f500 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5a78007dc0 tx=0x7f5a78007df0 comp rx=0 tx=0).stop 2024-03-04T12:25:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.023+0000 7f5a88176700 1 -- 172.21.15.60:0/3868423992 shutdown_connections 2024-03-04T12:25:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.023+0000 7f5a88176700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5a680a1e90 0x7f5a680a4350 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.023+0000 7f5a88176700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a80103970 0x7f5a8019f500 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.023+0000 7f5a88176700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a80101e50 0x7f5a80197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.023+0000 7f5a88176700 1 --2- 172.21.15.60:0/3868423992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a80102fb0 0x7f5a80198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:27.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.023+0000 7f5a88176700 1 -- 172.21.15.60:0/3868423992 >> 172.21.15.60:0/3868423992 conn(0x7f5a800fd430 msgr2=0x7f5a80106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:27.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.023+0000 7f5a88176700 1 -- 172.21.15.60:0/3868423992 shutdown_connections 2024-03-04T12:25:27.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.023+0000 7f5a88176700 1 -- 172.21.15.60:0/3868423992 wait complete. 2024-03-04T12:25:27.710 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-04T12:25:27.720 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:25:27.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.824+0000 7f6542c27640 1 -- 172.21.15.60:0/2985726808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f653c101de0 msgr2=0x7f653c1041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:27.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.824+0000 7f6542c27640 1 --2- 172.21.15.60:0/2985726808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f653c101de0 0x7f653c1041d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f6530009900 tx=0x7f653002bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 -- 172.21.15.60:0/2985726808 shutdown_connections 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 --2- 172.21.15.60:0/2985726808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f653c107040 0x7f653c109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 --2- 172.21.15.60:0/2985726808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f653c104710 0x7f653c106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 --2- 172.21.15.60:0/2985726808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f653c101de0 0x7f653c1041d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 -- 172.21.15.60:0/2985726808 >> 172.21.15.60:0/2985726808 conn(0x7f653c0fb7f0 msgr2=0x7f653c0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 -- 172.21.15.60:0/2985726808 shutdown_connections 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 -- 172.21.15.60:0/2985726808 wait complete. 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 Processor -- start 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 -- start start 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f653c101de0 0x7f653c197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f653c104710 0x7f653c197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f653c107040 0x7f653c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:27.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c10d040 con 0x7f653c107040 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c10cec0 con 0x7f653c104710 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f6542c27640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f653c10d1c0 con 0x7f653c101de0 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.825+0000 7f65411c4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f653c107040 0x7f653c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f65411c4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f653c107040 0x7f653c19ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48398/0 (socket says 172.21.15.60:48398) 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f65411c4640 1 -- 172.21.15.60:0/979147021 learned_addr learned my addr 172.21.15.60:0/979147021 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f65409c3640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f653c101de0 0x7f653c197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f653bfff640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f653c104710 0x7f653c197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f65411c4640 1 -- 172.21.15.60:0/979147021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f653c101de0 msgr2=0x7f653c197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f65411c4640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f653c101de0 0x7f653c197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f65411c4640 1 -- 172.21.15.60:0/979147021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f653c104710 msgr2=0x7f653c197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f65411c4640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f653c104710 0x7f653c197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f65411c4640 1 -- 172.21.15.60:0/979147021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c19f240 con 0x7f653c107040 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f65411c4640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f653c107040 0x7f653c19ebd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f653400cce0 tx=0x7f653400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f6539ffb640 1 -- 172.21.15.60:0/979147021 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653400f450 con 0x7f653c107040 2024-03-04T12:25:27.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f6539ffb640 1 -- 172.21.15.60:0/979147021 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f653400fad0 con 0x7f653c107040 2024-03-04T12:25:27.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f6542c27640 1 -- 172.21.15.60:0/979147021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f653c19f530 con 0x7f653c107040 2024-03-04T12:25:27.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f6539ffb640 1 -- 172.21.15.60:0/979147021 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6534018c10 con 0x7f653c107040 2024-03-04T12:25:27.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.826+0000 7f6542c27640 1 -- 172.21.15.60:0/979147021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f653c19fa80 con 0x7f653c107040 2024-03-04T12:25:27.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.827+0000 7f6542c27640 1 -- 172.21.15.60:0/979147021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f653c0fd2e0 con 0x7f653c107040 2024-03-04T12:25:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.828+0000 7f6539ffb640 1 -- 172.21.15.60:0/979147021 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f653400f5f0 con 0x7f653c107040 2024-03-04T12:25:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.829+0000 7f6539ffb640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f652c0a1e00 0x7f652c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.829+0000 7f65409c3640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f652c0a1e00 0x7f652c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.829+0000 7f6539ffb640 1 -- 172.21.15.60:0/979147021 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f65340ce740 con 0x7f653c107040 2024-03-04T12:25:27.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.829+0000 7f65409c3640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f652c0a1e00 0x7f652c0a42c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f653002c240 tx=0x7f653002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:27.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:27.833+0000 7f6539ffb640 1 -- 172.21.15.60:0/979147021 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6534014030 con 0x7f653c107040 2024-03-04T12:25:28.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.018+0000 7f6542c27640 1 -- 172.21.15.60:0/979147021 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f653c061f30 con 0x7f653c107040 2024-03-04T12:25:28.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.020+0000 7f6539ffb640 1 -- 172.21.15.60:0/979147021 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24562 (secure 0 0 0) 0x7f653409b780 con 0x7f653c107040 2024-03-04T12:25:28.028 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:28.028 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":93,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:25:06.929248+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:25:28.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.023+0000 7f6542c27640 1 -- 172.21.15.60:0/979147021 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f652c0a1e00 msgr2=0x7f652c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.023+0000 7f6542c27640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f652c0a1e00 0x7f652c0a42c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f653002c240 tx=0x7f653002df90 comp rx=0 tx=0).stop 2024-03-04T12:25:28.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.023+0000 7f6542c27640 1 -- 172.21.15.60:0/979147021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f653c107040 msgr2=0x7f653c19ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.023+0000 7f6542c27640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f653c107040 0x7f653c19ebd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f653400cce0 tx=0x7f653400a620 comp rx=0 tx=0).stop 2024-03-04T12:25:28.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.023+0000 7f6542c27640 1 -- 172.21.15.60:0/979147021 shutdown_connections 2024-03-04T12:25:28.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.023+0000 7f6542c27640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f652c0a1e00 0x7f652c0a42c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.023+0000 7f6542c27640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f653c107040 0x7f653c19ebd0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.023+0000 7f6542c27640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f653c104710 0x7f653c197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.023+0000 7f6542c27640 1 --2- 172.21.15.60:0/979147021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f653c101de0 0x7f653c197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.023+0000 7f6542c27640 1 -- 172.21.15.60:0/979147021 >> 172.21.15.60:0/979147021 conn(0x7f653c0fb7f0 msgr2=0x7f653c102740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:28.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.023+0000 7f6542c27640 1 -- 172.21.15.60:0/979147021 shutdown_connections 2024-03-04T12:25:28.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.023+0000 7f6542c27640 1 -- 172.21.15.60:0/979147021 wait complete. 2024-03-04T12:25:28.042 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T12:23:10.438928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T12:23:11.472998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T12:23:15.162500+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T12:25:28.042 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:25:28.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.140+0000 7f3184efc640 1 -- 172.21.15.60:0/2947812132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3180101e00 msgr2=0x7f3180108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.140+0000 7f3184efc640 1 --2- 172.21.15.60:0/2947812132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3180101e00 0x7f3180108690 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3178009ca0 tx=0x7f317802c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.140+0000 7f3184efc640 1 -- 172.21.15.60:0/2947812132 shutdown_connections 2024-03-04T12:25:28.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.140+0000 7f3184efc640 1 --2- 172.21.15.60:0/2947812132 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3180101e00 0x7f3180108690 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.140+0000 7f3184efc640 1 --2- 172.21.15.60:0/2947812132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3180101440 0x7f31801018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.140+0000 7f3184efc640 1 --2- 172.21.15.60:0/2947812132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3180100250 0x7f3180100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.141+0000 7f3184efc640 1 -- 172.21.15.60:0/2947812132 >> 172.21.15.60:0/2947812132 conn(0x7f31800fb7c0 msgr2=0x7f31800fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:28.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.141+0000 7f3184efc640 1 -- 172.21.15.60:0/2947812132 shutdown_connections 2024-03-04T12:25:28.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.141+0000 7f3184efc640 1 -- 172.21.15.60:0/2947812132 wait complete. 2024-03-04T12:25:28.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.141+0000 7f3184efc640 1 Processor -- start 2024-03-04T12:25:28.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.141+0000 7f3184efc640 1 -- start start 2024-03-04T12:25:28.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.141+0000 7f3184efc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3180100250 0x7f31801975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.141+0000 7f3184efc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3180101440 0x7f3180197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.141+0000 7f3184efc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3180101e00 0x7f318019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.141+0000 7f3184efc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318010bf20 con 0x7f3180101440 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.141+0000 7f3184efc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318010bda0 con 0x7f3180100250 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.141+0000 7f3184efc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318010c0a0 con 0x7f3180101e00 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f3177fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3180101440 0x7f3180197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f3177fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3180101440 0x7f3180197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48410/0 (socket says 172.21.15.60:48410) 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f3177fff640 1 -- 172.21.15.60:0/2676507778 learned_addr learned my addr 172.21.15.60:0/2676507778 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f317e59c640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3180100250 0x7f31801975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f3177fff640 1 -- 172.21.15.60:0/2676507778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3180101e00 msgr2=0x7f318019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f3177fff640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3180101e00 0x7f318019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f3177fff640 1 -- 172.21.15.60:0/2676507778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3180100250 msgr2=0x7f31801975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f3177fff640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3180100250 0x7f31801975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f3177fff640 1 -- 172.21.15.60:0/2676507778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318019f1f0 con 0x7f3180101440 2024-03-04T12:25:28.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f317e59c640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3180100250 0x7f31801975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:28.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f3177fff640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3180101440 0x7f3180197b10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f316800b4a0 tx=0x7f316800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:28.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f31777fe640 1 -- 172.21.15.60:0/2676507778 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3168014070 con 0x7f3180101440 2024-03-04T12:25:28.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f31777fe640 1 -- 172.21.15.60:0/2676507778 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3168010af0 con 0x7f3180101440 2024-03-04T12:25:28.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f3184efc640 1 -- 172.21.15.60:0/2676507778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318019f4e0 con 0x7f3180101440 2024-03-04T12:25:28.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f31777fe640 1 -- 172.21.15.60:0/2676507778 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3168017d10 con 0x7f3180101440 2024-03-04T12:25:28.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.142+0000 7f3184efc640 1 -- 172.21.15.60:0/2676507778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318019fa80 con 0x7f3180101440 2024-03-04T12:25:28.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.143+0000 7f3184efc640 1 -- 172.21.15.60:0/2676507778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318019fd40 con 0x7f3180101440 2024-03-04T12:25:28.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.144+0000 7f31777fe640 1 -- 172.21.15.60:0/2676507778 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f3168017070 con 0x7f3180101440 2024-03-04T12:25:28.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.145+0000 7f31777fe640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f316c0a1e00 0x7f316c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:28.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.145+0000 7f317e59c640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f316c0a1e00 0x7f316c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:28.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.145+0000 7f31777fe640 1 -- 172.21.15.60:0/2676507778 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f31680cd990 con 0x7f3180101440 2024-03-04T12:25:28.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.145+0000 7f317e59c640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f316c0a1e00 0x7f316c0a42c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f31801012a0 tx=0x7f317000a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:28.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.148+0000 7f31777fe640 1 -- 172.21.15.60:0/2676507778 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f316809a9d0 con 0x7f3180101440 2024-03-04T12:25:28.295 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:27 smithi060 ceph-mon[32745]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:25:28.296 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:27 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3868423992' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-04T12:25:28.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:27 smithi086 ceph-mon[36851]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:25:28.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:27 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3868423992' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-04T12:25:28.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:27 smithi063 ceph-mon[36717]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:25:28.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:27 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3868423992' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-04T12:25:28.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.349+0000 7f3184efc640 1 -- 172.21.15.60:0/2676507778 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f318019ff10 con 0x7f3180101440 2024-03-04T12:25:28.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.350+0000 7f31777fe640 1 -- 172.21.15.60:0/2676507778 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f316809a2d0 con 0x7f3180101440 2024-03-04T12:25:28.357 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:25:28.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.352+0000 7f3184efc640 1 -- 172.21.15.60:0/2676507778 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f316c0a1e00 msgr2=0x7f316c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.352+0000 7f3184efc640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f316c0a1e00 0x7f316c0a42c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f31801012a0 tx=0x7f317000a5e0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.352+0000 7f3184efc640 1 -- 172.21.15.60:0/2676507778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3180101440 msgr2=0x7f3180197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.352+0000 7f3184efc640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3180101440 0x7f3180197b10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f316800b4a0 tx=0x7f316800b960 comp rx=0 tx=0).stop 2024-03-04T12:25:28.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.352+0000 7f3184efc640 1 -- 172.21.15.60:0/2676507778 shutdown_connections 2024-03-04T12:25:28.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.352+0000 7f3184efc640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f316c0a1e00 0x7f316c0a42c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.352+0000 7f3184efc640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3180101e00 0x7f318019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.352+0000 7f3184efc640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3180101440 0x7f3180197b10 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.352+0000 7f3184efc640 1 --2- 172.21.15.60:0/2676507778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3180100250 0x7f31801975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.352+0000 7f3184efc640 1 -- 172.21.15.60:0/2676507778 >> 172.21.15.60:0/2676507778 conn(0x7f31800fb7c0 msgr2=0x7f31800fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:28.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.352+0000 7f3184efc640 1 -- 172.21.15.60:0/2676507778 shutdown_connections 2024-03-04T12:25:28.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.352+0000 7f3184efc640 1 -- 172.21.15.60:0/2676507778 wait complete. 2024-03-04T12:25:28.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T12:25:28.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.469+0000 7f5e9e64d640 1 -- 172.21.15.60:0/56682059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e98102f90 msgr2=0x7f5e98109820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.469+0000 7f5e9e64d640 1 --2- 172.21.15.60:0/56682059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e98102f90 0x7f5e98109820 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5e90009cd0 tx=0x7f5e9002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.469+0000 7f5e9e64d640 1 -- 172.21.15.60:0/56682059 shutdown_connections 2024-03-04T12:25:28.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.469+0000 7f5e9e64d640 1 --2- 172.21.15.60:0/56682059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e98102f90 0x7f5e98109820 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.469+0000 7f5e9e64d640 1 --2- 172.21.15.60:0/56682059 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e981025d0 0x7f5e98102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.469+0000 7f5e9e64d640 1 --2- 172.21.15.60:0/56682059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e980ff600 0x7f5e980ffa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.469+0000 7f5e9e64d640 1 -- 172.21.15.60:0/56682059 >> 172.21.15.60:0/56682059 conn(0x7f5e980faf50 msgr2=0x7f5e980fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.469+0000 7f5e9e64d640 1 -- 172.21.15.60:0/56682059 shutdown_connections 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.469+0000 7f5e9e64d640 1 -- 172.21.15.60:0/56682059 wait complete. 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.469+0000 7f5e9e64d640 1 Processor -- start 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9e64d640 1 -- start start 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9e64d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e980ff600 0x7f5e981975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9e64d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e981025d0 0x7f5e98197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9e64d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e98102f90 0x7f5e9819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9e64d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9810cff0 con 0x7f5e98102f90 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9e64d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9810ce70 con 0x7f5e981025d0 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9e64d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9810d170 con 0x7f5e980ff600 2024-03-04T12:25:28.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9cbea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e98102f90 0x7f5e9819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9cbea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e98102f90 0x7f5e9819eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48434/0 (socket says 172.21.15.60:48434) 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9cbea640 1 -- 172.21.15.60:0/1878829949 learned_addr learned my addr 172.21.15.60:0/1878829949 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e97fff640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e980ff600 0x7f5e981975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e977fe640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e981025d0 0x7f5e98197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9cbea640 1 -- 172.21.15.60:0/1878829949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e980ff600 msgr2=0x7f5e981975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9cbea640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e980ff600 0x7f5e981975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9cbea640 1 -- 172.21.15.60:0/1878829949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e981025d0 msgr2=0x7f5e98197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9cbea640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e981025d0 0x7f5e98197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9cbea640 1 -- 172.21.15.60:0/1878829949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9819f1d0 con 0x7f5e98102f90 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.470+0000 7f5e9cbea640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e98102f90 0x7f5e9819eb60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5e90005e30 tx=0x7f5e900346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.471+0000 7f5e957fa640 1 -- 172.21.15.60:0/1878829949 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90041020 con 0x7f5e98102f90 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.471+0000 7f5e9e64d640 1 -- 172.21.15.60:0/1878829949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9819f460 con 0x7f5e98102f90 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.471+0000 7f5e957fa640 1 -- 172.21.15.60:0/1878829949 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e90032890 con 0x7f5e98102f90 2024-03-04T12:25:28.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.471+0000 7f5e957fa640 1 -- 172.21.15.60:0/1878829949 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90039040 con 0x7f5e98102f90 2024-03-04T12:25:28.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.471+0000 7f5e9e64d640 1 -- 172.21.15.60:0/1878829949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9819f950 con 0x7f5e98102f90 2024-03-04T12:25:28.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.472+0000 7f5e9e64d640 1 -- 172.21.15.60:0/1878829949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e98108700 con 0x7f5e98102f90 2024-03-04T12:25:28.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.473+0000 7f5e957fa640 1 -- 172.21.15.60:0/1878829949 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f5e90044020 con 0x7f5e98102f90 2024-03-04T12:25:28.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.473+0000 7f5e957fa640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5e800a1d80 0x7f5e800a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:28.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.474+0000 7f5e97fff640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5e800a1d80 0x7f5e800a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:28.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.474+0000 7f5e957fa640 1 -- 172.21.15.60:0/1878829949 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f5e900efee0 con 0x7f5e98102f90 2024-03-04T12:25:28.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.474+0000 7f5e97fff640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5e800a1d80 0x7f5e800a4240 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5e98069870 tx=0x7f5e88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:28.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.478+0000 7f5e957fa640 1 -- 172.21.15.60:0/1878829949 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5e9004b020 con 0x7f5e98102f90 2024-03-04T12:25:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.660+0000 7f5e9e64d640 1 -- 172.21.15.60:0/1878829949 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5e98061f30 con 0x7f5e98102f90 2024-03-04T12:25:28.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.661+0000 7f5e957fa640 1 -- 172.21.15.60:0/1878829949 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5e900bcf80 con 0x7f5e98102f90 2024-03-04T12:25:28.669 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:28.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.663+0000 7f5e9e64d640 1 -- 172.21.15.60:0/1878829949 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5e800a1d80 msgr2=0x7f5e800a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.663+0000 7f5e9e64d640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5e800a1d80 0x7f5e800a4240 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5e98069870 tx=0x7f5e88005f90 comp rx=0 tx=0).stop 2024-03-04T12:25:28.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.663+0000 7f5e9e64d640 1 -- 172.21.15.60:0/1878829949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e98102f90 msgr2=0x7f5e9819eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.663+0000 7f5e9e64d640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e98102f90 0x7f5e9819eb60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5e90005e30 tx=0x7f5e900346f0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.663+0000 7f5e9e64d640 1 -- 172.21.15.60:0/1878829949 shutdown_connections 2024-03-04T12:25:28.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.663+0000 7f5e9e64d640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5e800a1d80 0x7f5e800a4240 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.663+0000 7f5e9e64d640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5e98102f90 0x7f5e9819eb60 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.663+0000 7f5e9e64d640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5e981025d0 0x7f5e98197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.663+0000 7f5e9e64d640 1 --2- 172.21.15.60:0/1878829949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e980ff600 0x7f5e981975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.663+0000 7f5e9e64d640 1 -- 172.21.15.60:0/1878829949 >> 172.21.15.60:0/1878829949 conn(0x7f5e980faf50 msgr2=0x7f5e981058d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:28.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.663+0000 7f5e9e64d640 1 -- 172.21.15.60:0/1878829949 shutdown_connections 2024-03-04T12:25:28.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.663+0000 7f5e9e64d640 1 -- 172.21.15.60:0/1878829949 wait complete. 2024-03-04T12:25:28.682 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T12:25:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.780+0000 7f8ab804b640 1 -- 172.21.15.60:0/72347128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ab0102550 msgr2=0x7f8ab01029d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.780+0000 7f8ab804b640 1 --2- 172.21.15.60:0/72347128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ab0102550 0x7f8ab01029d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009980 tx=0x7f8aa0030060 comp rx=0 tx=0).stop 2024-03-04T12:25:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.780+0000 7f8ab804b640 1 -- 172.21.15.60:0/72347128 shutdown_connections 2024-03-04T12:25:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.780+0000 7f8ab804b640 1 --2- 172.21.15.60:0/72347128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ab0102f10 0x7f8ab01097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.780+0000 7f8ab804b640 1 --2- 172.21.15.60:0/72347128 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ab0102550 0x7f8ab01029d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.780+0000 7f8ab804b640 1 --2- 172.21.15.60:0/72347128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ab0101360 0x7f8ab0101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.780+0000 7f8ab804b640 1 -- 172.21.15.60:0/72347128 >> 172.21.15.60:0/72347128 conn(0x7f8ab00fc8f0 msgr2=0x7f8ab00fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.780+0000 7f8ab804b640 1 -- 172.21.15.60:0/72347128 shutdown_connections 2024-03-04T12:25:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.780+0000 7f8ab804b640 1 -- 172.21.15.60:0/72347128 wait complete. 2024-03-04T12:25:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab804b640 1 Processor -- start 2024-03-04T12:25:28.790 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab804b640 1 -- start start 2024-03-04T12:25:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab804b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ab0101360 0x7f8ab0195430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab804b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ab0102550 0x7f8ab0195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab804b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ab0102f10 0x7f8ab019c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab804b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab010ae00 con 0x7f8ab0101360 2024-03-04T12:25:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab804b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab010ac80 con 0x7f8ab0102f10 2024-03-04T12:25:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab804b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab010af80 con 0x7f8ab0102550 2024-03-04T12:25:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab5de7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ab0101360 0x7f8ab0195430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab5de7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ab0101360 0x7f8ab0195430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48442/0 (socket says 172.21.15.60:48442) 2024-03-04T12:25:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab5de7640 1 -- 172.21.15.60:0/1837018521 learned_addr learned my addr 172.21.15.60:0/1837018521 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab65e8640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ab0102f10 0x7f8ab019c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab55e6640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ab0102550 0x7f8ab0195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:28.791 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab5de7640 1 -- 172.21.15.60:0/1837018521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ab0102550 msgr2=0x7f8ab0195970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab5de7640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ab0102550 0x7f8ab0195970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab5de7640 1 -- 172.21.15.60:0/1837018521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ab0102f10 msgr2=0x7f8ab019c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab5de7640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ab0102f10 0x7f8ab019c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab5de7640 1 -- 172.21.15.60:0/1837018521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab019d050 con 0x7f8ab0101360 2024-03-04T12:25:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.781+0000 7f8ab65e8640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ab0102f10 0x7f8ab019c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.782+0000 7f8ab5de7640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ab0101360 0x7f8ab0195430 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00d770 tx=0x7f8aac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.782+0000 7f8aa6ffd640 1 -- 172.21.15.60:0/1837018521 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac017070 con 0x7f8ab0101360 2024-03-04T12:25:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.782+0000 7f8aa6ffd640 1 -- 172.21.15.60:0/1837018521 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8aac009c30 con 0x7f8ab0101360 2024-03-04T12:25:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.782+0000 7f8ab804b640 1 -- 172.21.15.60:0/1837018521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab019d340 con 0x7f8ab0101360 2024-03-04T12:25:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.782+0000 7f8aa6ffd640 1 -- 172.21.15.60:0/1837018521 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac003dd0 con 0x7f8ab0101360 2024-03-04T12:25:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.782+0000 7f8ab804b640 1 -- 172.21.15.60:0/1837018521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab019d8b0 con 0x7f8ab0101360 2024-03-04T12:25:28.792 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.783+0000 7f8ab804b640 1 -- 172.21.15.60:0/1837018521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab0108680 con 0x7f8ab0101360 2024-03-04T12:25:28.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.784+0000 7f8aa6ffd640 1 -- 172.21.15.60:0/1837018521 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f8aac003fb0 con 0x7f8ab0101360 2024-03-04T12:25:28.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.784+0000 7f8aa6ffd640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8a9c0a1e00 0x7f8a9c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:28.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.784+0000 7f8ab55e6640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8a9c0a1e00 0x7f8a9c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:28.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.784+0000 7f8aa6ffd640 1 -- 172.21.15.60:0/1837018521 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f8aac0cdb10 con 0x7f8ab0101360 2024-03-04T12:25:28.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.785+0000 7f8ab55e6640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8a9c0a1e00 0x7f8a9c0a42c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0006010 tx=0x7f8aa0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:28.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.789+0000 7f8aa6ffd640 1 -- 172.21.15.60:0/1837018521 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8aac009de0 con 0x7f8ab0101360 2024-03-04T12:25:28.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.974+0000 7f8ab804b640 1 -- 172.21.15.60:0/1837018521 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8ab0061f60 con 0x7f8ab0101360 2024-03-04T12:25:28.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.975+0000 7f8aa6ffd640 1 -- 172.21.15.60:0/1837018521 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8aac09abb0 con 0x7f8ab0101360 2024-03-04T12:25:28.982 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-03-04T12:25:28.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.977+0000 7f8ab804b640 1 -- 172.21.15.60:0/1837018521 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8a9c0a1e00 msgr2=0x7f8a9c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.977+0000 7f8ab804b640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8a9c0a1e00 0x7f8a9c0a42c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0006010 tx=0x7f8aa0000bc0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.984 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.977+0000 7f8ab804b640 1 -- 172.21.15.60:0/1837018521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ab0101360 msgr2=0x7f8ab0195430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:28.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.977+0000 7f8ab804b640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ab0101360 0x7f8ab0195430 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00d770 tx=0x7f8aac00dc30 comp rx=0 tx=0).stop 2024-03-04T12:25:28.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.977+0000 7f8ab804b640 1 -- 172.21.15.60:0/1837018521 shutdown_connections 2024-03-04T12:25:28.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.977+0000 7f8ab804b640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8a9c0a1e00 0x7f8a9c0a42c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.977+0000 7f8ab804b640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ab0102f10 0x7f8ab019c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.977+0000 7f8ab804b640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ab0102550 0x7f8ab0195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.977+0000 7f8ab804b640 1 --2- 172.21.15.60:0/1837018521 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8ab0101360 0x7f8ab0195430 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:28.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.977+0000 7f8ab804b640 1 -- 172.21.15.60:0/1837018521 >> 172.21.15.60:0/1837018521 conn(0x7f8ab00fc8f0 msgr2=0x7f8ab0105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:28.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.977+0000 7f8ab804b640 1 -- 172.21.15.60:0/1837018521 shutdown_connections 2024-03-04T12:25:28.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:28.977+0000 7f8ab804b640 1 -- 172.21.15.60:0/1837018521 wait complete. 2024-03-04T12:25:28.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T12:25:29.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.094+0000 7fd451268640 1 -- 172.21.15.60:0/1845500453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd44c101dd0 msgr2=0x7fd44c1041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.094+0000 7fd451268640 1 --2- 172.21.15.60:0/1845500453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd44c101dd0 0x7fd44c1041c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd43c009900 tx=0x7fd43c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.094+0000 7fd451268640 1 -- 172.21.15.60:0/1845500453 shutdown_connections 2024-03-04T12:25:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.094+0000 7fd451268640 1 --2- 172.21.15.60:0/1845500453 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd44c107030 0x7fd44c109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.094+0000 7fd451268640 1 --2- 172.21.15.60:0/1845500453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd44c104700 0x7fd44c106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.094+0000 7fd451268640 1 --2- 172.21.15.60:0/1845500453 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd44c101dd0 0x7fd44c1041c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.094+0000 7fd451268640 1 -- 172.21.15.60:0/1845500453 >> 172.21.15.60:0/1845500453 conn(0x7fd44c0fb7c0 msgr2=0x7fd44c0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.094+0000 7fd451268640 1 -- 172.21.15.60:0/1845500453 shutdown_connections 2024-03-04T12:25:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.094+0000 7fd451268640 1 -- 172.21.15.60:0/1845500453 wait complete. 2024-03-04T12:25:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd451268640 1 Processor -- start 2024-03-04T12:25:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd451268640 1 -- start start 2024-03-04T12:25:29.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd451268640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd44c101dd0 0x7fd44c101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd451268640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd44c104700 0x7fd44c0ffd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd451268640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd44c107030 0x7fd44c1002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd451268640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c10d030 con 0x7fd44c104700 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd451268640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c10ceb0 con 0x7fd44c107030 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd451268640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c10d1b0 con 0x7fd44c101dd0 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd443fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd44c104700 0x7fd44c0ffd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd443fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd44c104700 0x7fd44c0ffd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48456/0 (socket says 172.21.15.60:48456) 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd443fff640 1 -- 172.21.15.60:0/1812672817 learned_addr learned my addr 172.21.15.60:0/1812672817 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd44b7fe640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd44c107030 0x7fd44c1002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd44affd640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd44c101dd0 0x7fd44c101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd443fff640 1 -- 172.21.15.60:0/1812672817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd44c101dd0 msgr2=0x7fd44c101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd443fff640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd44c101dd0 0x7fd44c101740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd443fff640 1 -- 172.21.15.60:0/1812672817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd44c107030 msgr2=0x7fd44c1002c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd443fff640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd44c107030 0x7fd44c1002c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd443fff640 1 -- 172.21.15.60:0/1812672817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c100b50 con 0x7fd44c104700 2024-03-04T12:25:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.095+0000 7fd44b7fe640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd44c107030 0x7fd44c1002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.096+0000 7fd443fff640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd44c104700 0x7fd44c0ffd80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd43400c970 tx=0x7fd43400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.096+0000 7fd448ff9640 1 -- 172.21.15.60:0/1812672817 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434007bb0 con 0x7fd44c104700 2024-03-04T12:25:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.096+0000 7fd451268640 1 -- 172.21.15.60:0/1812672817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd44c1a17c0 con 0x7fd44c104700 2024-03-04T12:25:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.096+0000 7fd448ff9640 1 -- 172.21.15.60:0/1812672817 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd434007d50 con 0x7fd44c104700 2024-03-04T12:25:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.096+0000 7fd448ff9640 1 -- 172.21.15.60:0/1812672817 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4340187a0 con 0x7fd44c104700 2024-03-04T12:25:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.096+0000 7fd451268640 1 -- 172.21.15.60:0/1812672817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd44c1a1cc0 con 0x7fd44c104700 2024-03-04T12:25:29.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.097+0000 7fd451268640 1 -- 172.21.15.60:0/1812672817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd44c0fd2b0 con 0x7fd44c104700 2024-03-04T12:25:29.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.098+0000 7fd448ff9640 1 -- 172.21.15.60:0/1812672817 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fd43400f450 con 0x7fd44c104700 2024-03-04T12:25:29.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.098+0000 7fd448ff9640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd4380a1d80 0x7fd4380a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:29.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.099+0000 7fd44affd640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd4380a1d80 0x7fd4380a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:29.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.099+0000 7fd448ff9640 1 -- 172.21.15.60:0/1812672817 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fd4340ce2a0 con 0x7fd44c104700 2024-03-04T12:25:29.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.099+0000 7fd44affd640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd4380a1d80 0x7fd4380a4240 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0096c0 tx=0x7fd43c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:29.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.103+0000 7fd448ff9640 1 -- 172.21.15.60:0/1812672817 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd434014030 con 0x7fd44c104700 2024-03-04T12:25:29.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.283+0000 7fd451268640 1 -- 172.21.15.60:0/1812672817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd44c061f30 con 0x7fd44c104700 2024-03-04T12:25:29.291 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.284+0000 7fd448ff9640 1 -- 172.21.15.60:0/1812672817 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd43409acf0 con 0x7fd44c104700 2024-03-04T12:25:29.291 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:29.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.286+0000 7fd451268640 1 -- 172.21.15.60:0/1812672817 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd4380a1d80 msgr2=0x7fd4380a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.286+0000 7fd451268640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd4380a1d80 0x7fd4380a4240 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0096c0 tx=0x7fd43c02df90 comp rx=0 tx=0).stop 2024-03-04T12:25:29.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.286+0000 7fd451268640 1 -- 172.21.15.60:0/1812672817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd44c104700 msgr2=0x7fd44c0ffd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.286+0000 7fd451268640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd44c104700 0x7fd44c0ffd80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd43400c970 tx=0x7fd43400ce30 comp rx=0 tx=0).stop 2024-03-04T12:25:29.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.286+0000 7fd451268640 1 -- 172.21.15.60:0/1812672817 shutdown_connections 2024-03-04T12:25:29.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.286+0000 7fd451268640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd4380a1d80 0x7fd4380a4240 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.286+0000 7fd451268640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd44c107030 0x7fd44c1002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.286+0000 7fd451268640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd44c104700 0x7fd44c0ffd80 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.286+0000 7fd451268640 1 --2- 172.21.15.60:0/1812672817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd44c101dd0 0x7fd44c101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.286+0000 7fd451268640 1 -- 172.21.15.60:0/1812672817 >> 172.21.15.60:0/1812672817 conn(0x7fd44c0fb7c0 msgr2=0x7fd44c102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:29.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.286+0000 7fd451268640 1 -- 172.21.15.60:0/1812672817 shutdown_connections 2024-03-04T12:25:29.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.286+0000 7fd451268640 1 -- 172.21.15.60:0/1812672817 wait complete. 2024-03-04T12:25:29.303 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:28 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/979147021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:29.303 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:28 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2676507778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:29.303 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:28 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1878829949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:29.304 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:25:29.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:28 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/979147021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:29.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:28 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2676507778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:29.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:28 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1878829949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:29.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:28 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/979147021' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:29.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:28 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2676507778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:29.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:28 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1878829949' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:29.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.400+0000 7f0d6329a640 1 -- 172.21.15.60:0/905679034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5c1002d0 msgr2=0x7f0d5c1006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.400+0000 7f0d6329a640 1 --2- 172.21.15.60:0/905679034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5c1002d0 0x7f0d5c1006f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009900 tx=0x7f0d5402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:29.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.400+0000 7f0d6329a640 1 -- 172.21.15.60:0/905679034 shutdown_connections 2024-03-04T12:25:29.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.400+0000 7f0d6329a640 1 --2- 172.21.15.60:0/905679034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d5c101e80 0x7f0d5c108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.400+0000 7f0d6329a640 1 --2- 172.21.15.60:0/905679034 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d5c1014c0 0x7f0d5c101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.400+0000 7f0d6329a640 1 --2- 172.21.15.60:0/905679034 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5c1002d0 0x7f0d5c1006f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.400+0000 7f0d6329a640 1 -- 172.21.15.60:0/905679034 >> 172.21.15.60:0/905679034 conn(0x7f0d5c0fb860 msgr2=0x7f0d5c0fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:29.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.400+0000 7f0d6329a640 1 -- 172.21.15.60:0/905679034 shutdown_connections 2024-03-04T12:25:29.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.400+0000 7f0d6329a640 1 -- 172.21.15.60:0/905679034 wait complete. 2024-03-04T12:25:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d6329a640 1 Processor -- start 2024-03-04T12:25:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d6329a640 1 -- start start 2024-03-04T12:25:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d6329a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d5c1002d0 0x7f0d5c104260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d6329a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d5c1014c0 0x7f0d5c1028a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d6329a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5c101e80 0x7f0d5c102de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d6329a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c10bef0 con 0x7f0d5c101e80 2024-03-04T12:25:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d6329a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c10bd70 con 0x7f0d5c1014c0 2024-03-04T12:25:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d6329a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c10c070 con 0x7f0d5c1002d0 2024-03-04T12:25:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d61837640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5c101e80 0x7f0d5c102de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:29.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d61036640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d5c1002d0 0x7f0d5c104260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d61837640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5c101e80 0x7f0d5c102de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48464/0 (socket says 172.21.15.60:48464) 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d61837640 1 -- 172.21.15.60:0/3151562726 learned_addr learned my addr 172.21.15.60:0/3151562726 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d60835640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d5c1014c0 0x7f0d5c1028a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d61837640 1 -- 172.21.15.60:0/3151562726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d5c1002d0 msgr2=0x7f0d5c104260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d61837640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d5c1002d0 0x7f0d5c104260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d61837640 1 -- 172.21.15.60:0/3151562726 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d5c1014c0 msgr2=0x7f0d5c1028a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d61837640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d5c1014c0 0x7f0d5c1028a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d61837640 1 -- 172.21.15.60:0/3151562726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c103670 con 0x7f0d5c101e80 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.401+0000 7f0d61837640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5c101e80 0x7f0d5c102de0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00cad0 tx=0x7f0d4c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.402+0000 7f0d5a7fc640 1 -- 172.21.15.60:0/3151562726 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c00ae20 con 0x7f0d5c101e80 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.402+0000 7f0d5a7fc640 1 -- 172.21.15.60:0/3151562726 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d4c005020 con 0x7f0d5c101e80 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.402+0000 7f0d6329a640 1 -- 172.21.15.60:0/3151562726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5c1a17c0 con 0x7f0d5c101e80 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.402+0000 7f0d5a7fc640 1 -- 172.21.15.60:0/3151562726 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c00f6f0 con 0x7f0d5c101e80 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.402+0000 7f0d6329a640 1 -- 172.21.15.60:0/3151562726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c1a1d10 con 0x7f0d5c101e80 2024-03-04T12:25:29.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.403+0000 7f0d6329a640 1 -- 172.21.15.60:0/3151562726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5c1a1fd0 con 0x7f0d5c101e80 2024-03-04T12:25:29.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.404+0000 7f0d5a7fc640 1 -- 172.21.15.60:0/3151562726 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f0d4c0051d0 con 0x7f0d5c101e80 2024-03-04T12:25:29.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.404+0000 7f0d5a7fc640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0d440a1d80 0x7f0d440a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:29.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.404+0000 7f0d61036640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0d440a1d80 0x7f0d440a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:29.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.404+0000 7f0d5a7fc640 1 -- 172.21.15.60:0/3151562726 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f0d4c0ce430 con 0x7f0d5c101e80 2024-03-04T12:25:29.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.405+0000 7f0d61036640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0d440a1d80 0x7f0d440a4240 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0d54000c00 tx=0x7f0d54005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:29.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.409+0000 7f0d5a7fc640 1 -- 172.21.15.60:0/3151562726 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0d4c014030 con 0x7f0d5c101e80 2024-03-04T12:25:29.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.630+0000 7f0d6329a640 1 -- 172.21.15.60:0/3151562726 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d5c1a2360 con 0x7f0d5c101e80 2024-03-04T12:25:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.631+0000 7f0d5a7fc640 1 -- 172.21.15.60:0/3151562726 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11373 (secure 0 0 0) 0x7f0d4c0aee20 con 0x7f0d5c101e80 2024-03-04T12:25:29.638 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:29.638 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:23:41.896895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:25:29.638 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 28 2024-03-04T12:25:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.633+0000 7f0d6329a640 1 -- 172.21.15.60:0/3151562726 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0d440a1d80 msgr2=0x7f0d440a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.633+0000 7f0d6329a640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0d440a1d80 0x7f0d440a4240 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0d54000c00 tx=0x7f0d54005fd0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.633+0000 7f0d6329a640 1 -- 172.21.15.60:0/3151562726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5c101e80 msgr2=0x7f0d5c102de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.633+0000 7f0d6329a640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5c101e80 0x7f0d5c102de0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00cad0 tx=0x7f0d4c00cf90 comp rx=0 tx=0).stop 2024-03-04T12:25:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.633+0000 7f0d6329a640 1 -- 172.21.15.60:0/3151562726 shutdown_connections 2024-03-04T12:25:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.633+0000 7f0d6329a640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0d440a1d80 0x7f0d440a4240 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.633+0000 7f0d6329a640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0d5c101e80 0x7f0d5c102de0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.633+0000 7f0d6329a640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d5c1014c0 0x7f0d5c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.633+0000 7f0d6329a640 1 --2- 172.21.15.60:0/3151562726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d5c1002d0 0x7f0d5c104260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.633+0000 7f0d6329a640 1 -- 172.21.15.60:0/3151562726 >> 172.21.15.60:0/3151562726 conn(0x7f0d5c0fb860 msgr2=0x7f0d5c0fd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:29.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.633+0000 7f0d6329a640 1 -- 172.21.15.60:0/3151562726 shutdown_connections 2024-03-04T12:25:29.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.633+0000 7f0d6329a640 1 -- 172.21.15.60:0/3151562726 wait complete. 2024-03-04T12:25:29.651 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:25:29.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.747+0000 7f9670f69640 1 -- 172.21.15.60:0/1218757074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f966c100260 msgr2=0x7f966c100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.747+0000 7f9670f69640 1 --2- 172.21.15.60:0/1218757074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f966c100260 0x7f966c100680 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9664009900 tx=0x7f966402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:29.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.747+0000 7f9670f69640 1 -- 172.21.15.60:0/1218757074 shutdown_connections 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.747+0000 7f9670f69640 1 --2- 172.21.15.60:0/1218757074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f966c101e10 0x7f966c1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.747+0000 7f9670f69640 1 --2- 172.21.15.60:0/1218757074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f966c101450 0x7f966c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.747+0000 7f9670f69640 1 --2- 172.21.15.60:0/1218757074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f966c100260 0x7f966c100680 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.747+0000 7f9670f69640 1 -- 172.21.15.60:0/1218757074 >> 172.21.15.60:0/1218757074 conn(0x7f966c0fb7f0 msgr2=0x7f966c0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.747+0000 7f9670f69640 1 -- 172.21.15.60:0/1218757074 shutdown_connections 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.747+0000 7f9670f69640 1 -- 172.21.15.60:0/1218757074 wait complete. 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.747+0000 7f9670f69640 1 Processor -- start 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f9670f69640 1 -- start start 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f9670f69640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f966c100260 0x7f966c06a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f9670f69640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f966c101450 0x7f966c19c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f9670f69640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f966c101e10 0x7f966c19e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f9670f69640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c10bef0 con 0x7f966c101e10 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f9670f69640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c10bd70 con 0x7f966c100260 2024-03-04T12:25:29.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f9670f69640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c10c070 con 0x7f966c101450 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f966ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f966c101e10 0x7f966c19e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f966ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f966c101e10 0x7f966c19e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48482/0 (socket says 172.21.15.60:48482) 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f966ad9d640 1 -- 172.21.15.60:0/2241144049 learned_addr learned my addr 172.21.15.60:0/2241144049 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f966a59c640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f966c100260 0x7f966c06a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f9669d9b640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f966c101450 0x7f966c19c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f966ad9d640 1 -- 172.21.15.60:0/2241144049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f966c101450 msgr2=0x7f966c19c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f966ad9d640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f966c101450 0x7f966c19c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f966ad9d640 1 -- 172.21.15.60:0/2241144049 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f966c100260 msgr2=0x7f966c06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f966ad9d640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f966c100260 0x7f966c06a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f966ad9d640 1 -- 172.21.15.60:0/2241144049 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c19f270 con 0x7f966c101e10 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f9669d9b640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f966c101450 0x7f966c19c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.748+0000 7f966ad9d640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f966c101e10 0x7f966c19e9b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f965c00cce0 tx=0x7f965c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.749+0000 7f96637fe640 1 -- 172.21.15.60:0/2241144049 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c00f450 con 0x7f966c101e10 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.749+0000 7f9670f69640 1 -- 172.21.15.60:0/2241144049 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966c19f560 con 0x7f966c101e10 2024-03-04T12:25:29.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.749+0000 7f96637fe640 1 -- 172.21.15.60:0/2241144049 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f965c00fad0 con 0x7f966c101e10 2024-03-04T12:25:29.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.749+0000 7f96637fe640 1 -- 172.21.15.60:0/2241144049 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965c018c10 con 0x7f966c101e10 2024-03-04T12:25:29.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.749+0000 7f9670f69640 1 -- 172.21.15.60:0/2241144049 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966c19fa60 con 0x7f966c101e10 2024-03-04T12:25:29.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.750+0000 7f9670f69640 1 -- 172.21.15.60:0/2241144049 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f966c107580 con 0x7f966c101e10 2024-03-04T12:25:29.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.751+0000 7f96637fe640 1 -- 172.21.15.60:0/2241144049 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f965c00f5f0 con 0x7f966c101e10 2024-03-04T12:25:29.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.751+0000 7f96637fe640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f96580a1d80 0x7f96580a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:29.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.751+0000 7f966a59c640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f96580a1d80 0x7f96580a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:29.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.751+0000 7f96637fe640 1 -- 172.21.15.60:0/2241144049 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f965c0ce820 con 0x7f966c101e10 2024-03-04T12:25:29.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.752+0000 7f966a59c640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f96580a1d80 0x7f96580a4240 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9664000c00 tx=0x7f9664005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:29.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.756+0000 7f96637fe640 1 -- 172.21.15.60:0/2241144049 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f965c014030 con 0x7f966c101e10 2024-03-04T12:25:29.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.942+0000 7f9670f69640 1 -- 172.21.15.60:0/2241144049 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f966c061f30 con 0x7f966c101e10 2024-03-04T12:25:29.951 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.944+0000 7f96637fe640 1 -- 172.21.15.60:0/2241144049 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24562 (secure 0 0 0) 0x7f965c09b8c0 con 0x7f966c101e10 2024-03-04T12:25:29.952 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:29.952 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":93,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:25:06.929248+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:25:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.946+0000 7f9670f69640 1 -- 172.21.15.60:0/2241144049 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f96580a1d80 msgr2=0x7f96580a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.953 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.946+0000 7f9670f69640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f96580a1d80 0x7f96580a4240 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9664000c00 tx=0x7f9664005fd0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.946+0000 7f9670f69640 1 -- 172.21.15.60:0/2241144049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f966c101e10 msgr2=0x7f966c19e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.946+0000 7f9670f69640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f966c101e10 0x7f966c19e9b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f965c00cce0 tx=0x7f965c00a620 comp rx=0 tx=0).stop 2024-03-04T12:25:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.946+0000 7f9670f69640 1 -- 172.21.15.60:0/2241144049 shutdown_connections 2024-03-04T12:25:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.946+0000 7f9670f69640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f96580a1d80 0x7f96580a4240 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.946+0000 7f9670f69640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f966c101e10 0x7f966c19e9b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.946+0000 7f9670f69640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f966c101450 0x7f966c19c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.946+0000 7f9670f69640 1 --2- 172.21.15.60:0/2241144049 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f966c100260 0x7f966c06a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.946+0000 7f9670f69640 1 -- 172.21.15.60:0/2241144049 >> 172.21.15.60:0/2241144049 conn(0x7f966c0fb7f0 msgr2=0x7f966c104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.946+0000 7f9670f69640 1 -- 172.21.15.60:0/2241144049 shutdown_connections 2024-03-04T12:25:29.954 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:29.946+0000 7f9670f69640 1 -- 172.21.15.60:0/2241144049 wait complete. 2024-03-04T12:25:29.965 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T12:23:10.438928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T12:23:11.472998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T12:23:15.162500+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T12:25:29.965 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:25:30.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.064+0000 7f62351a2640 1 -- 172.21.15.60:0/454326799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6230101170 msgr2=0x7f6230101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.064+0000 7f62351a2640 1 --2- 172.21.15.60:0/454326799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6230101170 0x7f6230101590 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6224009900 tx=0x7f622402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:30.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.064+0000 7f62351a2640 1 -- 172.21.15.60:0/454326799 shutdown_connections 2024-03-04T12:25:30.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.064+0000 7f62351a2640 1 --2- 172.21.15.60:0/454326799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6230102d20 0x7f62301095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.064+0000 7f62351a2640 1 --2- 172.21.15.60:0/454326799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6230102360 0x7f62301027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.064+0000 7f62351a2640 1 --2- 172.21.15.60:0/454326799 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6230101170 0x7f6230101590 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.064+0000 7f62351a2640 1 -- 172.21.15.60:0/454326799 >> 172.21.15.60:0/454326799 conn(0x7f62300fc6e0 msgr2=0x7f62300feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:30.072 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.064+0000 7f62351a2640 1 -- 172.21.15.60:0/454326799 shutdown_connections 2024-03-04T12:25:30.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.064+0000 7f62351a2640 1 -- 172.21.15.60:0/454326799 wait complete. 2024-03-04T12:25:30.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.065+0000 7f62351a2640 1 Processor -- start 2024-03-04T12:25:30.073 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.065+0000 7f62351a2640 1 -- start start 2024-03-04T12:25:30.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.065+0000 7f62351a2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6230101170 0x7f6230197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:30.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.065+0000 7f62351a2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6230102360 0x7f6230197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:30.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.065+0000 7f62351a2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6230102d20 0x7f623019eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:30.075 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.065+0000 7f62351a2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623010ce40 con 0x7f6230102360 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.065+0000 7f62351a2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623010ccc0 con 0x7f6230101170 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.065+0000 7f62351a2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623010cfc0 con 0x7f6230102d20 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.065+0000 7f622f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6230102360 0x7f6230197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.065+0000 7f622f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6230102360 0x7f6230197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48502/0 (socket says 172.21.15.60:48502) 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.065+0000 7f622f7fe640 1 -- 172.21.15.60:0/3354812904 learned_addr learned my addr 172.21.15.60:0/3354812904 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.065+0000 7f622ffff640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6230101170 0x7f6230197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.066+0000 7f62349a1640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6230102d20 0x7f623019eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.066+0000 7f622f7fe640 1 -- 172.21.15.60:0/3354812904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6230102d20 msgr2=0x7f623019eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.066+0000 7f622f7fe640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6230102d20 0x7f623019eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.066+0000 7f622f7fe640 1 -- 172.21.15.60:0/3354812904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6230101170 msgr2=0x7f6230197510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.066+0000 7f622f7fe640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6230101170 0x7f6230197510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.066+0000 7f622f7fe640 1 -- 172.21.15.60:0/3354812904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623019f130 con 0x7f6230102360 2024-03-04T12:25:30.076 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.066+0000 7f622f7fe640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6230102360 0x7f6230197a50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f622000d950 tx=0x7f622000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:30.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.066+0000 7f622d7fa640 1 -- 172.21.15.60:0/3354812904 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6220017070 con 0x7f6230102360 2024-03-04T12:25:30.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.066+0000 7f622d7fa640 1 -- 172.21.15.60:0/3354812904 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6220009c30 con 0x7f6230102360 2024-03-04T12:25:30.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.066+0000 7f62351a2640 1 -- 172.21.15.60:0/3354812904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623019f420 con 0x7f6230102360 2024-03-04T12:25:30.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.066+0000 7f622d7fa640 1 -- 172.21.15.60:0/3354812904 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622000f6b0 con 0x7f6230102360 2024-03-04T12:25:30.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.066+0000 7f62351a2640 1 -- 172.21.15.60:0/3354812904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623019f970 con 0x7f6230102360 2024-03-04T12:25:30.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.067+0000 7f62351a2640 1 -- 172.21.15.60:0/3354812904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6230108490 con 0x7f6230102360 2024-03-04T12:25:30.077 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.068+0000 7f622d7fa640 1 -- 172.21.15.60:0/3354812904 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f62200109d0 con 0x7f6230102360 2024-03-04T12:25:30.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.069+0000 7f622d7fa640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f62180a1d80 0x7f62180a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:30.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.069+0000 7f622ffff640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f62180a1d80 0x7f62180a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:30.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.069+0000 7f622d7fa640 1 -- 172.21.15.60:0/3354812904 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f62200cdde0 con 0x7f6230102360 2024-03-04T12:25:30.078 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.069+0000 7f622ffff640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f62180a1d80 0x7f62180a4240 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6224000c00 tx=0x7f6224005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:30.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.073+0000 7f622d7fa640 1 -- 172.21.15.60:0/3354812904 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6220009de0 con 0x7f6230102360 2024-03-04T12:25:30.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.253+0000 7f62351a2640 1 -- 172.21.15.60:0/3354812904 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6230061f30 con 0x7f6230102360 2024-03-04T12:25:30.261 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.253+0000 7f622d7fa640 1 -- 172.21.15.60:0/3354812904 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f622009ae80 con 0x7f6230102360 2024-03-04T12:25:30.261 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:25:30.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.255+0000 7f62351a2640 1 -- 172.21.15.60:0/3354812904 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f62180a1d80 msgr2=0x7f62180a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.255+0000 7f62351a2640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f62180a1d80 0x7f62180a4240 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6224000c00 tx=0x7f6224005fd0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.255+0000 7f62351a2640 1 -- 172.21.15.60:0/3354812904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6230102360 msgr2=0x7f6230197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.255+0000 7f62351a2640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6230102360 0x7f6230197a50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f622000d950 tx=0x7f622000de10 comp rx=0 tx=0).stop 2024-03-04T12:25:30.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.255+0000 7f62351a2640 1 -- 172.21.15.60:0/3354812904 shutdown_connections 2024-03-04T12:25:30.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.255+0000 7f62351a2640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f62180a1d80 0x7f62180a4240 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.255+0000 7f62351a2640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6230102d20 0x7f623019eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.255+0000 7f62351a2640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6230102360 0x7f6230197a50 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.255+0000 7f62351a2640 1 --2- 172.21.15.60:0/3354812904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6230101170 0x7f6230197510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.255+0000 7f62351a2640 1 -- 172.21.15.60:0/3354812904 >> 172.21.15.60:0/3354812904 conn(0x7f62300fc6e0 msgr2=0x7f6230105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:30.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.256+0000 7f62351a2640 1 -- 172.21.15.60:0/3354812904 shutdown_connections 2024-03-04T12:25:30.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.256+0000 7f62351a2640 1 -- 172.21.15.60:0/3354812904 wait complete. 2024-03-04T12:25:30.273 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:29 smithi060 ceph-mon[32745]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:25:30.273 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:29 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1837018521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:30.273 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:29 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1812672817' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:30.273 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:29 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3151562726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:30.273 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:29 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2241144049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:30.274 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T12:25:30.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:29 smithi086 ceph-mon[36851]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:25:30.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:29 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1837018521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:30.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:29 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1812672817' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:30.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:29 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3151562726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:30.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:29 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2241144049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:30.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:29 smithi063 ceph-mon[36717]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-03-04T12:25:30.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:29 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1837018521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:30.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:29 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1812672817' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:30.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:29 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3151562726' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:30.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:29 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2241144049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:30.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.372+0000 7f0be8a2b640 1 -- 172.21.15.60:0/4200491675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be4101dd0 msgr2=0x7f0be41041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.372+0000 7f0be8a2b640 1 --2- 172.21.15.60:0/4200491675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be4101dd0 0x7f0be41041c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc009900 tx=0x7f0bdc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:30.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.372+0000 7f0be8a2b640 1 -- 172.21.15.60:0/4200491675 shutdown_connections 2024-03-04T12:25:30.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.372+0000 7f0be8a2b640 1 --2- 172.21.15.60:0/4200491675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0be4107030 0x7f0be4109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.372+0000 7f0be8a2b640 1 --2- 172.21.15.60:0/4200491675 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0be4104700 0x7f0be4106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.372+0000 7f0be8a2b640 1 --2- 172.21.15.60:0/4200491675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be4101dd0 0x7f0be41041c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.372+0000 7f0be8a2b640 1 -- 172.21.15.60:0/4200491675 >> 172.21.15.60:0/4200491675 conn(0x7f0be40fb7e0 msgr2=0x7f0be40fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:30.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.372+0000 7f0be8a2b640 1 -- 172.21.15.60:0/4200491675 shutdown_connections 2024-03-04T12:25:30.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.372+0000 7f0be8a2b640 1 -- 172.21.15.60:0/4200491675 wait complete. 2024-03-04T12:25:30.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.373+0000 7f0be8a2b640 1 Processor -- start 2024-03-04T12:25:30.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.373+0000 7f0be8a2b640 1 -- start start 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.373+0000 7f0be8a2b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0be4101dd0 0x7f0be4197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.373+0000 7f0be8a2b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be4104700 0x7f0be4197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.373+0000 7f0be8a2b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0be4107030 0x7f0be419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.373+0000 7f0be8a2b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be410cfe0 con 0x7f0be4104700 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.373+0000 7f0be8a2b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be410ce60 con 0x7f0be4101dd0 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.373+0000 7f0be8a2b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be410d160 con 0x7f0be4107030 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.373+0000 7f0be1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be4104700 0x7f0be4197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.373+0000 7f0be1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be4104700 0x7f0be4197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48510/0 (socket says 172.21.15.60:48510) 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.373+0000 7f0be1d9b640 1 -- 172.21.15.60:0/427593708 learned_addr learned my addr 172.21.15.60:0/427593708 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0be259c640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0be4101dd0 0x7f0be4197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0be2d9d640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0be4107030 0x7f0be419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0be1d9b640 1 -- 172.21.15.60:0/427593708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0be4107030 msgr2=0x7f0be419ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0be1d9b640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0be4107030 0x7f0be419ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0be1d9b640 1 -- 172.21.15.60:0/427593708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0be4101dd0 msgr2=0x7f0be4197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0be1d9b640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0be4101dd0 0x7f0be4197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0be1d9b640 1 -- 172.21.15.60:0/427593708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be419f270 con 0x7f0be4104700 2024-03-04T12:25:30.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0be1d9b640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be4104700 0x7f0be4197b90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc00d770 tx=0x7f0bcc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:30.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0bdb7fe640 1 -- 172.21.15.60:0/427593708 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc017070 con 0x7f0be4104700 2024-03-04T12:25:30.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0be8a2b640 1 -- 172.21.15.60:0/427593708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be419f560 con 0x7f0be4104700 2024-03-04T12:25:30.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0bdb7fe640 1 -- 172.21.15.60:0/427593708 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bcc009c30 con 0x7f0be4104700 2024-03-04T12:25:30.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0bdb7fe640 1 -- 172.21.15.60:0/427593708 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc00f620 con 0x7f0be4104700 2024-03-04T12:25:30.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.374+0000 7f0be8a2b640 1 -- 172.21.15.60:0/427593708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be419fb00 con 0x7f0be4104700 2024-03-04T12:25:30.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.375+0000 7f0be8a2b640 1 -- 172.21.15.60:0/427593708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be40fd2d0 con 0x7f0be4104700 2024-03-04T12:25:30.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.376+0000 7f0bdb7fe640 1 -- 172.21.15.60:0/427593708 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f0bcc01a070 con 0x7f0be4104700 2024-03-04T12:25:30.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.377+0000 7f0bdb7fe640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0bd00a1d80 0x7f0bd00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:30.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.377+0000 7f0be259c640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0bd00a1d80 0x7f0bd00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:30.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.377+0000 7f0bdb7fe640 1 -- 172.21.15.60:0/427593708 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f0bcc0cdc40 con 0x7f0be4104700 2024-03-04T12:25:30.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.378+0000 7f0be259c640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0bd00a1d80 0x7f0bd00a4240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc02c7b0 tx=0x7f0bdc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:30.389 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.382+0000 7f0bdb7fe640 1 -- 172.21.15.60:0/427593708 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0bcc009de0 con 0x7f0be4104700 2024-03-04T12:25:30.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.562+0000 7f0be8a2b640 1 -- 172.21.15.60:0/427593708 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0be4061f30 con 0x7f0be4104700 2024-03-04T12:25:30.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.562+0000 7f0bdb7fe640 1 -- 172.21.15.60:0/427593708 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0bcc09ace0 con 0x7f0be4104700 2024-03-04T12:25:30.570 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:30.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.564+0000 7f0be8a2b640 1 -- 172.21.15.60:0/427593708 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0bd00a1d80 msgr2=0x7f0bd00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.564+0000 7f0be8a2b640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0bd00a1d80 0x7f0bd00a4240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc02c7b0 tx=0x7f0bdc005fd0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.564+0000 7f0be8a2b640 1 -- 172.21.15.60:0/427593708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be4104700 msgr2=0x7f0be4197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.564+0000 7f0be8a2b640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be4104700 0x7f0be4197b90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc00d770 tx=0x7f0bcc00dc30 comp rx=0 tx=0).stop 2024-03-04T12:25:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.564+0000 7f0be8a2b640 1 -- 172.21.15.60:0/427593708 shutdown_connections 2024-03-04T12:25:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.564+0000 7f0be8a2b640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0bd00a1d80 0x7f0bd00a4240 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.564+0000 7f0be8a2b640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0be4107030 0x7f0be419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.564+0000 7f0be8a2b640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0be4104700 0x7f0be4197b90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.564+0000 7f0be8a2b640 1 --2- 172.21.15.60:0/427593708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0be4101dd0 0x7f0be4197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.564+0000 7f0be8a2b640 1 -- 172.21.15.60:0/427593708 >> 172.21.15.60:0/427593708 conn(0x7f0be40fb7e0 msgr2=0x7f0be4102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.564+0000 7f0be8a2b640 1 -- 172.21.15.60:0/427593708 shutdown_connections 2024-03-04T12:25:30.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.564+0000 7f0be8a2b640 1 -- 172.21.15.60:0/427593708 wait complete. 2024-03-04T12:25:30.582 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T12:25:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.680+0000 7f5a977c4640 1 -- 172.21.15.60:0/3195135675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a90101dd0 msgr2=0x7f5a901041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.680+0000 7f5a977c4640 1 --2- 172.21.15.60:0/3195135675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a90101dd0 0x7f5a901041c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f5a88009900 tx=0x7f5a8802bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.680+0000 7f5a977c4640 1 -- 172.21.15.60:0/3195135675 shutdown_connections 2024-03-04T12:25:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.680+0000 7f5a977c4640 1 --2- 172.21.15.60:0/3195135675 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a90107030 0x7f5a90109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.680+0000 7f5a977c4640 1 --2- 172.21.15.60:0/3195135675 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a90104700 0x7f5a90106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.680+0000 7f5a977c4640 1 --2- 172.21.15.60:0/3195135675 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a90101dd0 0x7f5a901041c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.680+0000 7f5a977c4640 1 -- 172.21.15.60:0/3195135675 >> 172.21.15.60:0/3195135675 conn(0x7f5a900fb7c0 msgr2=0x7f5a900fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:30.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a977c4640 1 -- 172.21.15.60:0/3195135675 shutdown_connections 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a977c4640 1 -- 172.21.15.60:0/3195135675 wait complete. 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a977c4640 1 Processor -- start 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a977c4640 1 -- start start 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a977c4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a90101dd0 0x7f5a90101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a977c4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a90104700 0x7f5a900ffd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a977c4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a90107030 0x7f5a901002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a977c4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9010cfe0 con 0x7f5a90107030 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a977c4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9010ce60 con 0x7f5a90101dd0 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a977c4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9010d160 con 0x7f5a90104700 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a95d61640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a90107030 0x7f5a901002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a95d61640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a90107030 0x7f5a901002c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48532/0 (socket says 172.21.15.60:48532) 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a95d61640 1 -- 172.21.15.60:0/1929307920 learned_addr learned my addr 172.21.15.60:0/1929307920 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:30.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.681+0000 7f5a94d5f640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a90104700 0x7f5a900ffd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.682+0000 7f5a95560640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a90101dd0 0x7f5a90101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.682+0000 7f5a95d61640 1 -- 172.21.15.60:0/1929307920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a90104700 msgr2=0x7f5a900ffd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.682+0000 7f5a95d61640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a90104700 0x7f5a900ffd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.682+0000 7f5a95d61640 1 -- 172.21.15.60:0/1929307920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a90101dd0 msgr2=0x7f5a90101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.682+0000 7f5a95d61640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a90101dd0 0x7f5a90101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.682+0000 7f5a95d61640 1 -- 172.21.15.60:0/1929307920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a90100b50 con 0x7f5a90107030 2024-03-04T12:25:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.682+0000 7f5a95d61640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a90107030 0x7f5a901002c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c00eaa0 tx=0x7f5a8c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.682+0000 7f5a867fc640 1 -- 172.21.15.60:0/1929307920 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c00cdb0 con 0x7f5a90107030 2024-03-04T12:25:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.682+0000 7f5a867fc640 1 -- 172.21.15.60:0/1929307920 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a8c005020 con 0x7f5a90107030 2024-03-04T12:25:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.682+0000 7f5a867fc640 1 -- 172.21.15.60:0/1929307920 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c0116b0 con 0x7f5a90107030 2024-03-04T12:25:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.682+0000 7f5a977c4640 1 -- 172.21.15.60:0/1929307920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a901a1780 con 0x7f5a90107030 2024-03-04T12:25:30.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.682+0000 7f5a977c4640 1 -- 172.21.15.60:0/1929307920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a901a1c80 con 0x7f5a90107030 2024-03-04T12:25:30.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.683+0000 7f5a977c4640 1 -- 172.21.15.60:0/1929307920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a900fd2b0 con 0x7f5a90107030 2024-03-04T12:25:30.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.684+0000 7f5a867fc640 1 -- 172.21.15.60:0/1929307920 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f5a8c005560 con 0x7f5a90107030 2024-03-04T12:25:30.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.685+0000 7f5a867fc640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5a7c0a1d80 0x7f5a7c0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:30.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.685+0000 7f5a95560640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5a7c0a1d80 0x7f5a7c0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:30.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.685+0000 7f5a867fc640 1 -- 172.21.15.60:0/1929307920 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f5a8c0ce7f0 con 0x7f5a90107030 2024-03-04T12:25:30.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.686+0000 7f5a95560640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5a7c0a1d80 0x7f5a7c0a4240 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5a88006010 tx=0x7f5a88000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:30.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.689+0000 7f5a867fc640 1 -- 172.21.15.60:0/1929307920 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5a8c016030 con 0x7f5a90107030 2024-03-04T12:25:30.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.870+0000 7f5a977c4640 1 -- 172.21.15.60:0/1929307920 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5a9002ceb0 con 0x7f5a90107030 2024-03-04T12:25:30.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.871+0000 7f5a867fc640 1 -- 172.21.15.60:0/1929307920 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5a8c09b890 con 0x7f5a90107030 2024-03-04T12:25:30.878 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-03-04T12:25:30.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.872+0000 7f5a977c4640 1 -- 172.21.15.60:0/1929307920 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5a7c0a1d80 msgr2=0x7f5a7c0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.872+0000 7f5a977c4640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5a7c0a1d80 0x7f5a7c0a4240 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5a88006010 tx=0x7f5a88000bc0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.873+0000 7f5a977c4640 1 -- 172.21.15.60:0/1929307920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a90107030 msgr2=0x7f5a901002c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:30.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.873+0000 7f5a977c4640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a90107030 0x7f5a901002c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c00eaa0 tx=0x7f5a8c00ef60 comp rx=0 tx=0).stop 2024-03-04T12:25:30.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.873+0000 7f5a977c4640 1 -- 172.21.15.60:0/1929307920 shutdown_connections 2024-03-04T12:25:30.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.873+0000 7f5a977c4640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f5a7c0a1d80 0x7f5a7c0a4240 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.873+0000 7f5a977c4640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5a90107030 0x7f5a901002c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.873+0000 7f5a977c4640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a90104700 0x7f5a900ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.873+0000 7f5a977c4640 1 --2- 172.21.15.60:0/1929307920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a90101dd0 0x7f5a90101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:30.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.873+0000 7f5a977c4640 1 -- 172.21.15.60:0/1929307920 >> 172.21.15.60:0/1929307920 conn(0x7f5a900fb7c0 msgr2=0x7f5a90102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:30.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.873+0000 7f5a977c4640 1 -- 172.21.15.60:0/1929307920 shutdown_connections 2024-03-04T12:25:30.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.873+0000 7f5a977c4640 1 -- 172.21.15.60:0/1929307920 wait complete. 2024-03-04T12:25:30.891 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T12:25:31.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.990+0000 7fe6901ab640 1 -- 172.21.15.60:0/4000845922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe688102fb0 msgr2=0x7fe688109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.990+0000 7fe6901ab640 1 --2- 172.21.15.60:0/4000845922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe688102fb0 0x7fe688109840 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe680009cd0 tx=0x7fe68002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.990+0000 7fe6901ab640 1 -- 172.21.15.60:0/4000845922 shutdown_connections 2024-03-04T12:25:31.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.990+0000 7fe6901ab640 1 --2- 172.21.15.60:0/4000845922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe688102fb0 0x7fe688109840 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.990+0000 7fe6901ab640 1 --2- 172.21.15.60:0/4000845922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6881025f0 0x7fe688102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.990+0000 7fe6901ab640 1 --2- 172.21.15.60:0/4000845922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6880ff620 0x7fe6880ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.990+0000 7fe6901ab640 1 -- 172.21.15.60:0/4000845922 >> 172.21.15.60:0/4000845922 conn(0x7fe6880faf50 msgr2=0x7fe6880fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:31.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.990+0000 7fe6901ab640 1 -- 172.21.15.60:0/4000845922 shutdown_connections 2024-03-04T12:25:31.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.990+0000 7fe6901ab640 1 -- 172.21.15.60:0/4000845922 wait complete. 2024-03-04T12:25:31.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe6901ab640 1 Processor -- start 2024-03-04T12:25:31.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe6901ab640 1 -- start start 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe6901ab640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6880ff620 0x7fe688197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe6901ab640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6881025f0 0x7fe688197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe6901ab640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe688102fb0 0x7fe68819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe6901ab640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68810d070 con 0x7fe6880ff620 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe6901ab640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68810cef0 con 0x7fe688102fb0 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe6901ab640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe68810d1f0 con 0x7fe6881025f0 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe68df47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6880ff620 0x7fe688197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe68df47640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6880ff620 0x7fe688197600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48546/0 (socket says 172.21.15.60:48546) 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe68df47640 1 -- 172.21.15.60:0/95808077 learned_addr learned my addr 172.21.15.60:0/95808077 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe68e748640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe688102fb0 0x7fe68819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe68d746640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6881025f0 0x7fe688197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe68df47640 1 -- 172.21.15.60:0/95808077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6881025f0 msgr2=0x7fe688197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe68df47640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6881025f0 0x7fe688197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe68df47640 1 -- 172.21.15.60:0/95808077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe688102fb0 msgr2=0x7fe68819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe68df47640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe688102fb0 0x7fe68819ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.991+0000 7fe68df47640 1 -- 172.21.15.60:0/95808077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe68819f220 con 0x7fe6880ff620 2024-03-04T12:25:31.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.992+0000 7fe68df47640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6880ff620 0x7fe688197600 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe68400b450 tx=0x7fe68400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:31.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.992+0000 7fe67effd640 1 -- 172.21.15.60:0/95808077 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684015020 con 0x7fe6880ff620 2024-03-04T12:25:31.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.992+0000 7fe67effd640 1 -- 172.21.15.60:0/95808077 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe684011d70 con 0x7fe6880ff620 2024-03-04T12:25:31.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.992+0000 7fe6901ab640 1 -- 172.21.15.60:0/95808077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe68819f510 con 0x7fe6880ff620 2024-03-04T12:25:31.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.992+0000 7fe67effd640 1 -- 172.21.15.60:0/95808077 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68400f340 con 0x7fe6880ff620 2024-03-04T12:25:31.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.992+0000 7fe6901ab640 1 -- 172.21.15.60:0/95808077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe68819fab0 con 0x7fe6880ff620 2024-03-04T12:25:31.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.993+0000 7fe6901ab640 1 -- 172.21.15.60:0/95808077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe688108650 con 0x7fe6880ff620 2024-03-04T12:25:31.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.994+0000 7fe67effd640 1 -- 172.21.15.60:0/95808077 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fe684018020 con 0x7fe6880ff620 2024-03-04T12:25:31.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.995+0000 7fe67effd640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe6740a1e00 0x7fe6740a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.995+0000 7fe68d746640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe6740a1e00 0x7fe6740a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.995+0000 7fe67effd640 1 -- 172.21.15.60:0/95808077 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fe6840cdc40 con 0x7fe6880ff620 2024-03-04T12:25:31.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.995+0000 7fe68d746640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe6740a1e00 0x7fe6740a42c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe688198b10 tx=0x7fe678005cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:31.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:30.999+0000 7fe67effd640 1 -- 172.21.15.60:0/95808077 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe68401f020 con 0x7fe6880ff620 2024-03-04T12:25:31.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:30 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3354812904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:31.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:30 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/427593708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:31.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:30 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1929307920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:31.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.180+0000 7fe6901ab640 1 -- 172.21.15.60:0/95808077 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe688061f30 con 0x7fe6880ff620 2024-03-04T12:25:31.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.180+0000 7fe67effd640 1 -- 172.21.15.60:0/95808077 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe68400d040 con 0x7fe6880ff620 2024-03-04T12:25:31.188 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.182+0000 7fe6901ab640 1 -- 172.21.15.60:0/95808077 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe6740a1e00 msgr2=0x7fe6740a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.182+0000 7fe6901ab640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe6740a1e00 0x7fe6740a42c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe688198b10 tx=0x7fe678005cc0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.182+0000 7fe6901ab640 1 -- 172.21.15.60:0/95808077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6880ff620 msgr2=0x7fe688197600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.182+0000 7fe6901ab640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6880ff620 0x7fe688197600 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe68400b450 tx=0x7fe68400b910 comp rx=0 tx=0).stop 2024-03-04T12:25:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.182+0000 7fe6901ab640 1 -- 172.21.15.60:0/95808077 shutdown_connections 2024-03-04T12:25:31.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.182+0000 7fe6901ab640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe6740a1e00 0x7fe6740a42c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.182+0000 7fe6901ab640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe688102fb0 0x7fe68819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.182+0000 7fe6901ab640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6881025f0 0x7fe688197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.182+0000 7fe6901ab640 1 --2- 172.21.15.60:0/95808077 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6880ff620 0x7fe688197600 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.182+0000 7fe6901ab640 1 -- 172.21.15.60:0/95808077 >> 172.21.15.60:0/95808077 conn(0x7fe6880faf50 msgr2=0x7fe688105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.183+0000 7fe6901ab640 1 -- 172.21.15.60:0/95808077 shutdown_connections 2024-03-04T12:25:31.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.183+0000 7fe6901ab640 1 -- 172.21.15.60:0/95808077 wait complete. 2024-03-04T12:25:31.201 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:25:31.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.296+0000 7f2230d53640 1 -- 172.21.15.60:0/3687951639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f222c1014b0 msgr2=0x7f222c101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.296+0000 7f2230d53640 1 --2- 172.21.15.60:0/3687951639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f222c1014b0 0x7f222c101930 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2214009900 tx=0x7f221402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.296+0000 7f2230d53640 1 -- 172.21.15.60:0/3687951639 shutdown_connections 2024-03-04T12:25:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.296+0000 7f2230d53640 1 --2- 172.21.15.60:0/3687951639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f222c101e70 0x7f222c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.296+0000 7f2230d53640 1 --2- 172.21.15.60:0/3687951639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f222c1014b0 0x7f222c101930 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.296+0000 7f2230d53640 1 --2- 172.21.15.60:0/3687951639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f222c1002c0 0x7f222c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.296+0000 7f2230d53640 1 -- 172.21.15.60:0/3687951639 >> 172.21.15.60:0/3687951639 conn(0x7f222c0fb830 msgr2=0x7f222c0fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.296+0000 7f2230d53640 1 -- 172.21.15.60:0/3687951639 shutdown_connections 2024-03-04T12:25:31.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.296+0000 7f2230d53640 1 -- 172.21.15.60:0/3687951639 wait complete. 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2230d53640 1 Processor -- start 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2230d53640 1 -- start start 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2230d53640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f222c1002c0 0x7f222c06a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2230d53640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f222c1014b0 0x7f222c19e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2230d53640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f222c101e70 0x7f222c1a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2230d53640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c10bf40 con 0x7f222c1014b0 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2230d53640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c10bdc0 con 0x7f222c1002c0 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2230d53640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222c10c0c0 con 0x7f222c101e70 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2221d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f222c1014b0 0x7f222c19e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2221d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f222c1014b0 0x7f222c19e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48558/0 (socket says 172.21.15.60:48558) 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2221d9b640 1 -- 172.21.15.60:0/2489964134 learned_addr learned my addr 172.21.15.60:0/2489964134 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f222a59c640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f222c1002c0 0x7f222c06a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f222ad9d640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f222c101e70 0x7f222c1a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2221d9b640 1 -- 172.21.15.60:0/2489964134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f222c101e70 msgr2=0x7f222c1a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2221d9b640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f222c101e70 0x7f222c1a0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2221d9b640 1 -- 172.21.15.60:0/2489964134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f222c1002c0 msgr2=0x7f222c06a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2221d9b640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f222c1002c0 0x7f222c06a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f2221d9b640 1 -- 172.21.15.60:0/2489964134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c1a1520 con 0x7f222c1014b0 2024-03-04T12:25:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.297+0000 7f222a59c640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f222c1002c0 0x7f222c06a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.298+0000 7f2221d9b640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f222c1014b0 0x7f222c19e7e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2214009230 tx=0x7f2214005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.298+0000 7f2223fff640 1 -- 172.21.15.60:0/2489964134 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214041070 con 0x7f222c1014b0 2024-03-04T12:25:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.298+0000 7f2230d53640 1 -- 172.21.15.60:0/2489964134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222c1a1750 con 0x7f222c1014b0 2024-03-04T12:25:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.298+0000 7f2223fff640 1 -- 172.21.15.60:0/2489964134 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22140054e0 con 0x7f222c1014b0 2024-03-04T12:25:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.298+0000 7f2223fff640 1 -- 172.21.15.60:0/2489964134 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214034440 con 0x7f222c1014b0 2024-03-04T12:25:31.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.298+0000 7f2230d53640 1 -- 172.21.15.60:0/2489964134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222c1a1c40 con 0x7f222c1014b0 2024-03-04T12:25:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.299+0000 7f2230d53640 1 -- 172.21.15.60:0/2489964134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f222c1075e0 con 0x7f222c1014b0 2024-03-04T12:25:31.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.300+0000 7f2223fff640 1 -- 172.21.15.60:0/2489964134 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f2214044070 con 0x7f222c1014b0 2024-03-04T12:25:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.300+0000 7f2223fff640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f22180a1db0 0x7f22180a4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.301+0000 7f222a59c640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f22180a1db0 0x7f22180a4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.301+0000 7f2223fff640 1 -- 172.21.15.60:0/2489964134 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f22140f0340 con 0x7f222c1014b0 2024-03-04T12:25:31.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.301+0000 7f222a59c640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f22180a1db0 0x7f22180a4270 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f222c101310 tx=0x7f221c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:31.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.304+0000 7f2223fff640 1 -- 172.21.15.60:0/2489964134 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f22140bd380 con 0x7f222c1014b0 2024-03-04T12:25:31.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:30 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3354812904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:31.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:30 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/427593708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:31.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:30 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1929307920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:31.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:30 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3354812904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:31.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:30 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/427593708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:31.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:30 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1929307920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:31.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.525+0000 7f2230d53640 1 -- 172.21.15.60:0/2489964134 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f222c02d070 con 0x7f222c1014b0 2024-03-04T12:25:31.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.525+0000 7f2223fff640 1 -- 172.21.15.60:0/2489964134 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11373 (secure 0 0 0) 0x7f22140bcc80 con 0x7f222c1014b0 2024-03-04T12:25:31.533 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:31.533 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:23:41.896895+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:25:31.533 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 28 2024-03-04T12:25:31.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.527+0000 7f2230d53640 1 -- 172.21.15.60:0/2489964134 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f22180a1db0 msgr2=0x7f22180a4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.527+0000 7f2230d53640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f22180a1db0 0x7f22180a4270 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f222c101310 tx=0x7f221c008040 comp rx=0 tx=0).stop 2024-03-04T12:25:31.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.527+0000 7f2230d53640 1 -- 172.21.15.60:0/2489964134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f222c1014b0 msgr2=0x7f222c19e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.527+0000 7f2230d53640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f222c1014b0 0x7f222c19e7e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2214009230 tx=0x7f2214005dd0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.527+0000 7f2230d53640 1 -- 172.21.15.60:0/2489964134 shutdown_connections 2024-03-04T12:25:31.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.527+0000 7f2230d53640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f22180a1db0 0x7f22180a4270 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.527+0000 7f2230d53640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f222c101e70 0x7f222c1a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.527+0000 7f2230d53640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f222c1014b0 0x7f222c19e7e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.527+0000 7f2230d53640 1 --2- 172.21.15.60:0/2489964134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f222c1002c0 0x7f222c06a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.527+0000 7f2230d53640 1 -- 172.21.15.60:0/2489964134 >> 172.21.15.60:0/2489964134 conn(0x7f222c0fb830 msgr2=0x7f222c1047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:31.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.527+0000 7f2230d53640 1 -- 172.21.15.60:0/2489964134 shutdown_connections 2024-03-04T12:25:31.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.527+0000 7f2230d53640 1 -- 172.21.15.60:0/2489964134 wait complete. 2024-03-04T12:25:31.546 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:25:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.641+0000 7f32c477e640 1 -- 172.21.15.60:0/1210098444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32bc107040 msgr2=0x7f32bc109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.641+0000 7f32c477e640 1 --2- 172.21.15.60:0/1210098444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32bc107040 0x7f32bc109490 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f32b4009ca0 tx=0x7f32b402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.641+0000 7f32c477e640 1 -- 172.21.15.60:0/1210098444 shutdown_connections 2024-03-04T12:25:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 --2- 172.21.15.60:0/1210098444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32bc107040 0x7f32bc109490 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 --2- 172.21.15.60:0/1210098444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32bc104710 0x7f32bc106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 --2- 172.21.15.60:0/1210098444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32bc101de0 0x7f32bc1041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 -- 172.21.15.60:0/1210098444 >> 172.21.15.60:0/1210098444 conn(0x7f32bc0fb7f0 msgr2=0x7f32bc0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 -- 172.21.15.60:0/1210098444 shutdown_connections 2024-03-04T12:25:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 -- 172.21.15.60:0/1210098444 wait complete. 2024-03-04T12:25:31.651 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 Processor -- start 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 -- start start 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32bc101de0 0x7f32bc197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32bc104710 0x7f32bc197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32bc107040 0x7f32bc19ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc10cff0 con 0x7f32bc104710 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc10ce70 con 0x7f32bc107040 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c477e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc10d170 con 0x7f32bc101de0 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c1d19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32bc104710 0x7f32bc197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c1d19640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32bc104710 0x7f32bc197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48568/0 (socket says 172.21.15.60:48568) 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c1d19640 1 -- 172.21.15.60:0/2384297746 learned_addr learned my addr 172.21.15.60:0/2384297746 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.642+0000 7f32c251a640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32bc101de0 0x7f32bc197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.643+0000 7f32c1d19640 1 -- 172.21.15.60:0/2384297746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32bc101de0 msgr2=0x7f32bc197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.652 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.643+0000 7f32c1d19640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32bc101de0 0x7f32bc197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.643+0000 7f32c2d1b640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32bc107040 0x7f32bc19ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.643+0000 7f32c1d19640 1 -- 172.21.15.60:0/2384297746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32bc107040 msgr2=0x7f32bc19ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.643+0000 7f32c1d19640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32bc107040 0x7f32bc19ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.643+0000 7f32c1d19640 1 -- 172.21.15.60:0/2384297746 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32bc19f230 con 0x7f32bc104710 2024-03-04T12:25:31.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.643+0000 7f32c1d19640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32bc104710 0x7f32bc197b50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00c970 tx=0x7f32ac00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:31.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.643+0000 7f32b37fe640 1 -- 172.21.15.60:0/2384297746 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac007bb0 con 0x7f32bc104710 2024-03-04T12:25:31.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.643+0000 7f32b37fe640 1 -- 172.21.15.60:0/2384297746 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32ac007d50 con 0x7f32bc104710 2024-03-04T12:25:31.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.643+0000 7f32c477e640 1 -- 172.21.15.60:0/2384297746 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32bc19f520 con 0x7f32bc104710 2024-03-04T12:25:31.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.643+0000 7f32b37fe640 1 -- 172.21.15.60:0/2384297746 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac00f450 con 0x7f32bc104710 2024-03-04T12:25:31.653 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.643+0000 7f32c477e640 1 -- 172.21.15.60:0/2384297746 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32bc19fa70 con 0x7f32bc104710 2024-03-04T12:25:31.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.644+0000 7f32c477e640 1 -- 172.21.15.60:0/2384297746 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32bc0fd2e0 con 0x7f32bc104710 2024-03-04T12:25:31.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.645+0000 7f32b37fe640 1 -- 172.21.15.60:0/2384297746 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f32ac0046c0 con 0x7f32bc104710 2024-03-04T12:25:31.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.646+0000 7f32b37fe640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f32a80a1d80 0x7f32a80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.646+0000 7f32c251a640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f32a80a1d80 0x7f32a80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.646+0000 7f32b37fe640 1 -- 172.21.15.60:0/2384297746 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f32ac0ce320 con 0x7f32bc104710 2024-03-04T12:25:31.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.646+0000 7f32c251a640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f32a80a1d80 0x7f32a80a4240 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f32b800bbe0 tx=0x7f32b800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:31.657 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.650+0000 7f32b37fe640 1 -- 172.21.15.60:0/2384297746 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f32ac014030 con 0x7f32bc104710 2024-03-04T12:25:31.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.831+0000 7f32c477e640 1 -- 172.21.15.60:0/2384297746 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32bc02d120 con 0x7f32bc104710 2024-03-04T12:25:31.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.833+0000 7f32b37fe640 1 -- 172.21.15.60:0/2384297746 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24562 (secure 0 0 0) 0x7f32ac09b3c0 con 0x7f32bc104710 2024-03-04T12:25:31.840 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:31.840 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":93,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:25:06.929248+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:25:31.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.835+0000 7f32c477e640 1 -- 172.21.15.60:0/2384297746 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f32a80a1d80 msgr2=0x7f32a80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.835+0000 7f32c477e640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f32a80a1d80 0x7f32a80a4240 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f32b800bbe0 tx=0x7f32b800b390 comp rx=0 tx=0).stop 2024-03-04T12:25:31.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.835+0000 7f32c477e640 1 -- 172.21.15.60:0/2384297746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32bc104710 msgr2=0x7f32bc197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.835+0000 7f32c477e640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32bc104710 0x7f32bc197b50 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00c970 tx=0x7f32ac00ce30 comp rx=0 tx=0).stop 2024-03-04T12:25:31.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.835+0000 7f32c477e640 1 -- 172.21.15.60:0/2384297746 shutdown_connections 2024-03-04T12:25:31.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.835+0000 7f32c477e640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f32a80a1d80 0x7f32a80a4240 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.835+0000 7f32c477e640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32bc107040 0x7f32bc19ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.835+0000 7f32c477e640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f32bc104710 0x7f32bc197b50 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.835+0000 7f32c477e640 1 --2- 172.21.15.60:0/2384297746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32bc101de0 0x7f32bc197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.835+0000 7f32c477e640 1 -- 172.21.15.60:0/2384297746 >> 172.21.15.60:0/2384297746 conn(0x7f32bc0fb7f0 msgr2=0x7f32bc102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:31.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.835+0000 7f32c477e640 1 -- 172.21.15.60:0/2384297746 shutdown_connections 2024-03-04T12:25:31.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.835+0000 7f32c477e640 1 -- 172.21.15.60:0/2384297746 wait complete. 2024-03-04T12:25:31.854 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-04T12:23:12.241967+0000', 'modified': '2024-03-04T12:23:41.896895+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24524, 'mds_1': 24487, 'mds_2': 24479, 'mds_3': 24407, 'mds_4': 24394}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14631': {'gid': 14631, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.60:6839/2180757903', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6838', 'nonce': 2180757903}, {'type': 'v1', 'addr': '172.21.15.60:6839', 'nonce': 2180757903}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14643': {'gid': 14643, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.60:6841/4175833227', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.60:6840', 'nonce': 4175833227}, {'type': 'v1', 'addr': '172.21.15.60:6841', 'nonce': 4175833227}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24394': {'gid': 24394, 'name': 'l', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.86:6833/777854669', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6832', 'nonce': 777854669}, {'type': 'v1', 'addr': '172.21.15.86:6833', 'nonce': 777854669}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24407': {'gid': 24407, 'name': 'k', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.63:6833/2270964219', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6832', 'nonce': 2270964219}, {'type': 'v1', 'addr': '172.21.15.63:6833', 'nonce': 2270964219}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.63:6835/2453356505', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6834', 'nonce': 2453356505}, {'type': 'v1', 'addr': '172.21.15.63:6835', 'nonce': 2453356505}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.86:6835/2934725061', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 2934725061}, {'type': 'v1', 'addr': '172.21.15.86:6835', 'nonce': 2934725061}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24463': {'gid': 24463, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.86:6837/472598778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6836', 'nonce': 472598778}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 472598778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.63:6837/3974574804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6836', 'nonce': 3974574804}, {'type': 'v1', 'addr': '172.21.15.63:6837', 'nonce': 3974574804}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.86:6839/3347800322', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6838', 'nonce': 3347800322}, {'type': 'v1', 'addr': '172.21.15.86:6839', 'nonce': 3347800322}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24524': {'gid': 24524, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.63:6839/2728229139', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6838', 'nonce': 2728229139}, {'type': 'v1', 'addr': '172.21.15.63:6839', 'nonce': 2728229139}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-04T12:25:31.854 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-04T12:25:31.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.l status 2024-03-04T12:25:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.950+0000 7fd2d1854640 1 -- 172.21.15.60:0/2419341963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2cc104700 msgr2=0x7fd2cc106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.950+0000 7fd2d1854640 1 --2- 172.21.15.60:0/2419341963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2cc104700 0x7fd2cc106af0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4009900 tx=0x7fd2b402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 -- 172.21.15.60:0/2419341963 shutdown_connections 2024-03-04T12:25:31.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 --2- 172.21.15.60:0/2419341963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2cc107030 0x7fd2cc109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 --2- 172.21.15.60:0/2419341963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2cc104700 0x7fd2cc106af0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 --2- 172.21.15.60:0/2419341963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2cc101dd0 0x7fd2cc1041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 -- 172.21.15.60:0/2419341963 >> 172.21.15.60:0/2419341963 conn(0x7fd2cc0fb7c0 msgr2=0x7fd2cc0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 -- 172.21.15.60:0/2419341963 shutdown_connections 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 -- 172.21.15.60:0/2419341963 wait complete. 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 Processor -- start 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 -- start start 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2cc101dd0 0x7fd2cc197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2cc104700 0x7fd2cc197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2cc107030 0x7fd2cc19ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc10cff0 con 0x7fd2cc104700 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc10ce70 con 0x7fd2cc107030 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2d1854640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc10d170 con 0x7fd2cc101dd0 2024-03-04T12:25:31.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.951+0000 7fd2ca7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2cc104700 0x7fd2cc197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2ca7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2cc104700 0x7fd2cc197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48582/0 (socket says 172.21.15.60:48582) 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2ca7fc640 1 -- 172.21.15.60:0/2450108368 learned_addr learned my addr 172.21.15.60:0/2450108368 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2caffd640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2cc101dd0 0x7fd2cc197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2ca7fc640 1 -- 172.21.15.60:0/2450108368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2cc101dd0 msgr2=0x7fd2cc197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2ca7fc640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2cc101dd0 0x7fd2cc197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2cb7fe640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2cc107030 0x7fd2cc19ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2ca7fc640 1 -- 172.21.15.60:0/2450108368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2cc107030 msgr2=0x7fd2cc19ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2ca7fc640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2cc107030 0x7fd2cc19ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2ca7fc640 1 -- 172.21.15.60:0/2450108368 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc19f330 con 0x7fd2cc104700 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2cb7fe640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2cc107030 0x7fd2cc19ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2ca7fc640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2cc104700 0x7fd2cc197ca0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4000c00 tx=0x7fd2b4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2d0852640 1 -- 172.21.15.60:0/2450108368 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4043020 con 0x7fd2cc104700 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2d1854640 1 -- 172.21.15.60:0/2450108368 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2cc19f5c0 con 0x7fd2cc104700 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2d0852640 1 -- 172.21.15.60:0/2450108368 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2b4004440 con 0x7fd2cc104700 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2d0852640 1 -- 172.21.15.60:0/2450108368 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b403e450 con 0x7fd2cc104700 2024-03-04T12:25:31.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.952+0000 7fd2d1854640 1 -- 172.21.15.60:0/2450108368 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2cc19fb10 con 0x7fd2cc104700 2024-03-04T12:25:31.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.954+0000 7fd2d0852640 1 -- 172.21.15.60:0/2450108368 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fd2b4046020 con 0x7fd2cc104700 2024-03-04T12:25:31.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2d4430740 1 Processor -- start 2024-03-04T12:25:31.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2d4430740 1 -- start start 2024-03-04T12:25:31.969 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2d4430740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 0x55f0327d4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2d4430740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327d92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2d4430740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327dbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2d4430740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0327397c0 con 0x55f0327d3df0 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2d4430740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0326e2b50 con 0x55f0327d9950 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2d4430740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f032735860 con 0x55f0327d47e0 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2cb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 0x55f0327d4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2cb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 0x55f0327d4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48596/0 (socket says 172.21.15.60:48596) 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2cb7fe640 1 -- 172.21.15.60:0/1666058157 learned_addr learned my addr 172.21.15.60:0/1666058157 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2caffd640 1 --2- 172.21.15.60:0/1666058157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327d92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2d0852640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd2b80a1d80 0x7fd2b80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2ca7fc640 1 --2- 172.21.15.60:0/1666058157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327dbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2cb7fe640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd2b80a1d80 0x7fd2b80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2caffd640 1 -- 172.21.15.60:0/1666058157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 msgr2=0x55f0327dbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2d0852640 1 -- 172.21.15.60:0/2450108368 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fd2b40f0540 con 0x7fd2cc104700 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2caffd640 1 --2- 172.21.15.60:0/1666058157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327dbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.970 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2caffd640 1 -- 172.21.15.60:0/1666058157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 msgr2=0x55f0327d4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2caffd640 1 --2- 172.21.15.60:0/1666058157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 0x55f0327d4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2caffd640 1 -- 172.21.15.60:0/1666058157 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0326f69a0 con 0x55f0327d47e0 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2cb7fe640 1 --2- 172.21.15.60:0/1666058157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 0x55f0327d4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.955+0000 7fd2ca7fc640 1 --2- 172.21.15.60:0/1666058157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327dbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2caffd640 1 --2- 172.21.15.60:0/1666058157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327d92e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400b9e0 tx=0x7fd2c400bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21335fd362e35158 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2cb7fe640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd2b80a1d80 0x7fd2b80a4240 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55f0327d5040 tx=0x7fd2bc004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2c1ffb640 1 -- 172.21.15.60:0/1666058157 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c400de20 con 0x55f0327d47e0 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2c1ffb640 1 -- 172.21.15.60:0/1666058157 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2c4017900 con 0x55f0327d47e0 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2c1ffb640 1 -- 172.21.15.60:0/1666058157 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c400fa30 con 0x55f0327d47e0 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2d4430740 1 -- 172.21.15.60:0/1666058157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 msgr2=0x55f0327d92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2d4430740 1 --2- 172.21.15.60:0/1666058157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327d92e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400b9e0 tx=0x7fd2c400bea0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2d4430740 1 -- 172.21.15.60:0/1666058157 shutdown_connections 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2d4430740 1 --2- 172.21.15.60:0/1666058157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327dbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2d4430740 1 --2- 172.21.15.60:0/1666058157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327d92e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2d4430740 1 --2- 172.21.15.60:0/1666058157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 0x55f0327d4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.971 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2d4430740 1 -- 172.21.15.60:0/1666058157 >> 172.21.15.60:0/1666058157 conn(0x55f0327cf360 msgr2=0x55f0327cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.956+0000 7fd2d4430740 1 -- 172.21.15.60:0/1666058157 shutdown_connections 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2d4430740 1 -- 172.21.15.60:0/1666058157 wait complete. 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2d4430740 1 -- start start 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2d4430740 1 Processor -- start 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2d4430740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d3df0 0x55f0327de2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2d4430740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d47e0 0x55f0327de7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2d4430740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327e5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2d4430740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0327dc900 con 0x55f0327d47e0 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2d4430740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0326f5f90 con 0x55f0327d9950 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2d4430740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0327dca80 con 0x55f0327d3df0 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2d4430740 10 client.0 _ll_register_callbacks cb 0x55f032717d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2ca7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d47e0 0x55f0327de7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2ca7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d47e0 0x55f0327de7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48600/0 (socket says 172.21.15.60:48600) 2024-03-04T12:25:31.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2ca7fc640 1 -- 172.21.15.60:0/4160048941 learned_addr learned my addr 172.21.15.60:0/4160048941 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.957+0000 7fd2caffd640 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d3df0 0x55f0327de2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2ca7fc640 1 -- 172.21.15.60:0/4160048941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d3df0 msgr2=0x55f0327de2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2cb7fe640 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327e5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2ca7fc640 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d3df0 0x55f0327de2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2ca7fc640 1 -- 172.21.15.60:0/4160048941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 msgr2=0x55f0327e5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2ca7fc640 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327e5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2ca7fc640 1 -- 172.21.15.60:0/4160048941 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f03244e2d0 con 0x55f0327d47e0 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2ca7fc640 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d47e0 0x55f0327de7f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40b98b0 tx=0x7fd2b4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4630303e729f3c29 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2aaffd640 1 -- 172.21.15.60:0/4160048941 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b40ef040 con 0x55f0327d47e0 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2aaffd640 1 -- 172.21.15.60:0/4160048941 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fd2b40f0250 con 0x55f0327d47e0 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2d4430740 1 -- 172.21.15.60:0/4160048941 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd2c4017900 con 0x55f0327d47e0 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2aaffd640 1 -- 172.21.15.60:0/4160048941 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2b40c1030 con 0x55f0327d47e0 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2d4430740 10 client.15447 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:31.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2d4430740 1 -- 172.21.15.60:0/4160048941 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f0325186b0 con 0x55f0327d47e0 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2aaffd640 1 -- 172.21.15.60:0/4160048941 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2c4017900 con 0x55f0327d47e0 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.958+0000 7fd2aaffd640 1 -- 172.21.15.60:0/4160048941 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd2b40c51f0 con 0x55f0327d47e0 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.959+0000 7fd2d4430740 10 client.15447 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.959+0000 7fd2d4430740 10 client.15447 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.959+0000 7fd2d4430740 20 client.15447 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.959+0000 7fd2d4430740 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f0327eae50 0x55f0327e9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.959+0000 7fd2d4430740 4 client.15447 mds_command: new command op to 24394 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.959+0000 7fd2d4430740 1 -- 172.21.15.60:0/4160048941 --> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f032746d00 con 0x55f0327eae50 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.959+0000 7fd2caffd640 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f0327eae50 0x55f0327e9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.959+0000 7fd2caffd640 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f0327eae50 0x55f0327e9d20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=65fa69bb1e7ef1e7 server_cookie=3450c7057532110d in_seq=0 out_seq=0 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.959+0000 7fd2aaffd640 10 client.15447 ms_handle_connect on v2:172.21.15.86:6832/777854669 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2aaffd640 1 -- 172.21.15.60:0/4160048941 <== mds.4 v2:172.21.15.86:6832/777854669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f032746d00 con 0x55f0327eae50 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2aaffd640 10 client.15447 handle_command_reply: tid=0 2024-03-04T12:25:31.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2d4430740 1 client.15447 shutdown 2024-03-04T12:25:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2d4430740 1 -- 172.21.15.60:0/4160048941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d47e0 msgr2=0x55f0327de7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2d4430740 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d47e0 0x55f0327de7f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd2b40b98b0 tx=0x7fd2b4009260 comp rx=0 tx=0).stop 2024-03-04T12:25:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2d4430740 1 -- 172.21.15.60:0/4160048941 shutdown_connections 2024-03-04T12:25:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2d4430740 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f0327eae50 0x55f0327e9d20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2d4430740 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327e5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2d4430740 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d47e0 0x55f0327de7f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2aaffd640 0 client.15447 ms_handle_reset on v2:172.21.15.86:6832/777854669 2024-03-04T12:25:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2d4430740 1 --2- 172.21.15.60:0/4160048941 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d3df0 0x55f0327de2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2d4430740 1 -- 172.21.15.60:0/4160048941 >> 172.21.15.60:0/4160048941 conn(0x55f0327cf360 msgr2=0x55f0327ce640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2d4430740 1 -- 172.21.15.60:0/4160048941 shutdown_connections 2024-03-04T12:25:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2d4430740 1 -- 172.21.15.60:0/4160048941 wait complete. 2024-03-04T12:25:31.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.960+0000 7fd2d4430740 20 client.15447 trim_cache size 0 max 16384 2024-03-04T12:25:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2d4430740 1 Processor -- start 2024-03-04T12:25:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2d4430740 1 -- start start 2024-03-04T12:25:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2d4430740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d3df0 0x55f0327de2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2d4430740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327e57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2d4430740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d9950 0x55f0327de7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2d4430740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0327ea670 con 0x55f0327d9950 2024-03-04T12:25:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2d4430740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0327eacd0 con 0x55f0327d3df0 2024-03-04T12:25:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2d4430740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0327eb330 con 0x55f0327d47e0 2024-03-04T12:25:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2cb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d9950 0x55f0327de7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2cb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d9950 0x55f0327de7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48614/0 (socket says 172.21.15.60:48614) 2024-03-04T12:25:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2cb7fe640 1 -- 172.21.15.60:0/2934685013 learned_addr learned my addr 172.21.15.60:0/2934685013 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:31.989 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2caffd640 1 --2- 172.21.15.60:0/2934685013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d3df0 0x55f0327de2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2cb7fe640 1 -- 172.21.15.60:0/2934685013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 msgr2=0x55f0327e57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2cb7fe640 1 --2- 172.21.15.60:0/2934685013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327e57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2cb7fe640 1 -- 172.21.15.60:0/2934685013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d3df0 msgr2=0x55f0327de2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2cb7fe640 1 --2- 172.21.15.60:0/2934685013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d3df0 0x55f0327de2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2cb7fe640 1 -- 172.21.15.60:0/2934685013 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0327f3b80 con 0x55f0327d9950 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2caffd640 1 --2- 172.21.15.60:0/2934685013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d3df0 0x55f0327de2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.976+0000 7fd2cb7fe640 1 --2- 172.21.15.60:0/2934685013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d9950 0x55f0327de7f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0143d0 tx=0x7fd2bc0087c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13623570dbbc9b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2c17fa640 1 -- 172.21.15.60:0/2934685013 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc023070 con 0x55f0327d9950 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2c17fa640 1 -- 172.21.15.60:0/2934685013 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2bc018040 con 0x55f0327d9950 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2c17fa640 1 -- 172.21.15.60:0/2934685013 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc01fc00 con 0x55f0327d9950 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2d4430740 1 -- 172.21.15.60:0/2934685013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d9950 msgr2=0x55f0327de7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2d4430740 1 --2- 172.21.15.60:0/2934685013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d9950 0x55f0327de7f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0143d0 tx=0x7fd2bc0087c0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2d4430740 1 -- 172.21.15.60:0/2934685013 shutdown_connections 2024-03-04T12:25:31.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2d4430740 1 --2- 172.21.15.60:0/2934685013 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d9950 0x55f0327de7f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2d4430740 1 --2- 172.21.15.60:0/2934685013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327e57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2d4430740 1 --2- 172.21.15.60:0/2934685013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d3df0 0x55f0327de2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2d4430740 1 -- 172.21.15.60:0/2934685013 >> 172.21.15.60:0/2934685013 conn(0x55f0327cf360 msgr2=0x55f0327d5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2d4430740 1 -- 172.21.15.60:0/2934685013 shutdown_connections 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2d4430740 1 -- 172.21.15.60:0/2934685013 wait complete. 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.977+0000 7fd2d4430740 1 -- start start 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2d4430740 1 Processor -- start 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2d4430740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 0x55f0327e2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2d4430740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327e33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2d4430740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327e4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2d4430740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0327ebff0 con 0x55f0327d3df0 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2d4430740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0327eb990 con 0x55f0327d9950 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2d4430740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0327ec650 con 0x55f0327d47e0 2024-03-04T12:25:31.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2d4430740 10 client.0 _ll_register_callbacks cb 0x7fd2c4005f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2caffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 0x55f0327e2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2caffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 0x55f0327e2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48626/0 (socket says 172.21.15.60:48626) 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2caffd640 1 -- 172.21.15.60:0/976504723 learned_addr learned my addr 172.21.15.60:0/976504723 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2cb7fe640 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327e4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2ca7fc640 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327e33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2caffd640 1 -- 172.21.15.60:0/976504723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 msgr2=0x55f0327e33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2caffd640 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327e33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2caffd640 1 -- 172.21.15.60:0/976504723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 msgr2=0x55f0327e4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2caffd640 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327e4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2caffd640 1 -- 172.21.15.60:0/976504723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0327e4910 con 0x55f0327d3df0 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2ca7fc640 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327e33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2cb7fe640 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327e4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2caffd640 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 0x55f0327e2e70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4019040 tx=0x7fd2c400ddf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=747338351f131e03 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:31.992 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.978+0000 7fd2aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2aaffd640 1 -- 172.21.15.60:0/976504723 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4005040 con 0x55f0327d3df0 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2aaffd640 1 -- 172.21.15.60:0/976504723 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fd2c40051e0 con 0x55f0327d3df0 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2d4430740 1 -- 172.21.15.60:0/976504723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd2bc018040 con 0x55f0327d3df0 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2aaffd640 1 -- 172.21.15.60:0/976504723 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2c4039050 con 0x55f0327d3df0 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2d4430740 10 client.15453 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2d4430740 1 -- 172.21.15.60:0/976504723 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f032842a60 con 0x55f0327d3df0 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2aaffd640 1 -- 172.21.15.60:0/976504723 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2bc018040 con 0x55f0327d3df0 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2aaffd640 1 -- 172.21.15.60:0/976504723 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd2c400f420 con 0x55f0327d3df0 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2d4430740 10 client.15453 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2d4430740 10 client.15453 resolve_mds: resolved name 'l' to daemon mds.l 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2d4430740 20 client.15453 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2d4430740 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f032846cb0 0x55f032842c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2d4430740 4 client.15453 mds_command: new command op to 24394 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2d4430740 1 -- 172.21.15.60:0/976504723 --> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f032741050 con 0x55f032846cb0 2024-03-04T12:25:31.993 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.979+0000 7fd2ca7fc640 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f032846cb0 0x55f032842c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2ca7fc640 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f032846cb0 0x55f032842c30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=82689885b84ab866 server_cookie=bb5e39e3e0df354a in_seq=0 out_seq=0 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2aaffd640 10 client.15453 ms_handle_connect on v2:172.21.15.86:6832/777854669 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2aaffd640 1 -- 172.21.15.60:0/976504723 <== mds.4 v2:172.21.15.86:6832/777854669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55f032741050 con 0x55f032846cb0 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2aaffd640 10 client.15453 handle_command_reply: tid=0 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2d4430740 1 client.15453 shutdown 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2d4430740 1 -- 172.21.15.60:0/976504723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 msgr2=0x55f0327e2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2d4430740 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 0x55f0327e2e70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4019040 tx=0x7fd2c400ddf0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2d4430740 1 -- 172.21.15.60:0/976504723 shutdown_connections 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2d4430740 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f032846cb0 0x55f032842c30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2d4430740 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f0327d9950 0x55f0327e4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2aaffd640 0 client.15453 ms_handle_reset on v2:172.21.15.86:6832/777854669 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2d4430740 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f0327d47e0 0x55f0327e33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2d4430740 1 --2- 172.21.15.60:0/976504723 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f0327d3df0 0x55f0327e2e70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2d4430740 1 -- 172.21.15.60:0/976504723 >> 172.21.15.60:0/976504723 conn(0x55f0327cf360 msgr2=0x55f0327de2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2d4430740 1 -- 172.21.15.60:0/976504723 shutdown_connections 2024-03-04T12:25:31.994 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.980+0000 7fd2d4430740 1 -- 172.21.15.60:0/976504723 wait complete. 2024-03-04T12:25:31.995 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d4430740 20 client.15453 trim_cache size 0 max 16384 2024-03-04T12:25:31.995 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:25:31.995 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:25:31.995 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 4, 2024-03-04T12:25:31.995 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 24394, 2024-03-04T12:25:31.995 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-03-04T12:25:31.995 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-03-04T12:25:31.995 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-03-04T12:25:31.995 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 111.089248039, 2024-03-04T12:25:31.995 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 28, 2024-03-04T12:25:31.996 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 93, 2024-03-04T12:25:31.996 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 89, 2024-03-04T12:25:31.996 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 177.378787012 2024-03-04T12:25:31.996 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:25:31.996 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d1854640 1 -- 172.21.15.60:0/2450108368 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd2b80a1d80 msgr2=0x7fd2b80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d1854640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd2b80a1d80 0x7fd2b80a4240 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x55f0327d5040 tx=0x7fd2bc004ab0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d1854640 1 -- 172.21.15.60:0/2450108368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2cc104700 msgr2=0x7fd2cc197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d1854640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2cc104700 0x7fd2cc197ca0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4000c00 tx=0x7fd2b4004040 comp rx=0 tx=0).stop 2024-03-04T12:25:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d1854640 1 -- 172.21.15.60:0/2450108368 shutdown_connections 2024-03-04T12:25:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d1854640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd2b80a1d80 0x7fd2b80a4240 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d1854640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2cc107030 0x7fd2cc19ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d1854640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2cc104700 0x7fd2cc197ca0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d1854640 1 --2- 172.21.15.60:0/2450108368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2cc101dd0 0x7fd2cc197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d1854640 1 -- 172.21.15.60:0/2450108368 >> 172.21.15.60:0/2450108368 conn(0x7fd2cc0fb7c0 msgr2=0x7fd2cc102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d1854640 1 -- 172.21.15.60:0/2450108368 shutdown_connections 2024-03-04T12:25:31.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:31.981+0000 7fd2d1854640 1 -- 172.21.15.60:0/2450108368 wait complete. 2024-03-04T12:25:32.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-04T12:25:32.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.096+0000 7fd15682b640 1 -- 172.21.15.60:0/3348569657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd150101440 msgr2=0x7fd1501018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.096+0000 7fd15682b640 1 --2- 172.21.15.60:0/3348569657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd150101440 0x7fd1501018c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd144009900 tx=0x7fd14402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:32.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.096+0000 7fd15682b640 1 -- 172.21.15.60:0/3348569657 shutdown_connections 2024-03-04T12:25:32.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.096+0000 7fd15682b640 1 --2- 172.21.15.60:0/3348569657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd150101e00 0x7fd150108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.096+0000 7fd15682b640 1 --2- 172.21.15.60:0/3348569657 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd150101440 0x7fd1501018c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.096+0000 7fd15682b640 1 --2- 172.21.15.60:0/3348569657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd150100250 0x7fd150100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.096+0000 7fd15682b640 1 -- 172.21.15.60:0/3348569657 >> 172.21.15.60:0/3348569657 conn(0x7fd1500fb7c0 msgr2=0x7fd1500fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.097+0000 7fd15682b640 1 -- 172.21.15.60:0/3348569657 shutdown_connections 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.097+0000 7fd15682b640 1 -- 172.21.15.60:0/3348569657 wait complete. 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.097+0000 7fd15682b640 1 Processor -- start 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.097+0000 7fd15682b640 1 -- start start 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.097+0000 7fd15682b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd150100250 0x7fd1501976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.097+0000 7fd15682b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd150101440 0x7fd150197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.097+0000 7fd15682b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd150101e00 0x7fd15019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.097+0000 7fd15682b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15010bee0 con 0x7fd150100250 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.097+0000 7fd15682b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15010bd60 con 0x7fd150101440 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.097+0000 7fd15682b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15010c060 con 0x7fd150101e00 2024-03-04T12:25:32.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd150100250 0x7fd1501976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd150100250 0x7fd1501976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48634/0 (socket says 172.21.15.60:48634) 2024-03-04T12:25:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14ffff640 1 -- 172.21.15.60:0/1175909687 learned_addr learned my addr 172.21.15.60:0/1175909687 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd154dc8640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd150101e00 0x7fd15019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14f7fe640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd150101440 0x7fd150197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14ffff640 1 -- 172.21.15.60:0/1175909687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd150101e00 msgr2=0x7fd15019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14ffff640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd150101e00 0x7fd15019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14ffff640 1 -- 172.21.15.60:0/1175909687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd150101440 msgr2=0x7fd150197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14ffff640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd150101440 0x7fd150197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14ffff640 1 -- 172.21.15.60:0/1175909687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15019f250 con 0x7fd150100250 2024-03-04T12:25:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14f7fe640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd150101440 0x7fd150197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14ffff640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd150100250 0x7fd1501976a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd14000d950 tx=0x7fd14000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14d7fa640 1 -- 172.21.15.60:0/1175909687 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140017020 con 0x7fd150100250 2024-03-04T12:25:32.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd15682b640 1 -- 172.21.15.60:0/1175909687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15019f540 con 0x7fd150100250 2024-03-04T12:25:32.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14d7fa640 1 -- 172.21.15.60:0/1175909687 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd140009c30 con 0x7fd150100250 2024-03-04T12:25:32.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd14d7fa640 1 -- 172.21.15.60:0/1175909687 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14000f6b0 con 0x7fd150100250 2024-03-04T12:25:32.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.098+0000 7fd15682b640 1 -- 172.21.15.60:0/1175909687 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15019fab0 con 0x7fd150100250 2024-03-04T12:25:32.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd14d7fa640 1 -- 172.21.15.60:0/1175909687 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fd14001a020 con 0x7fd150100250 2024-03-04T12:25:32.116 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd159411740 1 Processor -- start 2024-03-04T12:25:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd159411740 1 -- start start 2024-03-04T12:25:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd159411740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba1df0 0x5652c2ba2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd159411740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba27e0 0x5652c2ba72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd159411740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba7950 0x5652c2ba9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd159411740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2b077c0 con 0x5652c2ba27e0 2024-03-04T12:25:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd159411740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2ab0b50 con 0x5652c2ba1df0 2024-03-04T12:25:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd159411740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2b03860 con 0x5652c2ba7950 2024-03-04T12:25:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba27e0 0x5652c2ba72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba27e0 0x5652c2ba72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48644/0 (socket says 172.21.15.60:48644) 2024-03-04T12:25:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd14ffff640 1 -- 172.21.15.60:0/786584931 learned_addr learned my addr 172.21.15.60:0/786584931 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.117 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd14d7fa640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd1380a1d80 0x7fd1380a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd14f7fe640 1 --2- 172.21.15.60:0/786584931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba7950 0x5652c2ba9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.101+0000 7fd154dc8640 1 --2- 172.21.15.60:0/786584931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba1df0 0x5652c2ba2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd154dc8640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd1380a1d80 0x7fd1380a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd14d7fa640 1 -- 172.21.15.60:0/1175909687 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fd1400ce170 con 0x7fd150100250 2024-03-04T12:25:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd14ffff640 1 -- 172.21.15.60:0/786584931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba7950 msgr2=0x5652c2ba9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd14ffff640 1 --2- 172.21.15.60:0/786584931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba7950 0x5652c2ba9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd14ffff640 1 -- 172.21.15.60:0/786584931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba1df0 msgr2=0x5652c2ba2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd14ffff640 1 --2- 172.21.15.60:0/786584931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba1df0 0x5652c2ba2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd14ffff640 1 -- 172.21.15.60:0/786584931 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652c2ac49a0 con 0x5652c2ba27e0 2024-03-04T12:25:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd14f7fe640 1 --2- 172.21.15.60:0/786584931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba7950 0x5652c2ba9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:25:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd14ffff640 1 --2- 172.21.15.60:0/786584931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba27e0 0x5652c2ba72e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd1400974e0 tx=0x7fd140000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08a79dee46719bd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.118 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd154dc8640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd1380a1d80 0x7fd1380a4240 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd148007fd0 tx=0x7fd148007520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd1367fc640 1 -- 172.21.15.60:0/786584931 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14009ab10 con 0x5652c2ba27e0 2024-03-04T12:25:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd1367fc640 1 -- 172.21.15.60:0/786584931 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd14009ec60 con 0x5652c2ba27e0 2024-03-04T12:25:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.102+0000 7fd1367fc640 1 -- 172.21.15.60:0/786584931 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140097180 con 0x5652c2ba27e0 2024-03-04T12:25:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.103+0000 7fd159411740 1 -- 172.21.15.60:0/786584931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba27e0 msgr2=0x5652c2ba72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.103+0000 7fd159411740 1 --2- 172.21.15.60:0/786584931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba27e0 0x5652c2ba72e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd1400974e0 tx=0x7fd140000bc0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.103+0000 7fd159411740 1 -- 172.21.15.60:0/786584931 shutdown_connections 2024-03-04T12:25:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.103+0000 7fd159411740 1 --2- 172.21.15.60:0/786584931 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba7950 0x5652c2ba9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.103+0000 7fd159411740 1 --2- 172.21.15.60:0/786584931 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba27e0 0x5652c2ba72e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.103+0000 7fd159411740 1 --2- 172.21.15.60:0/786584931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba1df0 0x5652c2ba2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.119 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.103+0000 7fd159411740 1 -- 172.21.15.60:0/786584931 >> 172.21.15.60:0/786584931 conn(0x5652c2b9d360 msgr2=0x5652c2b9d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.103+0000 7fd159411740 1 -- 172.21.15.60:0/786584931 shutdown_connections 2024-03-04T12:25:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.103+0000 7fd159411740 1 -- 172.21.15.60:0/786584931 wait complete. 2024-03-04T12:25:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.103+0000 7fd159411740 1 -- start start 2024-03-04T12:25:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.103+0000 7fd159411740 1 Processor -- start 2024-03-04T12:25:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd159411740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba1df0 0x5652c2bac200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd159411740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bac740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd159411740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba7950 0x5652c2bb3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd159411740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2baa840 con 0x5652c2ba1df0 2024-03-04T12:25:32.120 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd159411740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2ac3f90 con 0x5652c2ba27e0 2024-03-04T12:25:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd159411740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2baa9c0 con 0x5652c2ba7950 2024-03-04T12:25:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd159411740 10 client.0 _ll_register_callbacks cb 0x5652c2ae5d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba1df0 0x5652c2bac200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba1df0 0x5652c2bac200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48656/0 (socket says 172.21.15.60:48656) 2024-03-04T12:25:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd14ffff640 1 -- 172.21.15.60:0/4054305567 learned_addr learned my addr 172.21.15.60:0/4054305567 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd154dc8640 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba7950 0x5652c2bb3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd14f7fe640 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd14ffff640 1 -- 172.21.15.60:0/4054305567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba7950 msgr2=0x5652c2bb3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd14ffff640 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba7950 0x5652c2bb3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd14ffff640 1 -- 172.21.15.60:0/4054305567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 msgr2=0x5652c2bac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd14ffff640 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bac740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.121 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd14ffff640 1 -- 172.21.15.60:0/4054305567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652c281c2d0 con 0x5652c2ba1df0 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd14ffff640 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba1df0 0x5652c2bac200 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd1400ce140 tx=0x7fd14009b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995d9a4f3b0a42e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd12f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd12f7fe640 1 -- 172.21.15.60:0/4054305567 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1400b3060 con 0x5652c2ba1df0 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd12f7fe640 1 -- 172.21.15.60:0/4054305567 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fd140097180 con 0x5652c2ba1df0 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd159411740 1 -- 172.21.15.60:0/4054305567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd14009ec60 con 0x5652c2ba1df0 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd12f7fe640 1 -- 172.21.15.60:0/4054305567 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1400a32b0 con 0x5652c2ba1df0 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.104+0000 7fd159411740 10 client.15465 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.105+0000 7fd159411740 1 -- 172.21.15.60:0/4054305567 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5652c28e66b0 con 0x5652c2ba1df0 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.105+0000 7fd12f7fe640 1 -- 172.21.15.60:0/4054305567 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd14009ec60 con 0x5652c2ba1df0 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.105+0000 7fd12f7fe640 1 -- 172.21.15.60:0/4054305567 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd1400aec80 con 0x5652c2ba1df0 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.105+0000 7fd159411740 10 client.15465 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.105+0000 7fd159411740 10 client.15465 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.105+0000 7fd159411740 20 client.15465 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:32.122 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.105+0000 7fd159411740 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x5652c2bb8790 0x5652c2bbcc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.105+0000 7fd159411740 4 client.15465 mds_command: new command op to 24407 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.105+0000 7fd159411740 1 -- 172.21.15.60:0/4054305567 --> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652c2b14d00 con 0x5652c2bb8790 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.105+0000 7fd14f7fe640 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x5652c2bb8790 0x5652c2bbcc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.106+0000 7fd14f7fe640 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x5652c2bb8790 0x5652c2bbcc10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6deaea6dfbdfc6ca server_cookie=44ea73bbe4ffb3fc in_seq=0 out_seq=0 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.106+0000 7fd12f7fe640 10 client.15465 ms_handle_connect on v2:172.21.15.63:6832/2270964219 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.106+0000 7fd12f7fe640 1 -- 172.21.15.60:0/4054305567 <== mds.3 v2:172.21.15.63:6832/2270964219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5652c2b14d00 con 0x5652c2bb8790 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.106+0000 7fd12f7fe640 10 client.15465 handle_command_reply: tid=0 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.106+0000 7fd159411740 1 client.15465 shutdown 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.107+0000 7fd159411740 1 -- 172.21.15.60:0/4054305567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba1df0 msgr2=0x5652c2bac200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.107+0000 7fd159411740 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba1df0 0x5652c2bac200 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd1400ce140 tx=0x7fd14009b340 comp rx=0 tx=0).stop 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.107+0000 7fd159411740 1 -- 172.21.15.60:0/4054305567 shutdown_connections 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.107+0000 7fd159411740 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x5652c2bb8790 0x5652c2bbcc10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.107+0000 7fd159411740 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba7950 0x5652c2bb3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.123 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.107+0000 7fd159411740 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.107+0000 7fd159411740 1 --2- 172.21.15.60:0/4054305567 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba1df0 0x5652c2bac200 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.107+0000 7fd12f7fe640 0 client.15465 ms_handle_reset on v2:172.21.15.63:6832/2270964219 2024-03-04T12:25:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.107+0000 7fd159411740 1 -- 172.21.15.60:0/4054305567 >> 172.21.15.60:0/4054305567 conn(0x5652c2b9d360 msgr2=0x5652c2b9c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.107+0000 7fd159411740 1 -- 172.21.15.60:0/4054305567 shutdown_connections 2024-03-04T12:25:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.107+0000 7fd159411740 1 -- 172.21.15.60:0/4054305567 wait complete. 2024-03-04T12:25:32.124 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.107+0000 7fd159411740 20 client.15465 trim_cache size 0 max 16384 2024-03-04T12:25:32.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.122+0000 7fd159411740 1 Processor -- start 2024-03-04T12:25:32.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.122+0000 7fd159411740 1 -- start start 2024-03-04T12:25:32.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.122+0000 7fd159411740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba1df0 0x5652c2bc1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.122+0000 7fd159411740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bb3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.122+0000 7fd159411740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 0x5652c2b9bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.122+0000 7fd159411740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2bb7fb0 con 0x5652c2ba7950 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.122+0000 7fd159411740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2bb8610 con 0x5652c2ba27e0 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd159411740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2bb8c70 con 0x5652c2ba1df0 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd154dc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 0x5652c2b9bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd154dc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 0x5652c2b9bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48670/0 (socket says 172.21.15.60:48670) 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd154dc8640 1 -- 172.21.15.60:0/2545432335 learned_addr learned my addr 172.21.15.60:0/2545432335 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd14f7fe640 1 --2- 172.21.15.60:0/2545432335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bb3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd154dc8640 1 -- 172.21.15.60:0/2545432335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba1df0 msgr2=0x5652c2bc1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd154dc8640 1 --2- 172.21.15.60:0/2545432335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba1df0 0x5652c2bc1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd154dc8640 1 -- 172.21.15.60:0/2545432335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 msgr2=0x5652c2bb3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd154dc8640 1 --2- 172.21.15.60:0/2545432335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bb3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd154dc8640 1 -- 172.21.15.60:0/2545432335 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652c2ba6fb0 con 0x5652c2ba7950 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd14f7fe640 1 --2- 172.21.15.60:0/2545432335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bb3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:32.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd154dc8640 1 --2- 172.21.15.60:0/2545432335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 0x5652c2b9bdb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd148011c90 tx=0x7fd1480078c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5d95a902a42124 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd135ffb640 1 -- 172.21.15.60:0/2545432335 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148004530 con 0x5652c2ba7950 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd135ffb640 1 -- 172.21.15.60:0/2545432335 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1480046d0 con 0x5652c2ba7950 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.123+0000 7fd135ffb640 1 -- 172.21.15.60:0/2545432335 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14800f480 con 0x5652c2ba7950 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.124+0000 7fd159411740 1 -- 172.21.15.60:0/2545432335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 msgr2=0x5652c2b9bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.124+0000 7fd159411740 1 --2- 172.21.15.60:0/2545432335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 0x5652c2b9bdb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd148011c90 tx=0x7fd1480078c0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.124+0000 7fd159411740 1 -- 172.21.15.60:0/2545432335 shutdown_connections 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.124+0000 7fd159411740 1 --2- 172.21.15.60:0/2545432335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 0x5652c2b9bdb0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.124+0000 7fd159411740 1 --2- 172.21.15.60:0/2545432335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bb3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.124+0000 7fd159411740 1 --2- 172.21.15.60:0/2545432335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba1df0 0x5652c2bc1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.124+0000 7fd159411740 1 -- 172.21.15.60:0/2545432335 >> 172.21.15.60:0/2545432335 conn(0x5652c2b9d360 msgr2=0x5652c2bc0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.124+0000 7fd159411740 1 -- 172.21.15.60:0/2545432335 shutdown_connections 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.124+0000 7fd159411740 1 -- 172.21.15.60:0/2545432335 wait complete. 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.124+0000 7fd159411740 1 -- start start 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.124+0000 7fd159411740 1 Processor -- start 2024-03-04T12:25:32.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.124+0000 7fd159411740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba1df0 0x5652c2bb0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd159411740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bb0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd159411740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 0x5652c2bb1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd159411740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2bb9930 con 0x5652c2ba7950 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd159411740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2bb92d0 con 0x5652c2ba27e0 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd159411740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2bb9f90 con 0x5652c2ba1df0 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd159411740 10 client.0 _ll_register_callbacks cb 0x7fd140096be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd154dc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 0x5652c2bb1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd154dc8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 0x5652c2bb1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48678/0 (socket says 172.21.15.60:48678) 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd154dc8640 1 -- 172.21.15.60:0/2843469144 learned_addr learned my addr 172.21.15.60:0/2843469144 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd154dc8640 1 -- 172.21.15.60:0/2843469144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba1df0 msgr2=0x5652c2bb0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd14ffff640 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba1df0 0x5652c2bb0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd14f7fe640 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bb0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd154dc8640 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba1df0 0x5652c2bb0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd154dc8640 1 -- 172.21.15.60:0/2843469144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 msgr2=0x5652c2bb0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd154dc8640 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bb0d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd154dc8640 1 -- 172.21.15.60:0/2843469144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652c2bb2370 con 0x5652c2ba7950 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd154dc8640 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 0x5652c2bb1d80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd148007a30 tx=0x7fd148003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=145bdb2c63b576e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd12f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd12f7fe640 1 -- 172.21.15.60:0/2843469144 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148004530 con 0x5652c2ba7950 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd12f7fe640 1 -- 172.21.15.60:0/2843469144 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fd14802c480 con 0x5652c2ba7950 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd159411740 1 -- 172.21.15.60:0/2843469144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1480046d0 con 0x5652c2ba7950 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd12f7fe640 1 -- 172.21.15.60:0/2843469144 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd148004e50 con 0x5652c2ba7950 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd159411740 10 client.15471 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.125+0000 7fd159411740 1 -- 172.21.15.60:0/2843469144 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5652c2bb35e0 con 0x5652c2ba7950 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.126+0000 7fd12f7fe640 1 -- 172.21.15.60:0/2843469144 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1480046d0 con 0x5652c2ba7950 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.126+0000 7fd12f7fe640 1 -- 172.21.15.60:0/2843469144 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd148036430 con 0x5652c2ba7950 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.126+0000 7fd159411740 10 client.15471 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:32.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.126+0000 7fd159411740 10 client.15471 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.126+0000 7fd159411740 20 client.15471 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.126+0000 7fd159411740 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x5652c2c14710 0x5652c2c108e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.126+0000 7fd159411740 4 client.15471 mds_command: new command op to 24407 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.126+0000 7fd159411740 1 -- 172.21.15.60:0/2843469144 --> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5652c2b0f050 con 0x5652c2c14710 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.126+0000 7fd14ffff640 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x5652c2c14710 0x5652c2c108e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.126+0000 7fd12f7fe640 10 client.15471 ms_handle_connect on v2:172.21.15.63:6832/2270964219 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.126+0000 7fd14ffff640 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x5652c2c14710 0x5652c2c108e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3acd53b3112f8a5b server_cookie=b108a8ac1ad38880 in_seq=0 out_seq=0 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd12f7fe640 1 -- 172.21.15.60:0/2843469144 <== mds.3 v2:172.21.15.63:6832/2270964219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5652c2b0f050 con 0x5652c2c14710 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd12f7fe640 10 client.15471 handle_command_reply: tid=0 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd159411740 1 client.15471 shutdown 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd159411740 1 -- 172.21.15.60:0/2843469144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 msgr2=0x5652c2bb1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd159411740 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 0x5652c2bb1d80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd148007a30 tx=0x7fd148003f70 comp rx=0 tx=0).stop 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd159411740 1 -- 172.21.15.60:0/2843469144 shutdown_connections 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd159411740 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x5652c2c14710 0x5652c2c108e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd159411740 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5652c2ba7950 0x5652c2bb1d80 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd159411740 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5652c2ba27e0 0x5652c2bb0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd12f7fe640 0 client.15471 ms_handle_reset on v2:172.21.15.63:6832/2270964219 2024-03-04T12:25:32.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd159411740 1 --2- 172.21.15.60:0/2843469144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5652c2ba1df0 0x5652c2bb0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd159411740 1 -- 172.21.15.60:0/2843469144 >> 172.21.15.60:0/2843469144 conn(0x5652c2b9d360 msgr2=0x5652c2bc14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd159411740 1 -- 172.21.15.60:0/2843469144 shutdown_connections 2024-03-04T12:25:32.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd159411740 1 -- 172.21.15.60:0/2843469144 wait complete. 2024-03-04T12:25:32.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd159411740 20 client.15471 trim_cache size 0 max 16384 2024-03-04T12:25:32.141 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:25:32.141 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:25:32.141 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 3, 2024-03-04T12:25:32.141 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 24407, 2024-03-04T12:25:32.142 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-03-04T12:25:32.142 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-03-04T12:25:32.142 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-03-04T12:25:32.142 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 112.24142210799999, 2024-03-04T12:25:32.142 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 28, 2024-03-04T12:25:32.142 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 93, 2024-03-04T12:25:32.142 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 89, 2024-03-04T12:25:32.142 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 174.50807157400001 2024-03-04T12:25:32.142 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:25:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd15682b640 1 -- 172.21.15.60:0/1175909687 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd1380a1d80 msgr2=0x7fd1380a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd15682b640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd1380a1d80 0x7fd1380a4240 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd148007fd0 tx=0x7fd148007520 comp rx=0 tx=0).stop 2024-03-04T12:25:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd15682b640 1 -- 172.21.15.60:0/1175909687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd150100250 msgr2=0x7fd1501976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.127+0000 7fd15682b640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd150100250 0x7fd1501976a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd14000d950 tx=0x7fd14000de10 comp rx=0 tx=0).stop 2024-03-04T12:25:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.128+0000 7fd15682b640 1 -- 172.21.15.60:0/1175909687 shutdown_connections 2024-03-04T12:25:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.128+0000 7fd15682b640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd1380a1d80 0x7fd1380a4240 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.128+0000 7fd15682b640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd150101e00 0x7fd15019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.128+0000 7fd15682b640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd150101440 0x7fd150197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.128+0000 7fd15682b640 1 --2- 172.21.15.60:0/1175909687 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd150100250 0x7fd1501976a0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.128+0000 7fd15682b640 1 -- 172.21.15.60:0/1175909687 >> 172.21.15.60:0/1175909687 conn(0x7fd1500fb7c0 msgr2=0x7fd1500fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.128+0000 7fd15682b640 1 -- 172.21.15.60:0/1175909687 shutdown_connections 2024-03-04T12:25:32.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.128+0000 7fd15682b640 1 -- 172.21.15.60:0/1175909687 wait complete. 2024-03-04T12:25:32.144 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:31 smithi060 ceph-mon[32745]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:25:32.144 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:31 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/95808077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:32.145 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:31 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2489964134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:32.145 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:31 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2384297746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:32.146 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-04T12:25:32.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/1258383424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9ec101340 msgr2=0x7fd9ec101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 --2- 172.21.15.60:0/1258383424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9ec101340 0x7fd9ec101760 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8009900 tx=0x7fd9e802bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/1258383424 shutdown_connections 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 --2- 172.21.15.60:0/1258383424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec102ef0 0x7fd9ec109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 --2- 172.21.15.60:0/1258383424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9ec102530 0x7fd9ec1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 --2- 172.21.15.60:0/1258383424 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9ec101340 0x7fd9ec101760 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/1258383424 >> 172.21.15.60:0/1258383424 conn(0x7fd9ec0fc8b0 msgr2=0x7fd9ec0fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/1258383424 shutdown_connections 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/1258383424 wait complete. 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 Processor -- start 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 -- start start 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec101340 0x7fd9ec1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9ec102530 0x7fd9ec197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.242+0000 7fd9f3bf1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9ec102ef0 0x7fd9ec19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f3bf1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec10d020 con 0x7fd9ec102530 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f3bf1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec10cea0 con 0x7fd9ec102ef0 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f3bf1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec10d1a0 con 0x7fd9ec101340 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f118c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9ec102530 0x7fd9ec197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f118c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9ec102530 0x7fd9ec197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48694/0 (socket says 172.21.15.60:48694) 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f118c640 1 -- 172.21.15.60:0/2726430517 learned_addr learned my addr 172.21.15.60:0/2726430517 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f218e640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9ec102ef0 0x7fd9ec19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f198d640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec101340 0x7fd9ec1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f118c640 1 -- 172.21.15.60:0/2726430517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec101340 msgr2=0x7fd9ec1975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f118c640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec101340 0x7fd9ec1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f118c640 1 -- 172.21.15.60:0/2726430517 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9ec102ef0 msgr2=0x7fd9ec19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f118c640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9ec102ef0 0x7fd9ec19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f118c640 1 -- 172.21.15.60:0/2726430517 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec19f200 con 0x7fd9ec102530 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f118c640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9ec102530 0x7fd9ec197b20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00c920 tx=0x7fd9dc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9e2ffd640 1 -- 172.21.15.60:0/2726430517 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc007c70 con 0x7fd9ec102530 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9e2ffd640 1 -- 172.21.15.60:0/2726430517 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9dc007e10 con 0x7fd9ec102530 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/2726430517 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ec19f4f0 con 0x7fd9ec102530 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9e2ffd640 1 -- 172.21.15.60:0/2726430517 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc018890 con 0x7fd9ec102530 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.243+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/2726430517 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ec19fa40 con 0x7fd9ec102530 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.245+0000 7fd9e2ffd640 1 -- 172.21.15.60:0/2726430517 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fd9dc018a70 con 0x7fd9ec102530 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f67cf740 1 Processor -- start 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f67cf740 1 -- start start 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f67cf740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 0x559822cf8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f67cf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cf8bf0 0x559822cfd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f67cf740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cfdd60 0x559822d001f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f67cf740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x559822c597c0 con 0x559822cf8200 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f67cf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x559822c02b50 con 0x559822cf8bf0 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f67cf740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x559822c55860 con 0x559822cfdd60 2024-03-04T12:25:32.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9e2ffd640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9d80a1d80 0x7fd9d80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f218e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 0x559822cf8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f218e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 0x559822cf8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48710/0 (socket says 172.21.15.60:48710) 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f218e640 1 -- 172.21.15.60:0/1712090786 learned_addr learned my addr 172.21.15.60:0/1712090786 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f218e640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9d80a1d80 0x7fd9d80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f198d640 1 --2- 172.21.15.60:0/1712090786 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cf8bf0 0x559822cfd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.246+0000 7fd9f118c640 1 --2- 172.21.15.60:0/1712090786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cfdd60 0x559822d001f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.247+0000 7fd9e2ffd640 1 -- 172.21.15.60:0/2726430517 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fd9dc0ce2e0 con 0x7fd9ec102530 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.247+0000 7fd9f218e640 1 -- 172.21.15.60:0/1712090786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cfdd60 msgr2=0x559822d001f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.247+0000 7fd9f218e640 1 --2- 172.21.15.60:0/1712090786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cfdd60 0x559822d001f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.247+0000 7fd9f218e640 1 -- 172.21.15.60:0/1712090786 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cf8bf0 msgr2=0x559822cfd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.247+0000 7fd9f218e640 1 --2- 172.21.15.60:0/1712090786 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cf8bf0 0x559822cfd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.247+0000 7fd9f218e640 1 -- 172.21.15.60:0/1712090786 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559822c169a0 con 0x559822cf8200 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.247+0000 7fd9f218e640 1 --2- 172.21.15.60:0/1712090786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 0x559822cf8620 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4012d00 tx=0x7fd9e4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd24a6afede814ac server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.247+0000 7fd9f218e640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9d80a1d80 0x7fd9d80a4240 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009fd0 tx=0x7fd9e4017000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.247+0000 7fd9d7fff640 1 -- 172.21.15.60:0/1712090786 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4022070 con 0x559822cf8200 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.247+0000 7fd9d7fff640 1 -- 172.21.15.60:0/1712090786 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9e40124f0 con 0x559822cf8200 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.247+0000 7fd9d7fff640 1 -- 172.21.15.60:0/1712090786 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4005020 con 0x559822cf8200 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.248+0000 7fd9f67cf740 1 -- 172.21.15.60:0/1712090786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 msgr2=0x559822cf8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.248+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/1712090786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 0x559822cf8620 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4012d00 tx=0x7fd9e4010740 comp rx=0 tx=0).stop 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.248+0000 7fd9f67cf740 1 -- 172.21.15.60:0/1712090786 shutdown_connections 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.248+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/1712090786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cfdd60 0x559822d001f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.248+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/1712090786 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cf8bf0 0x559822cfd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.248+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/1712090786 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 0x559822cf8620 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.248+0000 7fd9f67cf740 1 -- 172.21.15.60:0/1712090786 >> 172.21.15.60:0/1712090786 conn(0x559822cef360 msgr2=0x559822cef760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.248+0000 7fd9f67cf740 1 -- 172.21.15.60:0/1712090786 shutdown_connections 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.248+0000 7fd9f67cf740 1 -- 172.21.15.60:0/1712090786 wait complete. 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.248+0000 7fd9f67cf740 1 -- start start 2024-03-04T12:25:32.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.248+0000 7fd9f67cf740 1 Processor -- start 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f67cf740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 0x559822d02710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f67cf740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8bf0 0x559822d02c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f67cf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cfdd60 0x559822d09c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f67cf740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x559822d00cb0 con 0x559822cf8200 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f67cf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x559822c15f90 con 0x559822cfdd60 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f67cf740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x559822d00e30 con 0x559822cf8bf0 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f67cf740 10 client.0 _ll_register_callbacks cb 0x559822c37d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f198d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 0x559822d02710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f198d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 0x559822d02710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48724/0 (socket says 172.21.15.60:48724) 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f198d640 1 -- 172.21.15.60:0/964318407 learned_addr learned my addr 172.21.15.60:0/964318407 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f118c640 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8bf0 0x559822d02c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f218e640 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cfdd60 0x559822d09c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f198d640 1 -- 172.21.15.60:0/964318407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8bf0 msgr2=0x559822d02c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f198d640 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8bf0 0x559822d02c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f198d640 1 -- 172.21.15.60:0/964318407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cfdd60 msgr2=0x559822d09c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f198d640 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cfdd60 0x559822d09c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f198d640 1 -- 172.21.15.60:0/964318407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55982296e2d0 con 0x559822cf8200 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9f198d640 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 0x559822d02710 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8009230 tx=0x7fd9e8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d489fede0e6b3d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.249+0000 7fd9d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9d57fa640 1 -- 172.21.15.60:0/964318407 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8005d50 con 0x559822cf8200 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9d57fa640 1 -- 172.21.15.60:0/964318407 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fd9e8005020 con 0x559822cf8200 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9f67cf740 1 -- 172.21.15.60:0/964318407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9e40124f0 con 0x559822cf8200 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9d57fa640 1 -- 172.21.15.60:0/964318407 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9e804f050 con 0x559822cf8200 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9f67cf740 10 client.15483 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9f67cf740 1 -- 172.21.15.60:0/964318407 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559822a386b0 con 0x559822cf8200 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9d57fa640 1 -- 172.21.15.60:0/964318407 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9e40124f0 con 0x559822cf8200 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9d57fa640 1 -- 172.21.15.60:0/964318407 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd9e8003780 con 0x559822cf8200 2024-03-04T12:25:32.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9f67cf740 10 client.15483 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9f67cf740 10 client.15483 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9f67cf740 20 client.15483 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x559822d0f280 0x559822d0e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9f67cf740 4 client.15483 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9f67cf740 1 -- 172.21.15.60:0/964318407 --> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559822c66d00 con 0x559822d0f280 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.250+0000 7fd9f118c640 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x559822d0f280 0x559822d0e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.251+0000 7fd9f118c640 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x559822d0f280 0x559822d0e360 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9c807e2bfdb768cb server_cookie=f50db53bd9b7e4d8 in_seq=0 out_seq=0 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.251+0000 7fd9d57fa640 10 client.15483 ms_handle_connect on v2:172.21.15.63:6836/3974574804 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.251+0000 7fd9d57fa640 1 -- 172.21.15.60:0/964318407 <== mds.2 v2:172.21.15.63:6836/3974574804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559822c66d00 con 0x559822d0f280 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.251+0000 7fd9d57fa640 10 client.15483 handle_command_reply: tid=0 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.251+0000 7fd9f67cf740 1 client.15483 shutdown 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.252+0000 7fd9f67cf740 1 -- 172.21.15.60:0/964318407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 msgr2=0x559822d02710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.252+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 0x559822d02710 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8009230 tx=0x7fd9e8004270 comp rx=0 tx=0).stop 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.252+0000 7fd9f67cf740 1 -- 172.21.15.60:0/964318407 shutdown_connections 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.252+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x559822d0f280 0x559822d0e360 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.342 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.252+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cfdd60 0x559822d09c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.252+0000 7fd9d57fa640 0 client.15483 ms_handle_reset on v2:172.21.15.63:6836/3974574804 2024-03-04T12:25:32.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.252+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8bf0 0x559822d02c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.252+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/964318407 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8200 0x559822d02710 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.252+0000 7fd9f67cf740 1 -- 172.21.15.60:0/964318407 >> 172.21.15.60:0/964318407 conn(0x559822cef360 msgr2=0x559822cee640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.252+0000 7fd9f67cf740 1 -- 172.21.15.60:0/964318407 shutdown_connections 2024-03-04T12:25:32.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.252+0000 7fd9f67cf740 1 -- 172.21.15.60:0/964318407 wait complete. 2024-03-04T12:25:32.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.252+0000 7fd9f67cf740 20 client.15483 trim_cache size 0 max 16384 2024-03-04T12:25:32.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.267+0000 7fd9f67cf740 1 Processor -- start 2024-03-04T12:25:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.267+0000 7fd9f67cf740 1 -- start start 2024-03-04T12:25:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f67cf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cf8200 0x559822cfa150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f67cf740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8bf0 0x559822cfa690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f67cf740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cfdd60 0x559822d09bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f67cf740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x559822d0eaa0 con 0x559822cfdd60 2024-03-04T12:25:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f67cf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x559822d0f100 con 0x559822cf8200 2024-03-04T12:25:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f67cf740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x559822d0f760 con 0x559822cf8bf0 2024-03-04T12:25:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f218e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cfdd60 0x559822d09bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f218e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cfdd60 0x559822d09bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48726/0 (socket says 172.21.15.60:48726) 2024-03-04T12:25:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f218e640 1 -- 172.21.15.60:0/2380755226 learned_addr learned my addr 172.21.15.60:0/2380755226 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f118c640 1 --2- 172.21.15.60:0/2380755226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8bf0 0x559822cfa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f218e640 1 -- 172.21.15.60:0/2380755226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8bf0 msgr2=0x559822cfa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f218e640 1 --2- 172.21.15.60:0/2380755226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8bf0 0x559822cfa690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f218e640 1 -- 172.21.15.60:0/2380755226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cf8200 msgr2=0x559822cfa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f218e640 1 --2- 172.21.15.60:0/2380755226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cf8200 0x559822cfa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f218e640 1 -- 172.21.15.60:0/2380755226 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559822d17fb0 con 0x559822cfdd60 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9f218e640 1 --2- 172.21.15.60:0/2380755226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cfdd60 0x559822d09bb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4012ba0 tx=0x7fd9e400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4475fa4a2959929 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9d77fe640 1 -- 172.21.15.60:0/2380755226 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4022040 con 0x559822cfdd60 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9d77fe640 1 -- 172.21.15.60:0/2380755226 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9e40124f0 con 0x559822cfdd60 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.268+0000 7fd9d77fe640 1 -- 172.21.15.60:0/2380755226 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4005040 con 0x559822cfdd60 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.269+0000 7fd9f67cf740 1 -- 172.21.15.60:0/2380755226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cfdd60 msgr2=0x559822d09bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.269+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/2380755226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cfdd60 0x559822d09bb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4012ba0 tx=0x7fd9e400cf70 comp rx=0 tx=0).stop 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.269+0000 7fd9f67cf740 1 -- 172.21.15.60:0/2380755226 shutdown_connections 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.269+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/2380755226 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cfdd60 0x559822d09bb0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.269+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/2380755226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8bf0 0x559822cfa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.350 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.269+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/2380755226 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cf8200 0x559822cfa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.269+0000 7fd9f67cf740 1 -- 172.21.15.60:0/2380755226 >> 172.21.15.60:0/2380755226 conn(0x559822cef360 msgr2=0x559822cef740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.269+0000 7fd9f67cf740 1 -- 172.21.15.60:0/2380755226 shutdown_connections 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.269+0000 7fd9f67cf740 1 -- 172.21.15.60:0/2380755226 wait complete. 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.269+0000 7fd9f67cf740 1 -- start start 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.269+0000 7fd9f67cf740 1 Processor -- start 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f67cf740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8200 0x559822d07120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f67cf740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8bf0 0x559822d07660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f67cf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cfdd60 0x559822d08670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f67cf740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x559822d10420 con 0x559822cf8bf0 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f67cf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x559822d0fdc0 con 0x559822cfdd60 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f67cf740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x559822d10a80 con 0x559822cf8200 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f67cf740 10 client.0 _ll_register_callbacks cb 0x7fd9e4009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f118c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8bf0 0x559822d07660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f118c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8bf0 0x559822d07660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48728/0 (socket says 172.21.15.60:48728) 2024-03-04T12:25:32.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f118c640 1 -- 172.21.15.60:0/687198834 learned_addr learned my addr 172.21.15.60:0/687198834 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f218e640 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cfdd60 0x559822d08670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f198d640 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8200 0x559822d07120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f118c640 1 -- 172.21.15.60:0/687198834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8200 msgr2=0x559822d07120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f118c640 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8200 0x559822d07120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f118c640 1 -- 172.21.15.60:0/687198834 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cfdd60 msgr2=0x559822d08670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f118c640 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cfdd60 0x559822d08670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f118c640 1 -- 172.21.15.60:0/687198834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559822d08c60 con 0x559822cf8bf0 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f118c640 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8bf0 0x559822d07660 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0970e0 tx=0x7fd9dc01d620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83892f10b8bf5ebc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9d57fa640 1 -- 172.21.15.60:0/687198834 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc09edd0 con 0x559822cf8bf0 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9d57fa640 1 -- 172.21.15.60:0/687198834 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fd9dc098f80 con 0x559822cf8bf0 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f67cf740 1 -- 172.21.15.60:0/687198834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9e40124f0 con 0x559822cf8bf0 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9d57fa640 1 -- 172.21.15.60:0/687198834 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9dc0a3420 con 0x559822cf8bf0 2024-03-04T12:25:32.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f67cf740 10 client.15489 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.270+0000 7fd9f67cf740 1 -- 172.21.15.60:0/687198834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559822d0a030 con 0x559822cf8bf0 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.271+0000 7fd9d57fa640 1 -- 172.21.15.60:0/687198834 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9e40124f0 con 0x559822cf8bf0 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.271+0000 7fd9d57fa640 1 -- 172.21.15.60:0/687198834 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd9dc0152b0 con 0x559822cf8bf0 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.271+0000 7fd9f67cf740 10 client.15489 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.271+0000 7fd9f67cf740 10 client.15489 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.271+0000 7fd9f67cf740 20 client.15489 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.271+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x559822d6b280 0x559822d66fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.271+0000 7fd9f67cf740 4 client.15489 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.271+0000 7fd9f67cf740 1 -- 172.21.15.60:0/687198834 --> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559822c61050 con 0x559822d6b280 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.271+0000 7fd9f198d640 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x559822d6b280 0x559822d66fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.271+0000 7fd9d57fa640 10 client.15489 ms_handle_connect on v2:172.21.15.63:6836/3974574804 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.271+0000 7fd9f198d640 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x559822d6b280 0x559822d66fa0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d57d8d5c75a49db7 server_cookie=96312fa6959f731 in_seq=0 out_seq=0 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9d57fa640 1 -- 172.21.15.60:0/687198834 <== mds.2 v2:172.21.15.63:6836/3974574804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x559822c61050 con 0x559822d6b280 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9d57fa640 10 client.15489 handle_command_reply: tid=0 2024-03-04T12:25:32.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f67cf740 1 client.15489 shutdown 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f67cf740 1 -- 172.21.15.60:0/687198834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8bf0 msgr2=0x559822d07660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8bf0 0x559822d07660 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc0970e0 tx=0x7fd9dc01d620 comp rx=0 tx=0).stop 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f67cf740 1 -- 172.21.15.60:0/687198834 shutdown_connections 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x559822d6b280 0x559822d66fa0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559822cfdd60 0x559822d08670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9d57fa640 0 client.15489 ms_handle_reset on v2:172.21.15.63:6836/3974574804 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x559822cf8bf0 0x559822d07660 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f67cf740 1 --2- 172.21.15.60:0/687198834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559822cf8200 0x559822d07120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f67cf740 1 -- 172.21.15.60:0/687198834 >> 172.21.15.60:0/687198834 conn(0x559822cef360 msgr2=0x559822cfa150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f67cf740 1 -- 172.21.15.60:0/687198834 shutdown_connections 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f67cf740 1 -- 172.21.15.60:0/687198834 wait complete. 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f67cf740 20 client.15489 trim_cache size 0 max 16384 2024-03-04T12:25:32.354 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:25:32.355 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:25:32.355 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 2, 2024-03-04T12:25:32.355 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 24479, 2024-03-04T12:25:32.355 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-03-04T12:25:32.355 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-03-04T12:25:32.355 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-03-04T12:25:32.355 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 113.266152185, 2024-03-04T12:25:32.355 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 28, 2024-03-04T12:25:32.355 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 93, 2024-03-04T12:25:32.355 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 89, 2024-03-04T12:25:32.355 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 157.21017410300001 2024-03-04T12:25:32.355 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:25:32.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/2726430517 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9d80a1d80 msgr2=0x7fd9d80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f3bf1640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9d80a1d80 0x7fd9d80a4240 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009fd0 tx=0x7fd9e4017000 comp rx=0 tx=0).stop 2024-03-04T12:25:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/2726430517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9ec102530 msgr2=0x7fd9ec197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f3bf1640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9ec102530 0x7fd9ec197b20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00c920 tx=0x7fd9dc00cde0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/2726430517 shutdown_connections 2024-03-04T12:25:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f3bf1640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd9d80a1d80 0x7fd9d80a4240 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f3bf1640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd9ec102ef0 0x7fd9ec19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f3bf1640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd9ec102530 0x7fd9ec197b20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.272+0000 7fd9f3bf1640 1 --2- 172.21.15.60:0/2726430517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec101340 0x7fd9ec1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.273+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/2726430517 >> 172.21.15.60:0/2726430517 conn(0x7fd9ec0fc8b0 msgr2=0x7fd9ec1057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.273+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/2726430517 shutdown_connections 2024-03-04T12:25:32.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.273+0000 7fd9f3bf1640 1 -- 172.21.15.60:0/2726430517 wait complete. 2024-03-04T12:25:32.357 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-04T12:25:32.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:31 smithi063 ceph-mon[36717]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:25:32.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:31 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/95808077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:32.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:31 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2489964134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:32.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:31 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2384297746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:32.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:31 smithi086 ceph-mon[36851]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:25:32.358 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:31 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/95808077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:32.359 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:31 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2489964134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:32.359 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:31 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2384297746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:32.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.455+0000 7f26f8da9640 1 -- 172.21.15.60:0/288508804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26f41004f0 msgr2=0x7f26f4100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.455+0000 7f26f8da9640 1 --2- 172.21.15.60:0/288508804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26f41004f0 0x7f26f4100970 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f26dc009930 tx=0x7f26dc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.455+0000 7f26f8da9640 1 -- 172.21.15.60:0/288508804 shutdown_connections 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.455+0000 7f26f8da9640 1 --2- 172.21.15.60:0/288508804 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f4100eb0 0x7f26f4108750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.455+0000 7f26f8da9640 1 --2- 172.21.15.60:0/288508804 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26f41004f0 0x7f26f4100970 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.455+0000 7f26f8da9640 1 --2- 172.21.15.60:0/288508804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f4101850 0x7f26f4101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.455+0000 7f26f8da9640 1 -- 172.21.15.60:0/288508804 >> 172.21.15.60:0/288508804 conn(0x7f26f40faf40 msgr2=0x7f26f40fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.455+0000 7f26f8da9640 1 -- 172.21.15.60:0/288508804 shutdown_connections 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.455+0000 7f26f8da9640 1 -- 172.21.15.60:0/288508804 wait complete. 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f8da9640 1 Processor -- start 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f8da9640 1 -- start start 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f8da9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26f41004f0 0x7f26f41998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f8da9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f4100eb0 0x7f26f4199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f8da9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f4101850 0x7f26f41a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f8da9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f410bf20 con 0x7f26f41004f0 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f8da9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f410bda0 con 0x7f26f4101850 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f8da9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f410c0a0 con 0x7f26f4100eb0 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26f41004f0 0x7f26f41998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26f41004f0 0x7f26f41998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48734/0 (socket says 172.21.15.60:48734) 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f259c640 1 -- 172.21.15.60:0/1005975526 learned_addr learned my addr 172.21.15.60:0/1005975526 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f1d9b640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f4100eb0 0x7f26f4199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f2d9d640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f4101850 0x7f26f41a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f259c640 1 -- 172.21.15.60:0/1005975526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f4100eb0 msgr2=0x7f26f4199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f259c640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f4100eb0 0x7f26f4199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f259c640 1 -- 172.21.15.60:0/1005975526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f4101850 msgr2=0x7f26f41a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f259c640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f4101850 0x7f26f41a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f259c640 1 -- 172.21.15.60:0/1005975526 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f41a14f0 con 0x7f26f41004f0 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f1d9b640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f4100eb0 0x7f26f4199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f2d9d640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f4101850 0x7f26f41a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:32.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26f259c640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26f41004f0 0x7f26f41998d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00d950 tx=0x7f26ec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26eb7fe640 1 -- 172.21.15.60:0/1005975526 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec017020 con 0x7f26f41004f0 2024-03-04T12:25:32.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.456+0000 7f26eb7fe640 1 -- 172.21.15.60:0/1005975526 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26ec009c30 con 0x7f26f41004f0 2024-03-04T12:25:32.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.457+0000 7f26f8da9640 1 -- 172.21.15.60:0/1005975526 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f41a17e0 con 0x7f26f41004f0 2024-03-04T12:25:32.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.457+0000 7f26eb7fe640 1 -- 172.21.15.60:0/1005975526 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec00f720 con 0x7f26f41004f0 2024-03-04T12:25:32.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.457+0000 7f26f8da9640 1 -- 172.21.15.60:0/1005975526 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f41a1d30 con 0x7f26f41004f0 2024-03-04T12:25:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.459+0000 7f26eb7fe640 1 -- 172.21.15.60:0/1005975526 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f26ec01a020 con 0x7f26f41004f0 2024-03-04T12:25:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.459+0000 7f26fb984740 1 Processor -- start 2024-03-04T12:25:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.459+0000 7f26fb984740 1 -- start start 2024-03-04T12:25:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.459+0000 7f26eb7fe640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f26e00a1d80 0x7f26e00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.459+0000 7f26fb984740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a641df0 0x55e24a642210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.459+0000 7f26fb984740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 0x55e24a6472e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.459+0000 7f26fb984740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a647950 0x55e24a649de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.459+0000 7f26fb984740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24a5a77c0 con 0x55e24a6427e0 2024-03-04T12:25:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.459+0000 7f26fb984740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24a550b50 con 0x55e24a647950 2024-03-04T12:25:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.459+0000 7f26fb984740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24a5a3860 con 0x55e24a641df0 2024-03-04T12:25:32.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f2d9d640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f26e00a1d80 0x7f26e00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26eb7fe640 1 -- 172.21.15.60:0/1005975526 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f26ec0ce0a0 con 0x7f26f41004f0 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 0x55e24a6472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 0x55e24a6472e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48746/0 (socket says 172.21.15.60:48746) 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f1d9b640 1 -- 172.21.15.60:0/1656001954 learned_addr learned my addr 172.21.15.60:0/1656001954 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f259c640 1 --2- 172.21.15.60:0/1656001954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a641df0 0x55e24a642210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f2d9d640 1 --2- 172.21.15.60:0/1656001954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a647950 0x55e24a649de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f1d9b640 1 -- 172.21.15.60:0/1656001954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a641df0 msgr2=0x55e24a642210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f1d9b640 1 --2- 172.21.15.60:0/1656001954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a641df0 0x55e24a642210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f1d9b640 1 -- 172.21.15.60:0/1656001954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a647950 msgr2=0x55e24a649de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f1d9b640 1 --2- 172.21.15.60:0/1656001954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a647950 0x55e24a649de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f1d9b640 1 -- 172.21.15.60:0/1656001954 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e24a5649a0 con 0x55e24a6427e0 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f259c640 1 --2- 172.21.15.60:0/1656001954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a641df0 0x55e24a642210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f2d9d640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f26e00a1d80 0x7f26e00a4240 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009270 tx=0x7f26e4013a80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26f1d9b640 1 --2- 172.21.15.60:0/1656001954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 0x55e24a6472e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f26dc036040 tx=0x7f26dc009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b27264161f9b68 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26e8ff9640 1 -- 172.21.15.60:0/1656001954 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc02cc40 con 0x55e24a6427e0 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26e8ff9640 1 -- 172.21.15.60:0/1656001954 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26dc005040 con 0x55e24a6427e0 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.460+0000 7f26e8ff9640 1 -- 172.21.15.60:0/1656001954 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc0049b0 con 0x55e24a6427e0 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.461+0000 7f26fb984740 1 -- 172.21.15.60:0/1656001954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 msgr2=0x55e24a6472e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.461+0000 7f26fb984740 1 --2- 172.21.15.60:0/1656001954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 0x55e24a6472e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f26dc036040 tx=0x7f26dc009630 comp rx=0 tx=0).stop 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.461+0000 7f26fb984740 1 -- 172.21.15.60:0/1656001954 shutdown_connections 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.461+0000 7f26fb984740 1 --2- 172.21.15.60:0/1656001954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a647950 0x55e24a649de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.461+0000 7f26fb984740 1 --2- 172.21.15.60:0/1656001954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 0x55e24a6472e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.461+0000 7f26fb984740 1 --2- 172.21.15.60:0/1656001954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a641df0 0x55e24a642210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.461+0000 7f26fb984740 1 -- 172.21.15.60:0/1656001954 >> 172.21.15.60:0/1656001954 conn(0x55e24a63d360 msgr2=0x55e24a63d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.461+0000 7f26fb984740 1 -- 172.21.15.60:0/1656001954 shutdown_connections 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.461+0000 7f26fb984740 1 -- 172.21.15.60:0/1656001954 wait complete. 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.461+0000 7f26fb984740 1 -- start start 2024-03-04T12:25:32.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26fb984740 1 Processor -- start 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26fb984740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a641df0 0x55e24a64c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26fb984740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 0x55e24a64c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26fb984740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a647950 0x55e24a6537f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26fb984740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24a64a8a0 con 0x55e24a6427e0 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26fb984740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24a563f90 con 0x55e24a641df0 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26fb984740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24a64aa20 con 0x55e24a647950 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26fb984740 10 client.0 _ll_register_callbacks cb 0x55e24a585d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26f1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 0x55e24a64c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26f1d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 0x55e24a64c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48762/0 (socket says 172.21.15.60:48762) 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26f1d9b640 1 -- 172.21.15.60:0/970064259 learned_addr learned my addr 172.21.15.60:0/970064259 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26f259c640 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a641df0 0x55e24a64c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26f2d9d640 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a647950 0x55e24a6537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26f1d9b640 1 -- 172.21.15.60:0/970064259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a647950 msgr2=0x55e24a6537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26f1d9b640 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a647950 0x55e24a6537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26f1d9b640 1 -- 172.21.15.60:0/970064259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a641df0 msgr2=0x55e24a64c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26f1d9b640 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a641df0 0x55e24a64c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26f1d9b640 1 -- 172.21.15.60:0/970064259 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e24a2bc2d0 con 0x55e24a6427e0 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26f259c640 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a641df0 0x55e24a64c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26f1d9b640 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 0x55e24a64c7b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f26dc000c00 tx=0x7f26dc0042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07e53fb09816ff7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26d9ffb640 1 -- 172.21.15.60:0/970064259 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc005600 con 0x55e24a6427e0 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26d9ffb640 1 -- 172.21.15.60:0/970064259 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f26dc004730 con 0x55e24a6427e0 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.462+0000 7f26fb984740 1 -- 172.21.15.60:0/970064259 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f26dc005040 con 0x55e24a6427e0 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.463+0000 7f26d9ffb640 1 -- 172.21.15.60:0/970064259 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26dc056050 con 0x55e24a6427e0 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.463+0000 7f26fb984740 10 client.15501 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.463+0000 7f26fb984740 1 -- 172.21.15.60:0/970064259 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e24a3866b0 con 0x55e24a6427e0 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.463+0000 7f26d9ffb640 1 -- 172.21.15.60:0/970064259 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26dc005040 con 0x55e24a6427e0 2024-03-04T12:25:32.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.463+0000 7f26d9ffb640 1 -- 172.21.15.60:0/970064259 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f26dc034440 con 0x55e24a6427e0 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.463+0000 7f26fb984740 10 client.15501 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.463+0000 7f26fb984740 10 client.15501 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.463+0000 7f26fb984740 20 client.15501 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.463+0000 7f26fb984740 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55e24a657ce0 0x55e24a65cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.463+0000 7f26fb984740 4 client.15501 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.463+0000 7f26fb984740 1 -- 172.21.15.60:0/970064259 --> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e24a5b4d00 con 0x55e24a657ce0 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.463+0000 7f26f259c640 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55e24a657ce0 0x55e24a65cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26d9ffb640 10 client.15501 ms_handle_connect on v2:172.21.15.86:6838/3347800322 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26f259c640 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55e24a657ce0 0x55e24a65cdf0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=20d15c9d9eea02fd server_cookie=e4f0b38d5978caba in_seq=0 out_seq=0 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26d9ffb640 1 -- 172.21.15.60:0/970064259 <== mds.1 v2:172.21.15.86:6838/3347800322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e24a5b4d00 con 0x55e24a657ce0 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26d9ffb640 10 client.15501 handle_command_reply: tid=0 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26fb984740 1 client.15501 shutdown 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26fb984740 1 -- 172.21.15.60:0/970064259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 msgr2=0x55e24a64c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26fb984740 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 0x55e24a64c7b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f26dc000c00 tx=0x7f26dc0042e0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26fb984740 1 -- 172.21.15.60:0/970064259 shutdown_connections 2024-03-04T12:25:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26fb984740 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a647950 0x55e24a6537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26fb984740 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55e24a657ce0 0x55e24a65cdf0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26fb984740 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a6427e0 0x55e24a64c7b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26fb984740 1 --2- 172.21.15.60:0/970064259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a641df0 0x55e24a64c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26d9ffb640 0 client.15501 ms_handle_reset on v2:172.21.15.86:6838/3347800322 2024-03-04T12:25:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.464+0000 7f26fb984740 1 -- 172.21.15.60:0/970064259 >> 172.21.15.60:0/970064259 conn(0x55e24a63d360 msgr2=0x55e24a63c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.465+0000 7f26fb984740 1 -- 172.21.15.60:0/970064259 shutdown_connections 2024-03-04T12:25:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.465+0000 7f26fb984740 1 -- 172.21.15.60:0/970064259 wait complete. 2024-03-04T12:25:32.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.465+0000 7f26fb984740 20 client.15501 trim_cache size 0 max 16384 2024-03-04T12:25:32.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.480+0000 7f26fb984740 1 Processor -- start 2024-03-04T12:25:32.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.480+0000 7f26fb984740 1 -- start start 2024-03-04T12:25:32.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26fb984740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 0x55e24a64b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26fb984740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a6427e0 0x55e24a6537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26fb984740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a647950 0x55e24a643c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26fb984740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24a658190 con 0x55e24a641df0 2024-03-04T12:25:32.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26fb984740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24a6587f0 con 0x55e24a6427e0 2024-03-04T12:25:32.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26fb984740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24a658e50 con 0x55e24a647950 2024-03-04T12:25:32.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26f259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 0x55e24a64b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26f259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 0x55e24a64b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48774/0 (socket says 172.21.15.60:48774) 2024-03-04T12:25:32.493 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26f259c640 1 -- 172.21.15.60:0/2611288233 learned_addr learned my addr 172.21.15.60:0/2611288233 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26f259c640 1 -- 172.21.15.60:0/2611288233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a647950 msgr2=0x55e24a643c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26f259c640 1 --2- 172.21.15.60:0/2611288233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a647950 0x55e24a643c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26f259c640 1 -- 172.21.15.60:0/2611288233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a6427e0 msgr2=0x55e24a6537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26f259c640 1 --2- 172.21.15.60:0/2611288233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a6427e0 0x55e24a6537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26f259c640 1 -- 172.21.15.60:0/2611288233 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e24a642bc0 con 0x55e24a641df0 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26f259c640 1 --2- 172.21.15.60:0/2611288233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 0x55e24a64b890 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f26ec09a9d0 tx=0x7f26ec0cb6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b0b5b1820e8492a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26dbfff640 1 -- 172.21.15.60:0/2611288233 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0a31e0 con 0x55e24a641df0 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26dbfff640 1 -- 172.21.15.60:0/2611288233 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26ec0cb920 con 0x55e24a641df0 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.481+0000 7f26dbfff640 1 -- 172.21.15.60:0/2611288233 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec06ace0 con 0x55e24a641df0 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 -- 172.21.15.60:0/2611288233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 msgr2=0x55e24a64b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 --2- 172.21.15.60:0/2611288233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 0x55e24a64b890 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f26ec09a9d0 tx=0x7f26ec0cb6e0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 -- 172.21.15.60:0/2611288233 shutdown_connections 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 --2- 172.21.15.60:0/2611288233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a647950 0x55e24a643c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 --2- 172.21.15.60:0/2611288233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a6427e0 0x55e24a6537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 --2- 172.21.15.60:0/2611288233 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 0x55e24a64b890 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 -- 172.21.15.60:0/2611288233 >> 172.21.15.60:0/2611288233 conn(0x55e24a63d360 msgr2=0x55e24a63d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 -- 172.21.15.60:0/2611288233 shutdown_connections 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 -- 172.21.15.60:0/2611288233 wait complete. 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 -- start start 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 Processor -- start 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 0x55e24a6534e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a6427e0 0x55e24a64ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a647950 0x55e24a650520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24a659b10 con 0x55e24a641df0 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24a6594b0 con 0x55e24a647950 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.482+0000 7f26fb984740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24a65a170 con 0x55e24a6427e0 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26fb984740 10 client.0 _ll_register_callbacks cb 0x7f26dc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26f259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 0x55e24a6534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26f259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 0x55e24a6534e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48782/0 (socket says 172.21.15.60:48782) 2024-03-04T12:25:32.495 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26f259c640 1 -- 172.21.15.60:0/3473861270 learned_addr learned my addr 172.21.15.60:0/3473861270 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26f1d9b640 1 --2- 172.21.15.60:0/3473861270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a6427e0 0x55e24a64ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26f259c640 1 -- 172.21.15.60:0/3473861270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a6427e0 msgr2=0x55e24a64ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26f259c640 1 --2- 172.21.15.60:0/3473861270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a6427e0 0x55e24a64ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26f259c640 1 -- 172.21.15.60:0/3473861270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a647950 msgr2=0x55e24a650520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26f259c640 1 --2- 172.21.15.60:0/3473861270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a647950 0x55e24a650520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26f259c640 1 -- 172.21.15.60:0/3473861270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e24a650b90 con 0x55e24a641df0 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26f259c640 1 --2- 172.21.15.60:0/3473861270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 0x55e24a6534e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0a2cb0 tx=0x7f26ec0cb7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610fcf5525ac1960 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26d9ffb640 1 -- 172.21.15.60:0/3473861270 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0970b0 con 0x55e24a641df0 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26d9ffb640 1 -- 172.21.15.60:0/3473861270 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f26ec06ace0 con 0x55e24a641df0 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26fb984740 1 -- 172.21.15.60:0/3473861270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f26ec0cb920 con 0x55e24a641df0 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26d9ffb640 1 -- 172.21.15.60:0/3473861270 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26ec097250 con 0x55e24a641df0 2024-03-04T12:25:32.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26fb984740 10 client.15507 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26fb984740 1 -- 172.21.15.60:0/3473861270 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e24a6afb90 con 0x55e24a641df0 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.483+0000 7f26d9ffb640 1 -- 172.21.15.60:0/3473861270 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26ec0cb920 con 0x55e24a641df0 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.484+0000 7f26d9ffb640 1 -- 172.21.15.60:0/3473861270 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f26ec06c040 con 0x55e24a641df0 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.484+0000 7f26fb984740 10 client.15507 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.484+0000 7f26fb984740 10 client.15507 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.484+0000 7f26fb984740 20 client.15507 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.484+0000 7f26fb984740 1 --2- 172.21.15.60:0/3473861270 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55e24a6b3e50 0x55e24a6afd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.484+0000 7f26fb984740 4 client.15507 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.484+0000 7f26fb984740 1 -- 172.21.15.60:0/3473861270 --> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e24a5af050 con 0x55e24a6b3e50 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.484+0000 7f26f1d9b640 1 --2- 172.21.15.60:0/3473861270 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55e24a6b3e50 0x55e24a6afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.484+0000 7f26d9ffb640 10 client.15507 ms_handle_connect on v2:172.21.15.86:6838/3347800322 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.484+0000 7f26f1d9b640 1 --2- 172.21.15.60:0/3473861270 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55e24a6b3e50 0x55e24a6afd60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5181ed2e0bee7380 server_cookie=c365311fc366aa85 in_seq=0 out_seq=0 2024-03-04T12:25:32.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.484+0000 7f26d9ffb640 1 -- 172.21.15.60:0/3473861270 <== mds.1 v2:172.21.15.86:6838/3347800322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55e24a5af050 con 0x55e24a6b3e50 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26d9ffb640 10 client.15507 handle_command_reply: tid=0 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26fb984740 1 client.15507 shutdown 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26fb984740 1 -- 172.21.15.60:0/3473861270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 msgr2=0x55e24a6534e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26fb984740 1 --2- 172.21.15.60:0/3473861270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 0x55e24a6534e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0a2cb0 tx=0x7f26ec0cb7b0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26fb984740 1 -- 172.21.15.60:0/3473861270 shutdown_connections 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26fb984740 1 --2- 172.21.15.60:0/3473861270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e24a647950 0x55e24a650520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26fb984740 1 --2- 172.21.15.60:0/3473861270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e24a6427e0 0x55e24a64ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26fb984740 1 --2- 172.21.15.60:0/3473861270 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55e24a6b3e50 0x55e24a6afd60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26fb984740 1 --2- 172.21.15.60:0/3473861270 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e24a641df0 0x55e24a6534e0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26d9ffb640 0 client.15507 ms_handle_reset on v2:172.21.15.86:6838/3347800322 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26fb984740 1 -- 172.21.15.60:0/3473861270 >> 172.21.15.60:0/3473861270 conn(0x55e24a63d360 msgr2=0x55e24a643c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26fb984740 1 -- 172.21.15.60:0/3473861270 shutdown_connections 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26fb984740 1 -- 172.21.15.60:0/3473861270 wait complete. 2024-03-04T12:25:32.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26fb984740 20 client.15507 trim_cache size 0 max 16384 2024-03-04T12:25:32.499 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:25:32.499 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:25:32.499 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 1, 2024-03-04T12:25:32.499 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 24487, 2024-03-04T12:25:32.499 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-03-04T12:25:32.499 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-03-04T12:25:32.499 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-03-04T12:25:32.499 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 114.48951706699999, 2024-03-04T12:25:32.499 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 28, 2024-03-04T12:25:32.499 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 93, 2024-03-04T12:25:32.500 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 89, 2024-03-04T12:25:32.500 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 154.67549662900001 2024-03-04T12:25:32.500 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:25:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26f8da9640 1 -- 172.21.15.60:0/1005975526 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f26e00a1d80 msgr2=0x7f26e00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26f8da9640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f26e00a1d80 0x7f26e00a4240 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009270 tx=0x7f26e4013a80 comp rx=0 tx=0).stop 2024-03-04T12:25:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26f8da9640 1 -- 172.21.15.60:0/1005975526 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26f41004f0 msgr2=0x7f26f41998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26f8da9640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26f41004f0 0x7f26f41998d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00d950 tx=0x7f26ec00de10 comp rx=0 tx=0).stop 2024-03-04T12:25:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26f8da9640 1 -- 172.21.15.60:0/1005975526 shutdown_connections 2024-03-04T12:25:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26f8da9640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f26e00a1d80 0x7f26e00a4240 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26f8da9640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26f4101850 0x7f26f41a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26f8da9640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f4100eb0 0x7f26f4199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26f8da9640 1 --2- 172.21.15.60:0/1005975526 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f26f41004f0 0x7f26f41998d0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26f8da9640 1 -- 172.21.15.60:0/1005975526 >> 172.21.15.60:0/1005975526 conn(0x7f26f40faf40 msgr2=0x7f26f40fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26f8da9640 1 -- 172.21.15.60:0/1005975526 shutdown_connections 2024-03-04T12:25:32.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.485+0000 7f26f8da9640 1 -- 172.21.15.60:0/1005975526 wait complete. 2024-03-04T12:25:32.503 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-04T12:25:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.600+0000 7f214e5ec640 1 -- 172.21.15.60:0/316674632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2148105120 msgr2=0x7f214810b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.600+0000 7f214e5ec640 1 --2- 172.21.15.60:0/316674632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2148105120 0x7f214810b9b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2140009ca0 tx=0x7f214002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.600+0000 7f214e5ec640 1 -- 172.21.15.60:0/316674632 shutdown_connections 2024-03-04T12:25:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.600+0000 7f214e5ec640 1 --2- 172.21.15.60:0/316674632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2148105120 0x7f214810b9b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.609 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.600+0000 7f214e5ec640 1 --2- 172.21.15.60:0/316674632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2148104760 0x7f2148104be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.600+0000 7f214e5ec640 1 --2- 172.21.15.60:0/316674632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2148103570 0x7f2148103990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.600+0000 7f214e5ec640 1 -- 172.21.15.60:0/316674632 >> 172.21.15.60:0/316674632 conn(0x7f21480feae0 msgr2=0x7f2148100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.600+0000 7f214e5ec640 1 -- 172.21.15.60:0/316674632 shutdown_connections 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.600+0000 7f214e5ec640 1 -- 172.21.15.60:0/316674632 wait complete. 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.600+0000 7f214e5ec640 1 Processor -- start 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.600+0000 7f214e5ec640 1 -- start start 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214e5ec640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2148103570 0x7f2148199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214e5ec640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2148104760 0x7f2148199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214e5ec640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2148105120 0x7f21481a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214e5ec640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214810f240 con 0x7f2148105120 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214e5ec640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214810f0c0 con 0x7f2148104760 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214e5ec640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214810f3c0 con 0x7f2148103570 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214cb89640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2148105120 0x7f21481a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214cb89640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2148105120 0x7f21481a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48790/0 (socket says 172.21.15.60:48790) 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214cb89640 1 -- 172.21.15.60:0/1214030421 learned_addr learned my addr 172.21.15.60:0/1214030421 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.610 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f2147fff640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2148103570 0x7f2148199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f21477fe640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2148104760 0x7f2148199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214cb89640 1 -- 172.21.15.60:0/1214030421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2148103570 msgr2=0x7f2148199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214cb89640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2148103570 0x7f2148199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214cb89640 1 -- 172.21.15.60:0/1214030421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2148104760 msgr2=0x7f2148199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214cb89640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2148104760 0x7f2148199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214cb89640 1 -- 172.21.15.60:0/1214030421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21481a1470 con 0x7f2148105120 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f21477fe640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2148104760 0x7f2148199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214cb89640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2148105120 0x7f21481a0e00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f214002c700 tx=0x7f2140032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f21457fa640 1 -- 172.21.15.60:0/1214030421 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2140039040 con 0x7f2148105120 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f21457fa640 1 -- 172.21.15.60:0/1214030421 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2140044070 con 0x7f2148105120 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214e5ec640 1 -- 172.21.15.60:0/1214030421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21481a1700 con 0x7f2148105120 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f21457fa640 1 -- 172.21.15.60:0/1214030421 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21400042a0 con 0x7f2148105120 2024-03-04T12:25:32.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.601+0000 7f214e5ec640 1 -- 172.21.15.60:0/1214030421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21481a1bf0 con 0x7f2148105120 2024-03-04T12:25:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21457fa640 1 -- 172.21.15.60:0/1214030421 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f2140003970 con 0x7f2148105120 2024-03-04T12:25:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21511c5740 1 Processor -- start 2024-03-04T12:25:32.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21511c5740 1 -- start start 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21511c5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 0x564ec4ef9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21511c5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4efe2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21511c5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 0x564ec4f00de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21511c5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec4e5e7c0 con 0x564ec4efe950 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21511c5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec4e07b50 con 0x564ec4ef97e0 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21511c5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec4e5a860 con 0x564ec4ef8df0 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21457fa640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21300a1d80 0x7f21300a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 0x564ec4f00de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 0x564ec4f00de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48800/0 (socket says 172.21.15.60:48800) 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.604+0000 7f21477fe640 1 -- 172.21.15.60:0/3605013044 learned_addr learned my addr 172.21.15.60:0/3605013044 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f214cb89640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21300a1d80 0x7f21300a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f21477fe640 1 --2- 172.21.15.60:0/3605013044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 0x564ec4ef9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f2147fff640 1 --2- 172.21.15.60:0/3605013044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4efe2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f21477fe640 1 -- 172.21.15.60:0/3605013044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 msgr2=0x564ec4ef9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f21457fa640 1 -- 172.21.15.60:0/1214030421 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f21400f0850 con 0x7f2148105120 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f21477fe640 1 --2- 172.21.15.60:0/3605013044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 0x564ec4ef9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f21477fe640 1 -- 172.21.15.60:0/3605013044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 msgr2=0x564ec4efe2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f21477fe640 1 --2- 172.21.15.60:0/3605013044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4efe2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f21477fe640 1 -- 172.21.15.60:0/3605013044 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ec4e1b9a0 con 0x564ec4efe950 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f21477fe640 1 --2- 172.21.15.60:0/3605013044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 0x564ec4f00de0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f213c006fd0 tx=0x7f213c00c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e019276f2676eb17 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f214cb89640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21300a1d80 0x7f21300a4240 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f21400c5400 tx=0x7f21400902e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f212e7fc640 1 -- 172.21.15.60:0/3605013044 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c01b070 con 0x564ec4efe950 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f212e7fc640 1 -- 172.21.15.60:0/3605013044 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f213c00ee10 con 0x564ec4efe950 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.605+0000 7f212e7fc640 1 -- 172.21.15.60:0/3605013044 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c00f040 con 0x564ec4efe950 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.606+0000 7f21511c5740 1 -- 172.21.15.60:0/3605013044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 msgr2=0x564ec4f00de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.606+0000 7f21511c5740 1 --2- 172.21.15.60:0/3605013044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 0x564ec4f00de0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f213c006fd0 tx=0x7f213c00c410 comp rx=0 tx=0).stop 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.606+0000 7f21511c5740 1 -- 172.21.15.60:0/3605013044 shutdown_connections 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.606+0000 7f21511c5740 1 --2- 172.21.15.60:0/3605013044 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 0x564ec4f00de0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.606+0000 7f21511c5740 1 --2- 172.21.15.60:0/3605013044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4efe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.606+0000 7f21511c5740 1 --2- 172.21.15.60:0/3605013044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 0x564ec4ef9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.606+0000 7f21511c5740 1 -- 172.21.15.60:0/3605013044 >> 172.21.15.60:0/3605013044 conn(0x564ec4ef4360 msgr2=0x564ec4ef4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.606+0000 7f21511c5740 1 -- 172.21.15.60:0/3605013044 shutdown_connections 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.606+0000 7f21511c5740 1 -- 172.21.15.60:0/3605013044 wait complete. 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.606+0000 7f21511c5740 1 -- start start 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.606+0000 7f21511c5740 1 Processor -- start 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21511c5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef8df0 0x564ec4f032e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21511c5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4f03820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21511c5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4efe950 0x564ec4f0a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21511c5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec4f01900 con 0x564ec4ef8df0 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21511c5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec4e1af90 con 0x564ec4ef97e0 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21511c5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec4f01a80 con 0x564ec4efe950 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21511c5740 10 client.0 _ll_register_callbacks cb 0x564ec4e3cd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:32.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef8df0 0x564ec4f032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef8df0 0x564ec4f032e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48802/0 (socket says 172.21.15.60:48802) 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21477fe640 1 -- 172.21.15.60:0/1302440694 learned_addr learned my addr 172.21.15.60:0/1302440694 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f2147fff640 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4f03820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21477fe640 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4efe950 0x564ec4f0a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21477fe640 1 -- 172.21.15.60:0/1302440694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4efe950 msgr2=0x564ec4f0a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21477fe640 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4efe950 0x564ec4f0a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21477fe640 1 -- 172.21.15.60:0/1302440694 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 msgr2=0x564ec4f03820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21477fe640 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4f03820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21477fe640 1 -- 172.21.15.60:0/1302440694 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ec4b732d0 con 0x564ec4ef8df0 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21477fe640 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef8df0 0x564ec4f032e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f213c016fd0 tx=0x7f213c016e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b0a317608533a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21277fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21277fe640 1 -- 172.21.15.60:0/1302440694 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c01b040 con 0x564ec4ef8df0 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21277fe640 1 -- 172.21.15.60:0/1302440694 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f213c016600 con 0x564ec4ef8df0 2024-03-04T12:25:32.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.607+0000 7f21511c5740 1 -- 172.21.15.60:0/1302440694 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f213c00ee10 con 0x564ec4ef8df0 2024-03-04T12:25:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.608+0000 7f21277fe640 1 -- 172.21.15.60:0/1302440694 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f213c00c040 con 0x564ec4ef8df0 2024-03-04T12:25:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.608+0000 7f21511c5740 10 client.15519 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.608+0000 7f21511c5740 1 -- 172.21.15.60:0/1302440694 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564ec4c3d6b0 con 0x564ec4ef8df0 2024-03-04T12:25:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.608+0000 7f21277fe640 1 -- 172.21.15.60:0/1302440694 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f213c00ee10 con 0x564ec4ef8df0 2024-03-04T12:25:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.608+0000 7f21277fe640 1 -- 172.21.15.60:0/1302440694 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f213c015070 con 0x564ec4ef8df0 2024-03-04T12:25:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.608+0000 7f21511c5740 10 client.15519 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.608+0000 7f21511c5740 10 client.15519 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T12:25:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.608+0000 7f21511c5740 20 client.15519 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.608+0000 7f21511c5740 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564ec4f0fa70 0x564ec4f13ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.608+0000 7f21511c5740 4 client.15519 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:25:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.608+0000 7f21511c5740 1 -- 172.21.15.60:0/1302440694 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564ec4e6bd00 con 0x564ec4f0fa70 2024-03-04T12:25:32.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.608+0000 7f2147fff640 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564ec4f0fa70 0x564ec4f13ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.609+0000 7f2147fff640 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564ec4f0fa70 0x564ec4f13ef0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7bc95fb1d459536 server_cookie=8d51a6a8a1e4c2b5 in_seq=0 out_seq=0 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.609+0000 7f21277fe640 10 client.15519 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.609+0000 7f21277fe640 1 -- 172.21.15.60:0/1302440694 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564ec4e6bd00 con 0x564ec4f0fa70 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.609+0000 7f21277fe640 10 client.15519 handle_command_reply: tid=0 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.609+0000 7f21511c5740 1 client.15519 shutdown 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.610+0000 7f21511c5740 1 -- 172.21.15.60:0/1302440694 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef8df0 msgr2=0x564ec4f032e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.610+0000 7f21511c5740 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef8df0 0x564ec4f032e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f213c016fd0 tx=0x7f213c016e50 comp rx=0 tx=0).stop 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.610+0000 7f21511c5740 1 -- 172.21.15.60:0/1302440694 shutdown_connections 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.610+0000 7f21511c5740 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564ec4f0fa70 0x564ec4f13ef0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.610+0000 7f21511c5740 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4efe950 0x564ec4f0a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.610+0000 7f21511c5740 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4f03820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.610+0000 7f21277fe640 0 client.15519 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:32.623 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.610+0000 7f21511c5740 1 --2- 172.21.15.60:0/1302440694 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef8df0 0x564ec4f032e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.610+0000 7f21511c5740 1 -- 172.21.15.60:0/1302440694 >> 172.21.15.60:0/1302440694 conn(0x564ec4ef4360 msgr2=0x564ec4ef3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.610+0000 7f21511c5740 1 -- 172.21.15.60:0/1302440694 shutdown_connections 2024-03-04T12:25:32.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.610+0000 7f21511c5740 1 -- 172.21.15.60:0/1302440694 wait complete. 2024-03-04T12:25:32.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.610+0000 7f21511c5740 20 client.15519 trim_cache size 0 max 16384 2024-03-04T12:25:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.625+0000 7f21511c5740 1 Processor -- start 2024-03-04T12:25:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.625+0000 7f21511c5740 1 -- start start 2024-03-04T12:25:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f21511c5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 0x564ec4f00d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f21511c5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef97e0 0x564ec4f0a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f21511c5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4efe950 0x564ec4ef2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f21511c5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec4f0f290 con 0x564ec4ef97e0 2024-03-04T12:25:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f21511c5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec4f0f8f0 con 0x564ec4efe950 2024-03-04T12:25:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f21511c5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec4f0ff50 con 0x564ec4ef8df0 2024-03-04T12:25:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f2147fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef97e0 0x564ec4f0a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f2147fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef97e0 0x564ec4f0a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48818/0 (socket says 172.21.15.60:48818) 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f2147fff640 1 -- 172.21.15.60:0/4167311051 learned_addr learned my addr 172.21.15.60:0/4167311051 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f2147fff640 1 -- 172.21.15.60:0/4167311051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 msgr2=0x564ec4f00d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f2147fff640 1 --2- 172.21.15.60:0/4167311051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 0x564ec4f00d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f2147fff640 1 -- 172.21.15.60:0/4167311051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4efe950 msgr2=0x564ec4ef2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f2147fff640 1 --2- 172.21.15.60:0/4167311051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4efe950 0x564ec4ef2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f2147fff640 1 -- 172.21.15.60:0/4167311051 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ec4ef4db0 con 0x564ec4ef97e0 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f2147fff640 1 --2- 172.21.15.60:0/4167311051 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef97e0 0x564ec4f0a7f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x564ec4ef9be0 tx=0x7f2138014840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3456ce3d8cdf194 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f212dffb640 1 -- 172.21.15.60:0/4167311051 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21380142d0 con 0x564ec4ef97e0 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f212dffb640 1 -- 172.21.15.60:0/4167311051 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2138021030 con 0x564ec4ef97e0 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.626+0000 7f212dffb640 1 -- 172.21.15.60:0/4167311051 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138004a00 con 0x564ec4ef97e0 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.627+0000 7f21511c5740 1 -- 172.21.15.60:0/4167311051 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef97e0 msgr2=0x564ec4f0a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.627+0000 7f21511c5740 1 --2- 172.21.15.60:0/4167311051 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef97e0 0x564ec4f0a7f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x564ec4ef9be0 tx=0x7f2138014840 comp rx=0 tx=0).stop 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.627+0000 7f21511c5740 1 -- 172.21.15.60:0/4167311051 shutdown_connections 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.627+0000 7f21511c5740 1 --2- 172.21.15.60:0/4167311051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4efe950 0x564ec4ef2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.627+0000 7f21511c5740 1 --2- 172.21.15.60:0/4167311051 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4ef97e0 0x564ec4f0a7f0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.627+0000 7f21511c5740 1 --2- 172.21.15.60:0/4167311051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 0x564ec4f00d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.627+0000 7f21511c5740 1 -- 172.21.15.60:0/4167311051 >> 172.21.15.60:0/4167311051 conn(0x564ec4ef4360 msgr2=0x564ec4f187a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.627+0000 7f21511c5740 1 -- 172.21.15.60:0/4167311051 shutdown_connections 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.627+0000 7f21511c5740 1 -- 172.21.15.60:0/4167311051 wait complete. 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.627+0000 7f21511c5740 1 -- start start 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.627+0000 7f21511c5740 1 Processor -- start 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21511c5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 0x564ec4f07db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21511c5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4f082f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21511c5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 0x564ec4f092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21511c5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec4f10c10 con 0x564ec4efe950 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21511c5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec4f105b0 con 0x564ec4ef97e0 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21511c5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec4f11270 con 0x564ec4ef8df0 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21511c5740 10 client.0 _ll_register_callbacks cb 0x7f213c00c2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 0x564ec4f092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 0x564ec4f092b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48826/0 (socket says 172.21.15.60:48826) 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21477fe640 1 -- 172.21.15.60:0/468224072 learned_addr learned my addr 172.21.15.60:0/468224072 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21477fe640 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 0x564ec4f07db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f2147fff640 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4f082f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21477fe640 1 -- 172.21.15.60:0/468224072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 msgr2=0x564ec4f07db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21477fe640 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 0x564ec4f07db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21477fe640 1 -- 172.21.15.60:0/468224072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 msgr2=0x564ec4f082f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21477fe640 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4f082f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21477fe640 1 -- 172.21.15.60:0/468224072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ec4f098a0 con 0x564ec4efe950 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f2147fff640 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4f082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21477fe640 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 0x564ec4f092b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f213c016fd0 tx=0x7f213c015300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99daddbff8bfc2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21277fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21277fe640 1 -- 172.21.15.60:0/468224072 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c03a060 con 0x564ec4efe950 2024-03-04T12:25:32.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21277fe640 1 -- 172.21.15.60:0/468224072 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f213c00c390 con 0x564ec4efe950 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21511c5740 1 -- 172.21.15.60:0/468224072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2138021030 con 0x564ec4efe950 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.628+0000 7f21277fe640 1 -- 172.21.15.60:0/468224072 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f213c01b040 con 0x564ec4efe950 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.629+0000 7f21511c5740 10 client.15525 fetch_fsmap learned FSMap version 28 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.629+0000 7f21511c5740 1 -- 172.21.15.60:0/468224072 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564ec4f0aba0 con 0x564ec4efe950 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.629+0000 7f21277fe640 1 -- 172.21.15.60:0/468224072 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2138021030 con 0x564ec4efe950 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.629+0000 7f21277fe640 1 -- 172.21.15.60:0/468224072 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 28) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f213c034040 con 0x564ec4efe950 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.629+0000 7f21511c5740 10 client.15525 fetch_fsmap finished waiting for FSMap version 28 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.629+0000 7f21511c5740 10 client.15525 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.629+0000 7f21511c5740 20 client.15525 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.629+0000 7f21511c5740 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564ec4f6bfd0 0x564ec4f67c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.629+0000 7f21511c5740 4 client.15525 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.629+0000 7f21511c5740 1 -- 172.21.15.60:0/468224072 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564ec4e66050 con 0x564ec4f6bfd0 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.629+0000 7f2147fff640 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564ec4f6bfd0 0x564ec4f67c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:32.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21277fe640 10 client.15525 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f2147fff640 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564ec4f6bfd0 0x564ec4f67c20 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dc1db5f8d4c4c66 server_cookie=f5edde57cf8d1bd6 in_seq=0 out_seq=0 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21277fe640 1 -- 172.21.15.60:0/468224072 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x564ec4e66050 con 0x564ec4f6bfd0 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21277fe640 10 client.15525 handle_command_reply: tid=0 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21511c5740 1 client.15525 shutdown 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21511c5740 1 -- 172.21.15.60:0/468224072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 msgr2=0x564ec4f092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21511c5740 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 0x564ec4f092b0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f213c016fd0 tx=0x7f213c015300 comp rx=0 tx=0).stop 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21511c5740 1 -- 172.21.15.60:0/468224072 shutdown_connections 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21511c5740 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x564ec4f6bfd0 0x564ec4f67c20 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21511c5740 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564ec4efe950 0x564ec4f092b0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21511c5740 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ec4ef97e0 0x564ec4f082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21277fe640 0 client.15525 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21511c5740 1 --2- 172.21.15.60:0/468224072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564ec4ef8df0 0x564ec4f07db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21511c5740 1 -- 172.21.15.60:0/468224072 >> 172.21.15.60:0/468224072 conn(0x564ec4ef4360 msgr2=0x564ec4ef75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21511c5740 1 -- 172.21.15.60:0/468224072 shutdown_connections 2024-03-04T12:25:32.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21511c5740 1 -- 172.21.15.60:0/468224072 wait complete. 2024-03-04T12:25:32.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.630+0000 7f21511c5740 20 client.15525 trim_cache size 0 max 16384 2024-03-04T12:25:32.644 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:25:32.644 INFO:teuthology.orchestra.run.smithi060.stdout: "cluster_fsid": "a1c49686-da20-11ee-95c6-87774f69a715", 2024-03-04T12:25:32.644 INFO:teuthology.orchestra.run.smithi060.stdout: "whoami": 0, 2024-03-04T12:25:32.644 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 24524, 2024-03-04T12:25:32.644 INFO:teuthology.orchestra.run.smithi060.stdout: "want_state": "up:active", 2024-03-04T12:25:32.644 INFO:teuthology.orchestra.run.smithi060.stdout: "state": "up:active", 2024-03-04T12:25:32.644 INFO:teuthology.orchestra.run.smithi060.stdout: "fs_name": "cephfs", 2024-03-04T12:25:32.644 INFO:teuthology.orchestra.run.smithi060.stdout: "rank_uptime": 139.72716598100001, 2024-03-04T12:25:32.644 INFO:teuthology.orchestra.run.smithi060.stdout: "mdsmap_epoch": 28, 2024-03-04T12:25:32.645 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch": 93, 2024-03-04T12:25:32.645 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_epoch_barrier": 89, 2024-03-04T12:25:32.645 INFO:teuthology.orchestra.run.smithi060.stdout: "uptime": 146.02851140999999 2024-03-04T12:25:32.645 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:25:32.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.631+0000 7f214e5ec640 1 -- 172.21.15.60:0/1214030421 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21300a1d80 msgr2=0x7f21300a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.631+0000 7f214e5ec640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21300a1d80 0x7f21300a4240 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f21400c5400 tx=0x7f21400902e0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.631+0000 7f214e5ec640 1 -- 172.21.15.60:0/1214030421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2148105120 msgr2=0x7f21481a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.631+0000 7f214e5ec640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2148105120 0x7f21481a0e00 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f214002c700 tx=0x7f2140032f20 comp rx=0 tx=0).stop 2024-03-04T12:25:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.631+0000 7f214e5ec640 1 -- 172.21.15.60:0/1214030421 shutdown_connections 2024-03-04T12:25:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.631+0000 7f214e5ec640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f21300a1d80 0x7f21300a4240 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.631+0000 7f214e5ec640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2148105120 0x7f21481a0e00 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.631+0000 7f214e5ec640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2148104760 0x7f2148199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.631+0000 7f214e5ec640 1 --2- 172.21.15.60:0/1214030421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2148103570 0x7f2148199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.631+0000 7f214e5ec640 1 -- 172.21.15.60:0/1214030421 >> 172.21.15.60:0/1214030421 conn(0x7f21480feae0 msgr2=0x7f2148107a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.631+0000 7f214e5ec640 1 -- 172.21.15.60:0/1214030421 shutdown_connections 2024-03-04T12:25:32.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:32.631+0000 7f214e5ec640 1 -- 172.21.15.60:0/1214030421 wait complete. 2024-03-04T12:25:32.649 INFO:teuthology.run_tasks:Running task kclient... 2024-03-04T12:25:32.662 INFO:tasks.kclient:Mounting kernel clients... 2024-03-04T12:25:32.662 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-04T12:25:32.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:32.662 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-03-04T12:25:32.677 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-04T12:25:32.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:32.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-04T12:25:32.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:32.745 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2024-03-04T12:25:32.853 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-04T12:25:32.853 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-04T12:25:32.854 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T12:25:32.854 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph fs ls 2024-03-04T12:25:33.104 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:25:34.290 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:33 smithi060 ceph-mon[32745]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:25:34.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:33 smithi086 ceph-mon[36851]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:25:34.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:33 smithi063 ceph-mon[36717]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:25:34.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.865+0000 7fca660f5700 1 -- 172.21.15.60:0/1738045978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca60069ad0 msgr2=0x7fca60069f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:34.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.865+0000 7fca660f5700 1 --2- 172.21.15.60:0/1738045978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca60069ad0 0x7fca60069f30 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fca540099c0 tx=0x7fca54009cd0 comp rx=0 tx=0).stop 2024-03-04T12:25:34.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.865+0000 7fca660f5700 1 -- 172.21.15.60:0/1738045978 shutdown_connections 2024-03-04T12:25:34.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.865+0000 7fca660f5700 1 --2- 172.21.15.60:0/1738045978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca6006a470 0x7fca6010ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:34.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.865+0000 7fca660f5700 1 --2- 172.21.15.60:0/1738045978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca60107e70 0x7fca60108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:34.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.865+0000 7fca660f5700 1 --2- 172.21.15.60:0/1738045978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca60069ad0 0x7fca60069f30 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:34.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.865+0000 7fca660f5700 1 -- 172.21.15.60:0/1738045978 >> 172.21.15.60:0/1738045978 conn(0x7fca60077c50 msgr2=0x7fca60078060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:34.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.865+0000 7fca660f5700 1 -- 172.21.15.60:0/1738045978 shutdown_connections 2024-03-04T12:25:34.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.865+0000 7fca660f5700 1 -- 172.21.15.60:0/1738045978 wait complete. 2024-03-04T12:25:34.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.866+0000 7fca660f5700 1 Processor -- start 2024-03-04T12:25:34.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.866+0000 7fca660f5700 1 -- start start 2024-03-04T12:25:34.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.866+0000 7fca660f5700 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca60069ad0 0x7fca60100df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:34.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.866+0000 7fca660f5700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca6006a470 0x7fca60101330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:34.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca660f5700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca60107e70 0x7fca60101870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca660f5700 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca601120d0 con 0x7fca6006a470 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca660f5700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca60111f50 con 0x7fca60107e70 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca660f5700 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca60112250 con 0x7fca60069ad0 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca5effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca6006a470 0x7fca60101330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca5effd700 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca6006a470 0x7fca60101330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48856/0 (socket says 172.21.15.60:48856) 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca5effd700 1 -- 172.21.15.60:0/1423437425 learned_addr learned my addr 172.21.15.60:0/1423437425 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca5f7fe700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca60069ad0 0x7fca60100df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca5ffff700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca60107e70 0x7fca60101870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca5effd700 1 -- 172.21.15.60:0/1423437425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca60069ad0 msgr2=0x7fca60100df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca5effd700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca60069ad0 0x7fca60100df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca5effd700 1 -- 172.21.15.60:0/1423437425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca60107e70 msgr2=0x7fca60101870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca5effd700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca60107e70 0x7fca60101870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca5effd700 1 -- 172.21.15.60:0/1423437425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca601a6b00 con 0x7fca6006a470 2024-03-04T12:25:34.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca5effd700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca6006a470 0x7fca60101330 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fca54006010 tx=0x7fca54009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:34.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.867+0000 7fca5cff9700 1 -- 172.21.15.60:0/1423437425 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54027070 con 0x7fca6006a470 2024-03-04T12:25:34.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.868+0000 7fca660f5700 1 -- 172.21.15.60:0/1423437425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca601a6d90 con 0x7fca6006a470 2024-03-04T12:25:34.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.868+0000 7fca5cff9700 1 -- 172.21.15.60:0/1423437425 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca54003780 con 0x7fca6006a470 2024-03-04T12:25:34.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.868+0000 7fca5cff9700 1 -- 172.21.15.60:0/1423437425 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54022da0 con 0x7fca6006a470 2024-03-04T12:25:34.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.868+0000 7fca660f5700 1 -- 172.21.15.60:0/1423437425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca601a71e0 con 0x7fca6006a470 2024-03-04T12:25:34.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.869+0000 7fca660f5700 1 -- 172.21.15.60:0/1423437425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca60068ae0 con 0x7fca6006a470 2024-03-04T12:25:34.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.870+0000 7fca5cff9700 1 -- 172.21.15.60:0/1423437425 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7fca5402a070 con 0x7fca6006a470 2024-03-04T12:25:34.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.870+0000 7fca5cff9700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fca480a1e10 0x7fca480a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:34.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.870+0000 7fca5f7fe700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fca480a1e10 0x7fca480a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:34.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.870+0000 7fca5cff9700 1 -- 172.21.15.60:0/1423437425 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fca540cef40 con 0x7fca6006a470 2024-03-04T12:25:34.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.871+0000 7fca5f7fe700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fca480a1e10 0x7fca480a42d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fca500045b0 tx=0x7fca5000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:34.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:34.875+0000 7fca5cff9700 1 -- 172.21.15.60:0/1423437425 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fca5405dd20 con 0x7fca6006a470 2024-03-04T12:25:35.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.126+0000 7fca660f5700 1 -- 172.21.15.60:0/1423437425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fca6002d870 con 0x7fca6006a470 2024-03-04T12:25:35.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.127+0000 7fca5cff9700 1 -- 172.21.15.60:0/1423437425 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+88 (secure 0 0 0) 0x7fca54013070 con 0x7fca6006a470 2024-03-04T12:25:35.134 INFO:teuthology.orchestra.run.smithi060.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-04T12:25:35.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.128+0000 7fca660f5700 1 -- 172.21.15.60:0/1423437425 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fca480a1e10 msgr2=0x7fca480a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:35.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.128+0000 7fca660f5700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fca480a1e10 0x7fca480a42d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fca500045b0 tx=0x7fca5000b3c0 comp rx=0 tx=0).stop 2024-03-04T12:25:35.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.129+0000 7fca660f5700 1 -- 172.21.15.60:0/1423437425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca6006a470 msgr2=0x7fca60101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:35.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.129+0000 7fca660f5700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca6006a470 0x7fca60101330 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fca54006010 tx=0x7fca54009f80 comp rx=0 tx=0).stop 2024-03-04T12:25:35.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.129+0000 7fca660f5700 1 -- 172.21.15.60:0/1423437425 shutdown_connections 2024-03-04T12:25:35.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.129+0000 7fca660f5700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca60107e70 0x7fca60101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:35.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.129+0000 7fca660f5700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca60069ad0 0x7fca60100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:35.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.129+0000 7fca660f5700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fca480a1e10 0x7fca480a42d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:35.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.129+0000 7fca660f5700 1 --2- 172.21.15.60:0/1423437425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fca6006a470 0x7fca60101330 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:35.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.129+0000 7fca660f5700 1 -- 172.21.15.60:0/1423437425 >> 172.21.15.60:0/1423437425 conn(0x7fca60077c50 msgr2=0x7fca600ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:35.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.129+0000 7fca660f5700 1 -- 172.21.15.60:0/1423437425 shutdown_connections 2024-03-04T12:25:35.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:35.129+0000 7fca660f5700 1 -- 172.21.15.60:0/1423437425 wait complete. 2024-03-04T12:25:35.782 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-04T12:25:35.783 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-04T12:25:35.783 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi060.front.sepia.ceph.com 2024-03-04T12:25:35.783 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-04T12:25:35.783 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-04T12:25:35.783 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-04T12:25:35.783 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-04T12:25:35.783 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-04T12:25:35.783 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-04T12:25:35.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:35.783 DEBUG:teuthology.orchestra.run.smithi060:> ip addr 2024-03-04T12:25:35.801 INFO:teuthology.orchestra.run.smithi060.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-04T12:25:35.801 INFO:teuthology.orchestra.run.smithi060.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-04T12:25:35.801 INFO:teuthology.orchestra.run.smithi060.stdout: inet 127.0.0.1/8 scope host lo 2024-03-04T12:25:35.801 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-03-04T12:25:35.801 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 ::1/128 scope host 2024-03-04T12:25:35.801 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-03-04T12:25:35.801 INFO:teuthology.orchestra.run.smithi060.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T12:25:35.801 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cc brd ff:ff:ff:ff:ff:ff 2024-03-04T12:25:35.801 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f0 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:6b:fe:cd brd ff:ff:ff:ff:ff:ff 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp7s0f1 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:60 brd ff:ff:ff:ff:ff:ff 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f0 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout: inet 172.21.15.60/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft 42044sec preferred_lft 42044sec 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout: inet6 fe80::ec4:7aff:febd:1560/64 scope link noprefixroute 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout: valid_lft forever preferred_lft forever 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout: link/ether 0c:c4:7a:bd:15:61 brd ff:ff:ff:ff:ff:ff 2024-03-04T12:25:35.802 INFO:teuthology.orchestra.run.smithi060.stdout: altname enp1s0f1 2024-03-04T12:25:35.803 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-04T12:25:35.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:35.803 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2024-03-04T12:25:35.803 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-03-04T12:25:35.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add name ceph-brx type bridge 2024-03-04T12:25:35.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip addr flush dev ceph-brx 2024-03-04T12:25:35.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set ceph-brx up 2024-03-04T12:25:35.803 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-04T12:25:35.803 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-03-04T12:25:35.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:35.918 DEBUG:teuthology.orchestra.run.smithi060:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-04T12:25:35.945 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-03-04T12:25:35.947 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:35.947 DEBUG:teuthology.orchestra.run.smithi060:> route 2024-03-04T12:25:36.003 INFO:teuthology.orchestra.run.smithi060.stdout:Kernel IP routing table 2024-03-04T12:25:36.003 INFO:teuthology.orchestra.run.smithi060.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-04T12:25:36.003 INFO:teuthology.orchestra.run.smithi060.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-04T12:25:36.003 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-04T12:25:36.003 INFO:teuthology.orchestra.run.smithi060.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-04T12:25:36.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:36.003 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2024-03-04T12:25:36.004 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-03-04T12:25:36.004 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-04T12:25:36.004 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-04T12:25:36.004 DEBUG:teuthology.orchestra.run.smithi060:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-04T12:25:36.004 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-03-04T12:25:36.068 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:35 smithi060 ceph-mon[32745]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:25:36.068 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:35 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1423437425' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T12:25:36.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:36.100 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-03-04T12:25:36.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:36.155 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list-id 2024-03-04T12:25:36.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:36.210 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2024-03-04T12:25:36.210 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-03-04T12:25:36.210 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T12:25:36.210 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-04T12:25:36.210 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-03-04T12:25:36.292 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-04T12:25:36.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:36.293 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2024-03-04T12:25:36.293 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-03-04T12:25:36.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-04T12:25:36.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-04T12:25:36.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-04T12:25:36.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-04T12:25:36.293 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-04T12:25:36.293 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-03-04T12:25:36.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:35 smithi086 ceph-mon[36851]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:25:36.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:35 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1423437425' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T12:25:36.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:35 smithi063 ceph-mon[36717]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-04T12:25:36.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:35 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1423437425' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-04T12:25:36.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:36.420 DEBUG:teuthology.orchestra.run.smithi060:> (cd / && exec bash -c ' 2024-03-04T12:25:36.420 DEBUG:teuthology.orchestra.run.smithi060:> set -e 2024-03-04T12:25:36.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set brx.0 up 2024-03-04T12:25:36.420 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link set dev brx.0 master ceph-brx 2024-03-04T12:25:36.420 DEBUG:teuthology.orchestra.run.smithi060:> ') 2024-03-04T12:25:36.461 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T12:25:36.461 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-04T12:25:36.517 INFO:teuthology.orchestra.run.smithi060.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-04T12:25:36.517 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-04T12:25:36.517 DEBUG:teuthology.orchestra.run.smithi060:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-04T12:25:36.571 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-04T12:25:36.571 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-04T12:25:36.572 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1 2024-03-04T12:25:36.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:36.572 DEBUG:teuthology.orchestra.run.smithi060:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-03-04T12:25:36.656 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-03-04T12:25:36.656 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync". 2024-03-04T12:25:36.781 INFO:teuthology.orchestra.run.smithi060.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-03-04T12:25:36.782 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync". 2024-03-04T12:25:36.782 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: resolved to: "172.21.15.60:3300,172.21.15.63:3300,172.21.15.86:3300" 2024-03-04T12:25:36.782 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: trying mount with new device syntax: 0@a1c49686-da20-11ee-95c6-87774f69a715.cephfs=/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1 2024-03-04T12:25:36.782 INFO:teuthology.orchestra.run.smithi060.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,name=0,key=0,mon_addr=172.21.15.60:3300/172.21.15.63:3300/172.21.15.86:3300" will pass to kernel 2024-03-04T12:25:36.782 INFO:teuthology.orchestra.run.smithi060.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-04T12:25:36.783 INFO:teuthology.orchestra.run.smithi060.stdout: You just mounted a file system that supports labels which does not 2024-03-04T12:25:36.783 INFO:teuthology.orchestra.run.smithi060.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-04T12:25:36.783 INFO:teuthology.orchestra.run.smithi060.stdout: applications will generate AVC messages and not be allowed access to 2024-03-04T12:25:36.783 INFO:teuthology.orchestra.run.smithi060.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-04T12:25:36.784 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-04T12:25:36.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:36.784 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-04T12:25:36.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:36.858 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T12:25:36.915 INFO:teuthology.orchestra.run.smithi060.stdout:client25139 2024-03-04T12:25:36.915 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:25:36.916 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T12:25:36.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:36.971 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T12:25:37.028 INFO:teuthology.orchestra.run.smithi060.stdout:client25139 2024-03-04T12:25:37.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:37.029 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-04T12:25:37.086 INFO:teuthology.orchestra.run.smithi060.stdout:client25139 2024-03-04T12:25:37.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:37.086 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a1c49686-da20-11ee-95c6-87774f69a715.client25139/status') 2024-03-04T12:25:37.152 INFO:teuthology.orchestra.run.smithi060.stdout:instance: client.25139 (3)172.21.15.60:0/2918082118 2024-03-04T12:25:37.153 INFO:teuthology.orchestra.run.smithi060.stdout:blocklisted: false 2024-03-04T12:25:37.153 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records in 2024-03-04T12:25:37.153 INFO:teuthology.orchestra.run.smithi060.stderr:0+1 records out 2024-03-04T12:25:37.153 INFO:teuthology.orchestra.run.smithi060.stderr:71 bytes copied, 2.0817e-05 s, 3.4 MB/s 2024-03-04T12:25:37.154 INFO:teuthology.run_tasks:Running task exec... 2024-03-04T12:25:37.164 INFO:teuthology.task.exec:Executing custom commands... 2024-03-04T12:25:37.165 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi060.front.sepia.ceph.com 2024-03-04T12:25:37.165 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-04T12:25:37.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.290+0000 7f41f9be1640 1 -- 172.21.15.60:0/1735342627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f4102570 msgr2=0x7f41f41029f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:37.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.290+0000 7f41f9be1640 1 --2- 172.21.15.60:0/1735342627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f4102570 0x7f41f41029f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009900 tx=0x7f41e802bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:37.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.290+0000 7f41f9be1640 1 -- 172.21.15.60:0/1735342627 shutdown_connections 2024-03-04T12:25:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.290+0000 7f41f9be1640 1 --2- 172.21.15.60:0/1735342627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f4102f30 0x7f41f41097c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.290+0000 7f41f9be1640 1 --2- 172.21.15.60:0/1735342627 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f4102570 0x7f41f41029f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.290+0000 7f41f9be1640 1 --2- 172.21.15.60:0/1735342627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41f4101380 0x7f41f41017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.290+0000 7f41f9be1640 1 -- 172.21.15.60:0/1735342627 >> 172.21.15.60:0/1735342627 conn(0x7f41f40fc8f0 msgr2=0x7f41f40fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.290+0000 7f41f9be1640 1 -- 172.21.15.60:0/1735342627 shutdown_connections 2024-03-04T12:25:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.290+0000 7f41f9be1640 1 -- 172.21.15.60:0/1735342627 wait complete. 2024-03-04T12:25:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.291+0000 7f41f9be1640 1 Processor -- start 2024-03-04T12:25:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.291+0000 7f41f9be1640 1 -- start start 2024-03-04T12:25:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.291+0000 7f41f9be1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f4101380 0x7f41f4197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.291+0000 7f41f9be1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41f4102570 0x7f41f4197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:37.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.291+0000 7f41f9be1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f4102f30 0x7f41f419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.291+0000 7f41f9be1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f410d010 con 0x7f41f4102f30 2024-03-04T12:25:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.291+0000 7f41f9be1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f410ce90 con 0x7f41f4102570 2024-03-04T12:25:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.291+0000 7f41f9be1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f410d190 con 0x7f41f4101380 2024-03-04T12:25:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.291+0000 7f41f3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f4102f30 0x7f41f419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.291+0000 7f41f3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f4102f30 0x7f41f419ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55612/0 (socket says 172.21.15.60:55612) 2024-03-04T12:25:37.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.291+0000 7f41f3fff640 1 -- 172.21.15.60:0/2759155591 learned_addr learned my addr 172.21.15.60:0/2759155591 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f2ffd640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41f4102570 0x7f41f4197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f37fe640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f4101380 0x7f41f4197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f3fff640 1 -- 172.21.15.60:0/2759155591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f4101380 msgr2=0x7f41f4197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f3fff640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f4101380 0x7f41f4197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f3fff640 1 -- 172.21.15.60:0/2759155591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41f4102570 msgr2=0x7f41f4197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f3fff640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41f4102570 0x7f41f4197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f3fff640 1 -- 172.21.15.60:0/2759155591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f419f2b0 con 0x7f41f4102f30 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f37fe640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f4101380 0x7f41f4197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f3fff640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f4102f30 0x7f41f419ec40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00eaa0 tx=0x7f41ec00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f0ff9640 1 -- 172.21.15.60:0/2759155591 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ec00cdb0 con 0x7f41f4102f30 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f9be1640 1 -- 172.21.15.60:0/2759155591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f419f5a0 con 0x7f41f4102f30 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f0ff9640 1 -- 172.21.15.60:0/2759155591 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41ec005020 con 0x7f41f4102f30 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f0ff9640 1 -- 172.21.15.60:0/2759155591 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41ec0116d0 con 0x7f41f4102f30 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.292+0000 7f41f9be1640 1 -- 172.21.15.60:0/2759155591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f419fac0 con 0x7f41f4102f30 2024-03-04T12:25:37.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.293+0000 7f41f9be1640 1 -- 172.21.15.60:0/2759155591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f41086a0 con 0x7f41f4102f30 2024-03-04T12:25:37.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.294+0000 7f41f0ff9640 1 -- 172.21.15.60:0/2759155591 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f41ec0054d0 con 0x7f41f4102f30 2024-03-04T12:25:37.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.295+0000 7f41f0ff9640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f41dc0a1d70 0x7f41dc0a4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:37.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.295+0000 7f41f37fe640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f41dc0a1d70 0x7f41dc0a4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:37.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.295+0000 7f41f0ff9640 1 -- 172.21.15.60:0/2759155591 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f41ec0ce370 con 0x7f41f4102f30 2024-03-04T12:25:37.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.295+0000 7f41f37fe640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f41dc0a1d70 0x7f41dc0a4230 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f41f41023d0 tx=0x7f41e4005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:37.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.299+0000 7f41f0ff9640 1 -- 172.21.15.60:0/2759155591 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f41ec016030 con 0x7f41f4102f30 2024-03-04T12:25:37.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.522+0000 7f41f9be1640 1 -- 172.21.15.60:0/2759155591 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7f41f4061f30 con 0x7f41f4102f30 2024-03-04T12:25:37.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.990+0000 7f41f0ff9640 1 -- 172.21.15.60:0/2759155591 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) v1 ==== 118+0+0 (secure 0 0 0) 0x7f41ec09b4b0 con 0x7f41f4102f30 2024-03-04T12:25:38.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.993+0000 7f41f9be1640 1 -- 172.21.15.60:0/2759155591 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f41dc0a1d70 msgr2=0x7f41dc0a4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.993+0000 7f41f9be1640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f41dc0a1d70 0x7f41dc0a4230 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f41f41023d0 tx=0x7f41e4005d30 comp rx=0 tx=0).stop 2024-03-04T12:25:38.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.993+0000 7f41f9be1640 1 -- 172.21.15.60:0/2759155591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f4102f30 msgr2=0x7f41f419ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.993+0000 7f41f9be1640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f4102f30 0x7f41f419ec40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00eaa0 tx=0x7f41ec00ef60 comp rx=0 tx=0).stop 2024-03-04T12:25:38.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.993+0000 7f41f9be1640 1 -- 172.21.15.60:0/2759155591 shutdown_connections 2024-03-04T12:25:38.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.993+0000 7f41f9be1640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f41dc0a1d70 0x7f41dc0a4230 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.993+0000 7f41f9be1640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f41f4102f30 0x7f41f419ec40 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.993+0000 7f41f9be1640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41f4102570 0x7f41f4197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.993+0000 7f41f9be1640 1 --2- 172.21.15.60:0/2759155591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41f4101380 0x7f41f4197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.993+0000 7f41f9be1640 1 -- 172.21.15.60:0/2759155591 >> 172.21.15.60:0/2759155591 conn(0x7f41f40fc8f0 msgr2=0x7f41f4105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:38.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.994+0000 7f41f9be1640 1 -- 172.21.15.60:0/2759155591 shutdown_connections 2024-03-04T12:25:38.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:37.994+0000 7f41f9be1640 1 -- 172.21.15.60:0/2759155591 wait complete. 2024-03-04T12:25:38.016 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-04T12:25:38.130 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.121+0000 7f3dc89c4640 1 -- 172.21.15.60:0/2988237558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc41002a0 msgr2=0x7f3dc41006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.121+0000 7f3dc89c4640 1 --2- 172.21.15.60:0/2988237558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc41002a0 0x7f3dc41006c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009900 tx=0x7f3db402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.121+0000 7f3dc89c4640 1 -- 172.21.15.60:0/2988237558 shutdown_connections 2024-03-04T12:25:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.121+0000 7f3dc89c4640 1 --2- 172.21.15.60:0/2988237558 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dc4101e50 0x7f3dc41086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.121+0000 7f3dc89c4640 1 --2- 172.21.15.60:0/2988237558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc4101490 0x7f3dc4101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.131 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.121+0000 7f3dc89c4640 1 --2- 172.21.15.60:0/2988237558 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc41002a0 0x7f3dc41006c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.121+0000 7f3dc89c4640 1 -- 172.21.15.60:0/2988237558 >> 172.21.15.60:0/2988237558 conn(0x7f3dc40fb810 msgr2=0x7f3dc40fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.121+0000 7f3dc89c4640 1 -- 172.21.15.60:0/2988237558 shutdown_connections 2024-03-04T12:25:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.121+0000 7f3dc89c4640 1 -- 172.21.15.60:0/2988237558 wait complete. 2024-03-04T12:25:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc89c4640 1 Processor -- start 2024-03-04T12:25:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc89c4640 1 -- start start 2024-03-04T12:25:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc89c4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc41002a0 0x7f3dc4197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc89c4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dc4101490 0x7f3dc4197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc89c4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc4101e50 0x7f3dc419ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:38.132 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc89c4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc410bfc0 con 0x7f3dc41002a0 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc89c4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc410be40 con 0x7f3dc4101e50 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc89c4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc410c140 con 0x7f3dc4101490 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc41002a0 0x7f3dc4197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc259c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc41002a0 0x7f3dc4197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55628/0 (socket says 172.21.15.60:55628) 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc259c640 1 -- 172.21.15.60:0/3689815297 learned_addr learned my addr 172.21.15.60:0/3689815297 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc1d9b640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dc4101490 0x7f3dc4197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc2d9d640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc4101e50 0x7f3dc419ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc259c640 1 -- 172.21.15.60:0/3689815297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dc4101490 msgr2=0x7f3dc4197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc259c640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dc4101490 0x7f3dc4197c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc259c640 1 -- 172.21.15.60:0/3689815297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc4101e50 msgr2=0x7f3dc419ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc259c640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc4101e50 0x7f3dc419ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.122+0000 7f3dc259c640 1 -- 172.21.15.60:0/3689815297 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc419f340 con 0x7f3dc41002a0 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.123+0000 7f3dc259c640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc41002a0 0x7f3dc4197720 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3db4000c00 tx=0x7f3db4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:38.133 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.123+0000 7f3db37fe640 1 -- 172.21.15.60:0/3689815297 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db4043020 con 0x7f3dc41002a0 2024-03-04T12:25:38.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.123+0000 7f3db37fe640 1 -- 172.21.15.60:0/3689815297 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3db4004440 con 0x7f3dc41002a0 2024-03-04T12:25:38.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.123+0000 7f3dc89c4640 1 -- 172.21.15.60:0/3689815297 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc419f5d0 con 0x7f3dc41002a0 2024-03-04T12:25:38.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.123+0000 7f3db37fe640 1 -- 172.21.15.60:0/3689815297 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db403e450 con 0x7f3dc41002a0 2024-03-04T12:25:38.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.123+0000 7f3dc89c4640 1 -- 172.21.15.60:0/3689815297 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc419fb70 con 0x7f3dc41002a0 2024-03-04T12:25:38.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.124+0000 7f3dc89c4640 1 -- 172.21.15.60:0/3689815297 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc419fd70 con 0x7f3dc41002a0 2024-03-04T12:25:38.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.125+0000 7f3db37fe640 1 -- 172.21.15.60:0/3689815297 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f3db4046020 con 0x7f3dc41002a0 2024-03-04T12:25:38.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.125+0000 7f3db37fe640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f3dac0a1df0 0x7f3dac0a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:38.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.126+0000 7f3db37fe640 1 -- 172.21.15.60:0/3689815297 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f3db40f0930 con 0x7f3dc41002a0 2024-03-04T12:25:38.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.126+0000 7f3dc1d9b640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f3dac0a1df0 0x7f3dac0a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:38.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.126+0000 7f3dc1d9b640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f3dac0a1df0 0x7f3dac0a42b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4198c30 tx=0x7f3db800a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:38.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.129+0000 7f3db37fe640 1 -- 172.21.15.60:0/3689815297 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3db404d020 con 0x7f3dc41002a0 2024-03-04T12:25:38.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:37 smithi086 ceph-mon[36851]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:25:38.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:37 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2759155591' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-04T12:25:38.349 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.342+0000 7f3dc89c4640 1 -- 172.21.15.60:0/3689815297 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7f3dc40fd970 con 0x7f3dac0a1df0 2024-03-04T12:25:38.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:37 smithi063 ceph-mon[36717]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:25:38.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:37 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2759155591' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-04T12:25:38.354 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:37 smithi060 ceph-mon[32745]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-04T12:25:38.354 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:37 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2759155591' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-04T12:25:38.367 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.360+0000 7f3db37fe640 1 -- 172.21.15.60:0/3689815297 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7f3dc40fd970 con 0x7f3dac0a1df0 2024-03-04T12:25:38.367 INFO:teuthology.orchestra.run.smithi060.stdout:{} 2024-03-04T12:25:38.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.362+0000 7f3dc89c4640 1 -- 172.21.15.60:0/3689815297 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f3dac0a1df0 msgr2=0x7f3dac0a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.362+0000 7f3dc89c4640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f3dac0a1df0 0x7f3dac0a42b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4198c30 tx=0x7f3db800a4c0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.362+0000 7f3dc89c4640 1 -- 172.21.15.60:0/3689815297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc41002a0 msgr2=0x7f3dc4197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.362+0000 7f3dc89c4640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc41002a0 0x7f3dc4197720 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3db4000c00 tx=0x7f3db4004040 comp rx=0 tx=0).stop 2024-03-04T12:25:38.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.362+0000 7f3dc89c4640 1 -- 172.21.15.60:0/3689815297 shutdown_connections 2024-03-04T12:25:38.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.362+0000 7f3dc89c4640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f3dac0a1df0 0x7f3dac0a42b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.362+0000 7f3dc89c4640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc4101e50 0x7f3dc419ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.362+0000 7f3dc89c4640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dc4101490 0x7f3dc4197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.362+0000 7f3dc89c4640 1 --2- 172.21.15.60:0/3689815297 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3dc41002a0 0x7f3dc4197720 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.362+0000 7f3dc89c4640 1 -- 172.21.15.60:0/3689815297 >> 172.21.15.60:0/3689815297 conn(0x7f3dc40fb810 msgr2=0x7f3dc40fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:38.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.363+0000 7f3dc89c4640 1 -- 172.21.15.60:0/3689815297 shutdown_connections 2024-03-04T12:25:38.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.363+0000 7f3dc89c4640 1 -- 172.21.15.60:0/3689815297 wait complete. 2024-03-04T12:25:38.380 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-04T12:25:38.392 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-04T12:25:38.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:25:38.506 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.497+0000 7f6597a43640 1 -- 172.21.15.60:0/3106895747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6590100260 msgr2=0x7f6590100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.497+0000 7f6597a43640 1 --2- 172.21.15.60:0/3106895747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6590100260 0x7f6590100680 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f658c009900 tx=0x7f658c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.497+0000 7f6597a43640 1 -- 172.21.15.60:0/3106895747 shutdown_connections 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.497+0000 7f6597a43640 1 --2- 172.21.15.60:0/3106895747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6590101e10 0x7f65901086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.497+0000 7f6597a43640 1 --2- 172.21.15.60:0/3106895747 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6590101450 0x7f65901018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.497+0000 7f6597a43640 1 --2- 172.21.15.60:0/3106895747 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6590100260 0x7f6590100680 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.497+0000 7f6597a43640 1 -- 172.21.15.60:0/3106895747 >> 172.21.15.60:0/3106895747 conn(0x7f65900fb7f0 msgr2=0x7f65900fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.497+0000 7f6597a43640 1 -- 172.21.15.60:0/3106895747 shutdown_connections 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.497+0000 7f6597a43640 1 -- 172.21.15.60:0/3106895747 wait complete. 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f6597a43640 1 Processor -- start 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f6597a43640 1 -- start start 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f6597a43640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6590100260 0x7f659006a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f6597a43640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6590101450 0x7f659019c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f6597a43640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6590101e10 0x7f659019ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:38.507 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f6597a43640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659010bf40 con 0x7f6590100260 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f6597a43640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659010bdc0 con 0x7f6590101450 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f6597a43640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659010c0c0 con 0x7f6590101e10 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f65957df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6590100260 0x7f659006a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f65957df640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6590100260 0x7f659006a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55664/0 (socket says 172.21.15.60:55664) 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f65957df640 1 -- 172.21.15.60:0/772748225 learned_addr learned my addr 172.21.15.60:0/772748225 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f6594fde640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6590101450 0x7f659019c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f6595fe0640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6590101e10 0x7f659019ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f65957df640 1 -- 172.21.15.60:0/772748225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6590101e10 msgr2=0x7f659019ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f65957df640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6590101e10 0x7f659019ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f65957df640 1 -- 172.21.15.60:0/772748225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6590101450 msgr2=0x7f659019c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f65957df640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6590101450 0x7f659019c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f65957df640 1 -- 172.21.15.60:0/772748225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659019f310 con 0x7f6590100260 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.498+0000 7f65957df640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6590100260 0x7f659006a580 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f658c000c00 tx=0x7f658c02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:38.508 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.499+0000 7f65867fc640 1 -- 172.21.15.60:0/772748225 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c043020 con 0x7f6590100260 2024-03-04T12:25:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.499+0000 7f65867fc640 1 -- 172.21.15.60:0/772748225 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f658c0054e0 con 0x7f6590100260 2024-03-04T12:25:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.499+0000 7f65867fc640 1 -- 172.21.15.60:0/772748225 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c003ce0 con 0x7f6590100260 2024-03-04T12:25:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.499+0000 7f6597a43640 1 -- 172.21.15.60:0/772748225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659019f540 con 0x7f6590100260 2024-03-04T12:25:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.499+0000 7f6597a43640 1 -- 172.21.15.60:0/772748225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659019fab0 con 0x7f6590100260 2024-03-04T12:25:38.509 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.500+0000 7f6597a43640 1 -- 172.21.15.60:0/772748225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6590107580 con 0x7f6590100260 2024-03-04T12:25:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.501+0000 7f65867fc640 1 -- 172.21.15.60:0/772748225 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f658c046020 con 0x7f6590100260 2024-03-04T12:25:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.501+0000 7f65867fc640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f657c0a1e00 0x7f657c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.501+0000 7f6594fde640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f657c0a1e00 0x7f657c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.501+0000 7f65867fc640 1 -- 172.21.15.60:0/772748225 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f658c0f0250 con 0x7f6590100260 2024-03-04T12:25:38.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.502+0000 7f6594fde640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f657c0a1e00 0x7f657c0a42c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f659006a2f0 tx=0x7f658000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:38.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.506+0000 7f65867fc640 1 -- 172.21.15.60:0/772748225 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f658c04d020 con 0x7f6590100260 2024-03-04T12:25:38.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:38 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:38.346+0000 7f59f83c7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:38.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:38 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:38.346+0000 7f59f83c7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:38.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:38 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:38.346+0000 7f59f83c7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:38.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:38 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:38.346+0000 7f59f83c7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:38.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:38 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:38.346+0000 7f59f83c7700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:38.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.689+0000 7f6597a43640 1 -- 172.21.15.60:0/772748225 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f659002d0d0 con 0x7f6590100260 2024-03-04T12:25:38.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.691+0000 7f65867fc640 1 -- 172.21.15.60:0/772748225 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24562 (secure 0 0 0) 0x7f658c0bd2f0 con 0x7f6590100260 2024-03-04T12:25:38.698 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:38.699 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":93,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:25:06.929248+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:25:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.693+0000 7f6597a43640 1 -- 172.21.15.60:0/772748225 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f657c0a1e00 msgr2=0x7f657c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.693+0000 7f6597a43640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f657c0a1e00 0x7f657c0a42c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f659006a2f0 tx=0x7f658000b390 comp rx=0 tx=0).stop 2024-03-04T12:25:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.693+0000 7f6597a43640 1 -- 172.21.15.60:0/772748225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6590100260 msgr2=0x7f659006a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.693+0000 7f6597a43640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6590100260 0x7f659006a580 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f658c000c00 tx=0x7f658c02c6c0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.693+0000 7f6597a43640 1 -- 172.21.15.60:0/772748225 shutdown_connections 2024-03-04T12:25:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.693+0000 7f6597a43640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f657c0a1e00 0x7f657c0a42c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.693+0000 7f6597a43640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6590101e10 0x7f659019ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.693+0000 7f6597a43640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6590101450 0x7f659019c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.693+0000 7f6597a43640 1 --2- 172.21.15.60:0/772748225 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6590100260 0x7f659006a580 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.693+0000 7f6597a43640 1 -- 172.21.15.60:0/772748225 >> 172.21.15.60:0/772748225 conn(0x7f65900fb7f0 msgr2=0x7f6590104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.693+0000 7f6597a43640 1 -- 172.21.15.60:0/772748225 shutdown_connections 2024-03-04T12:25:38.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.693+0000 7f6597a43640 1 -- 172.21.15.60:0/772748225 wait complete. 2024-03-04T12:25:38.712 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T12:23:10.438928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T12:23:11.472998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T12:23:15.162500+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T12:25:38.713 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:25:38.820 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.811+0000 7ff109661640 1 -- 172.21.15.60:0/4219675802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff104100250 msgr2=0x7ff104100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.811+0000 7ff109661640 1 --2- 172.21.15.60:0/4219675802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff104100250 0x7ff104100670 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009900 tx=0x7ff0f402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.811+0000 7ff109661640 1 -- 172.21.15.60:0/4219675802 shutdown_connections 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.811+0000 7ff109661640 1 --2- 172.21.15.60:0/4219675802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff104101e00 0x7ff104108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.811+0000 7ff109661640 1 --2- 172.21.15.60:0/4219675802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff104101440 0x7ff1041018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.811+0000 7ff109661640 1 --2- 172.21.15.60:0/4219675802 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff104100250 0x7ff104100670 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.811+0000 7ff109661640 1 -- 172.21.15.60:0/4219675802 >> 172.21.15.60:0/4219675802 conn(0x7ff1040fb7c0 msgr2=0x7ff1040fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.811+0000 7ff109661640 1 -- 172.21.15.60:0/4219675802 shutdown_connections 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.811+0000 7ff109661640 1 -- 172.21.15.60:0/4219675802 wait complete. 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff109661640 1 Processor -- start 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff109661640 1 -- start start 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff109661640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff104100250 0x7ff1041998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff109661640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff104101440 0x7ff104199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff109661640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff104101e00 0x7ff1041a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:38.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff109661640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10410bf30 con 0x7ff104101e00 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff109661640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10410bdb0 con 0x7ff104100250 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff109661640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10410c0b0 con 0x7ff104101440 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff1037fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff104101e00 0x7ff1041a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff1037fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff104101e00 0x7ff1041a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55688/0 (socket says 172.21.15.60:55688) 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff102ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff104100250 0x7ff1041998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff1037fe640 1 -- 172.21.15.60:0/1325290955 learned_addr learned my addr 172.21.15.60:0/1325290955 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff0fbfff640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff104101440 0x7ff104199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff1037fe640 1 -- 172.21.15.60:0/1325290955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff104101440 msgr2=0x7ff104199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff1037fe640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff104101440 0x7ff104199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff1037fe640 1 -- 172.21.15.60:0/1325290955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff104100250 msgr2=0x7ff1041998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff1037fe640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff104100250 0x7ff1041998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff1037fe640 1 -- 172.21.15.60:0/1325290955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1041a14f0 con 0x7ff104101e00 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.812+0000 7ff1037fe640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff104101e00 0x7ff1041a0e80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00cad0 tx=0x7ff0fc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:38.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.813+0000 7ff100ff9640 1 -- 172.21.15.60:0/1325290955 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc00ae20 con 0x7ff104101e00 2024-03-04T12:25:38.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.813+0000 7ff100ff9640 1 -- 172.21.15.60:0/1325290955 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0fc005020 con 0x7ff104101e00 2024-03-04T12:25:38.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.813+0000 7ff109661640 1 -- 172.21.15.60:0/1325290955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1041a17e0 con 0x7ff104101e00 2024-03-04T12:25:38.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.813+0000 7ff100ff9640 1 -- 172.21.15.60:0/1325290955 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc00f6f0 con 0x7ff104101e00 2024-03-04T12:25:38.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.813+0000 7ff109661640 1 -- 172.21.15.60:0/1325290955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1041a1ce0 con 0x7ff104101e00 2024-03-04T12:25:38.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.814+0000 7ff109661640 1 -- 172.21.15.60:0/1325290955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1041a1fa0 con 0x7ff104101e00 2024-03-04T12:25:38.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.815+0000 7ff100ff9640 1 -- 172.21.15.60:0/1325290955 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7ff0fc0051d0 con 0x7ff104101e00 2024-03-04T12:25:38.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.815+0000 7ff100ff9640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff0f00a1d80 0x7ff0f00a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:38.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.816+0000 7ff102ffd640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff0f00a1d80 0x7ff0f00a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:38.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.816+0000 7ff100ff9640 1 -- 172.21.15.60:0/1325290955 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7ff0fc0ce100 con 0x7ff104101e00 2024-03-04T12:25:38.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.816+0000 7ff102ffd640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff0f00a1d80 0x7ff0f00a4240 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff0f402c170 tx=0x7ff0f402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:38.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:38.820+0000 7ff100ff9640 1 -- 172.21.15.60:0/1325290955 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff0fc014030 con 0x7ff104101e00 2024-03-04T12:25:39.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.004+0000 7ff109661640 1 -- 172.21.15.60:0/1325290955 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff1041a2170 con 0x7ff104101e00 2024-03-04T12:25:39.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.005+0000 7ff100ff9640 1 -- 172.21.15.60:0/1325290955 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff0fc09ab50 con 0x7ff104101e00 2024-03-04T12:25:39.012 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:25:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.007+0000 7ff109661640 1 -- 172.21.15.60:0/1325290955 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff0f00a1d80 msgr2=0x7ff0f00a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.007+0000 7ff109661640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff0f00a1d80 0x7ff0f00a4240 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff0f402c170 tx=0x7ff0f402df90 comp rx=0 tx=0).stop 2024-03-04T12:25:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.007+0000 7ff109661640 1 -- 172.21.15.60:0/1325290955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff104101e00 msgr2=0x7ff1041a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.007+0000 7ff109661640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff104101e00 0x7ff1041a0e80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00cad0 tx=0x7ff0fc00cf90 comp rx=0 tx=0).stop 2024-03-04T12:25:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.007+0000 7ff109661640 1 -- 172.21.15.60:0/1325290955 shutdown_connections 2024-03-04T12:25:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.007+0000 7ff109661640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7ff0f00a1d80 0x7ff0f00a4240 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.007+0000 7ff109661640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff104101e00 0x7ff1041a0e80 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.007+0000 7ff109661640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff104101440 0x7ff104199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.007+0000 7ff109661640 1 --2- 172.21.15.60:0/1325290955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff104100250 0x7ff1041998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.007+0000 7ff109661640 1 -- 172.21.15.60:0/1325290955 >> 172.21.15.60:0/1325290955 conn(0x7ff1040fb7c0 msgr2=0x7ff1040fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.007+0000 7ff109661640 1 -- 172.21.15.60:0/1325290955 shutdown_connections 2024-03-04T12:25:39.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.007+0000 7ff109661640 1 -- 172.21.15.60:0/1325290955 wait complete. 2024-03-04T12:25:39.025 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T12:25:39.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.124+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3343128570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b10101340 msgr2=0x7f4b10101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.124+0000 7f4b17f9d640 1 --2- 172.21.15.60:0/3343128570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b10101340 0x7f4b10101760 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c009900 tx=0x7f4b0c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.124+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3343128570 shutdown_connections 2024-03-04T12:25:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.124+0000 7f4b17f9d640 1 --2- 172.21.15.60:0/3343128570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b10102ef0 0x7f4b10109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.124+0000 7f4b17f9d640 1 --2- 172.21.15.60:0/3343128570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b10102530 0x7f4b101029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.124+0000 7f4b17f9d640 1 --2- 172.21.15.60:0/3343128570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b10101340 0x7f4b10101760 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.124+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3343128570 >> 172.21.15.60:0/3343128570 conn(0x7f4b100fc8b0 msgr2=0x7f4b100fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.124+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3343128570 shutdown_connections 2024-03-04T12:25:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.124+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3343128570 wait complete. 2024-03-04T12:25:39.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.124+0000 7f4b17f9d640 1 Processor -- start 2024-03-04T12:25:39.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.124+0000 7f4b17f9d640 1 -- start start 2024-03-04T12:25:39.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b17f9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b10101340 0x7f4b101976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:39.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b17f9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b10102530 0x7f4b10197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:39.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b17f9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b10102ef0 0x7f4b1019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:39.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b17f9d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1010cfa0 con 0x7f4b10101340 2024-03-04T12:25:39.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b17f9d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1010ce20 con 0x7f4b10102530 2024-03-04T12:25:39.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b17f9d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1010d120 con 0x7f4b10102ef0 2024-03-04T12:25:39.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b15d39640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b10101340 0x7f4b101976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:39.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b15d39640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b10101340 0x7f4b101976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55718/0 (socket says 172.21.15.60:55718) 2024-03-04T12:25:39.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b15d39640 1 -- 172.21.15.60:0/3193439864 learned_addr learned my addr 172.21.15.60:0/3193439864 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:39.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b15538640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b10102530 0x7f4b10197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:39.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b15d39640 1 -- 172.21.15.60:0/3193439864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b10102ef0 msgr2=0x7f4b1019ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b15d39640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b10102ef0 0x7f4b1019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b15d39640 1 -- 172.21.15.60:0/3193439864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b10102530 msgr2=0x7f4b10197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b15d39640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b10102530 0x7f4b10197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b15d39640 1 -- 172.21.15.60:0/3193439864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1019f2f0 con 0x7f4b10101340 2024-03-04T12:25:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b15538640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b10102530 0x7f4b10197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b15d39640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b10101340 0x7f4b101976d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c000c00 tx=0x7f4b0c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b06ffd640 1 -- 172.21.15.60:0/3193439864 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c041070 con 0x7f4b10101340 2024-03-04T12:25:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b06ffd640 1 -- 172.21.15.60:0/3193439864 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b0c0044d0 con 0x7f4b10101340 2024-03-04T12:25:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3193439864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1019f580 con 0x7f4b10101340 2024-03-04T12:25:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b06ffd640 1 -- 172.21.15.60:0/3193439864 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c034440 con 0x7f4b10101340 2024-03-04T12:25:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.125+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3193439864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1019fac0 con 0x7f4b10101340 2024-03-04T12:25:39.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.126+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3193439864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b10108780 con 0x7f4b10101340 2024-03-04T12:25:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.128+0000 7f4b06ffd640 1 -- 172.21.15.60:0/3193439864 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f4b0c044070 con 0x7f4b10101340 2024-03-04T12:25:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.128+0000 7f4b06ffd640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4afc0a1e00 0x7f4afc0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.128+0000 7f4b15538640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4afc0a1e00 0x7f4afc0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.128+0000 7f4b06ffd640 1 -- 172.21.15.60:0/3193439864 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f4b0c0efef0 con 0x7f4b10101340 2024-03-04T12:25:39.141 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.129+0000 7f4b15538640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4afc0a1e00 0x7f4afc0a42c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4b10198be0 tx=0x7f4b00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:39.142 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.132+0000 7f4b06ffd640 1 -- 172.21.15.60:0/3193439864 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4b0c0bcf90 con 0x7f4b10101340 2024-03-04T12:25:39.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.313+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3193439864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4b1002ceb0 con 0x7f4b10101340 2024-03-04T12:25:39.320 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.313+0000 7f4b06ffd640 1 -- 172.21.15.60:0/3193439864 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4b0c0bc890 con 0x7f4b10101340 2024-03-04T12:25:39.320 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:39.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.315+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3193439864 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4afc0a1e00 msgr2=0x7f4afc0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.315+0000 7f4b17f9d640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4afc0a1e00 0x7f4afc0a42c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4b10198be0 tx=0x7f4b00005f90 comp rx=0 tx=0).stop 2024-03-04T12:25:39.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.315+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3193439864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b10101340 msgr2=0x7f4b101976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.315+0000 7f4b17f9d640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b10101340 0x7f4b101976d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c000c00 tx=0x7f4b0c005d50 comp rx=0 tx=0).stop 2024-03-04T12:25:39.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.315+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3193439864 shutdown_connections 2024-03-04T12:25:39.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.315+0000 7f4b17f9d640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f4afc0a1e00 0x7f4afc0a42c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.315+0000 7f4b17f9d640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b10102ef0 0x7f4b1019ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.315+0000 7f4b17f9d640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b10102530 0x7f4b10197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.315+0000 7f4b17f9d640 1 --2- 172.21.15.60:0/3193439864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4b10101340 0x7f4b101976d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.315+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3193439864 >> 172.21.15.60:0/3193439864 conn(0x7f4b100fc8b0 msgr2=0x7f4b10105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:39.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.315+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3193439864 shutdown_connections 2024-03-04T12:25:39.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.315+0000 7f4b17f9d640 1 -- 172.21.15.60:0/3193439864 wait complete. 2024-03-04T12:25:39.332 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:39 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2759155591' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-04T12:25:39.332 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:39 smithi060 ceph-mon[32745]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:25:39.332 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:39 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/772748225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:39.333 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T12:25:39.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:39 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2759155591' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-04T12:25:39.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:39 smithi086 ceph-mon[36851]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:25:39.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:39 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/772748225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:39.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:39 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2759155591' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-04T12:25:39.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:39 smithi063 ceph-mon[36717]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:25:39.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:39 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/772748225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:39.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.431+0000 7f190777e640 1 -- 172.21.15.60:0/2564382164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1900101e00 msgr2=0x7f1900108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.431+0000 7f190777e640 1 --2- 172.21.15.60:0/2564382164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1900101e00 0x7f1900108690 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009ca0 tx=0x7f18fc02c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.431+0000 7f190777e640 1 -- 172.21.15.60:0/2564382164 shutdown_connections 2024-03-04T12:25:39.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.431+0000 7f190777e640 1 --2- 172.21.15.60:0/2564382164 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1900101e00 0x7f1900108690 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.431+0000 7f190777e640 1 --2- 172.21.15.60:0/2564382164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1900101440 0x7f19001018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.441 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.431+0000 7f190777e640 1 --2- 172.21.15.60:0/2564382164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1900100250 0x7f1900100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.431+0000 7f190777e640 1 -- 172.21.15.60:0/2564382164 >> 172.21.15.60:0/2564382164 conn(0x7f19000fb7c0 msgr2=0x7f19000fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.431+0000 7f190777e640 1 -- 172.21.15.60:0/2564382164 shutdown_connections 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.431+0000 7f190777e640 1 -- 172.21.15.60:0/2564382164 wait complete. 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f190777e640 1 Processor -- start 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f190777e640 1 -- start start 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f190777e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1900100250 0x7f1900197540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f190777e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1900101440 0x7f1900197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f190777e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1900101e00 0x7f190019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f190777e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190010beb0 con 0x7f1900100250 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f190777e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190010bd30 con 0x7f1900101e00 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f190777e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190010c030 con 0x7f1900101440 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f190551a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1900100250 0x7f1900197540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f190551a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1900100250 0x7f1900197540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55736/0 (socket says 172.21.15.60:55736) 2024-03-04T12:25:39.442 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f190551a640 1 -- 172.21.15.60:0/4281224632 learned_addr learned my addr 172.21.15.60:0/4281224632 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f1904d19640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1900101440 0x7f1900197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.432+0000 7f1905d1b640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1900101e00 0x7f190019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.433+0000 7f190551a640 1 -- 172.21.15.60:0/4281224632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1900101440 msgr2=0x7f1900197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.433+0000 7f190551a640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1900101440 0x7f1900197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.433+0000 7f190551a640 1 -- 172.21.15.60:0/4281224632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1900101e00 msgr2=0x7f190019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.433+0000 7f190551a640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1900101e00 0x7f190019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.433+0000 7f190551a640 1 -- 172.21.15.60:0/4281224632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190019f160 con 0x7f1900100250 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.433+0000 7f1904d19640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1900101440 0x7f1900197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.433+0000 7f190551a640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1900100250 0x7f1900197540 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f18f800c970 tx=0x7f18f800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.433+0000 7f18f67fc640 1 -- 172.21.15.60:0/4281224632 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f8007c10 con 0x7f1900100250 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.433+0000 7f18f67fc640 1 -- 172.21.15.60:0/4281224632 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18f8007db0 con 0x7f1900100250 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.433+0000 7f190777e640 1 -- 172.21.15.60:0/4281224632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190019f450 con 0x7f1900100250 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.433+0000 7f18f67fc640 1 -- 172.21.15.60:0/4281224632 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f800f940 con 0x7f1900100250 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.433+0000 7f190777e640 1 -- 172.21.15.60:0/4281224632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190019f9a0 con 0x7f1900100250 2024-03-04T12:25:39.443 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.434+0000 7f190777e640 1 -- 172.21.15.60:0/4281224632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190019fc60 con 0x7f1900100250 2024-03-04T12:25:39.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.435+0000 7f18f67fc640 1 -- 172.21.15.60:0/4281224632 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f18f800fae0 con 0x7f1900100250 2024-03-04T12:25:39.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.435+0000 7f18f67fc640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f18ec0a1d80 0x7f18ec0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:39.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.436+0000 7f1904d19640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f18ec0a1d80 0x7f18ec0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:39.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.436+0000 7f18f67fc640 1 -- 172.21.15.60:0/4281224632 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f18f80ce5e0 con 0x7f1900100250 2024-03-04T12:25:39.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.436+0000 7f1904d19640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f18ec0a1d80 0x7f18ec0a4240 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f18f00058c0 tx=0x7f18f0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:39.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.440+0000 7f18f67fc640 1 -- 172.21.15.60:0/4281224632 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f18f809b6b0 con 0x7f1900100250 2024-03-04T12:25:39.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.621+0000 7f190777e640 1 -- 172.21.15.60:0/4281224632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f190019fe30 con 0x7f1900100250 2024-03-04T12:25:39.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.622+0000 7f18f67fc640 1 -- 172.21.15.60:0/4281224632 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f18f809afb0 con 0x7f1900100250 2024-03-04T12:25:39.629 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-03-04T12:25:39.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.624+0000 7f190777e640 1 -- 172.21.15.60:0/4281224632 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f18ec0a1d80 msgr2=0x7f18ec0a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.624+0000 7f190777e640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f18ec0a1d80 0x7f18ec0a4240 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f18f00058c0 tx=0x7f18f0005850 comp rx=0 tx=0).stop 2024-03-04T12:25:39.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.624+0000 7f190777e640 1 -- 172.21.15.60:0/4281224632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1900100250 msgr2=0x7f1900197540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.624+0000 7f190777e640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1900100250 0x7f1900197540 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f18f800c970 tx=0x7f18f800ce30 comp rx=0 tx=0).stop 2024-03-04T12:25:39.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.624+0000 7f190777e640 1 -- 172.21.15.60:0/4281224632 shutdown_connections 2024-03-04T12:25:39.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.624+0000 7f190777e640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f18ec0a1d80 0x7f18ec0a4240 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.624+0000 7f190777e640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1900101e00 0x7f190019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.624+0000 7f190777e640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1900101440 0x7f1900197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.624+0000 7f190777e640 1 --2- 172.21.15.60:0/4281224632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1900100250 0x7f1900197540 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.624+0000 7f190777e640 1 -- 172.21.15.60:0/4281224632 >> 172.21.15.60:0/4281224632 conn(0x7f19000fb7c0 msgr2=0x7f1900068600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:39.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.624+0000 7f190777e640 1 -- 172.21.15.60:0/4281224632 shutdown_connections 2024-03-04T12:25:39.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.624+0000 7f190777e640 1 -- 172.21.15.60:0/4281224632 wait complete. 2024-03-04T12:25:39.642 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T12:25:39.749 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.740+0000 7f599f0fe640 1 -- 172.21.15.60:0/3703463566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59981046f0 msgr2=0x7f5998106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.740+0000 7f599f0fe640 1 --2- 172.21.15.60:0/3703463566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59981046f0 0x7f5998106ae0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5984009900 tx=0x7f598402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.740+0000 7f599f0fe640 1 -- 172.21.15.60:0/3703463566 shutdown_connections 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.740+0000 7f599f0fe640 1 --2- 172.21.15.60:0/3703463566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5998107020 0x7f5998109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.740+0000 7f599f0fe640 1 --2- 172.21.15.60:0/3703463566 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f59981046f0 0x7f5998106ae0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.740+0000 7f599f0fe640 1 --2- 172.21.15.60:0/3703463566 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5998101dc0 0x7f59981041b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.740+0000 7f599f0fe640 1 -- 172.21.15.60:0/3703463566 >> 172.21.15.60:0/3703463566 conn(0x7f59980fb790 msgr2=0x7f59980fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.740+0000 7f599f0fe640 1 -- 172.21.15.60:0/3703463566 shutdown_connections 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.740+0000 7f599f0fe640 1 -- 172.21.15.60:0/3703463566 wait complete. 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599f0fe640 1 Processor -- start 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599f0fe640 1 -- start start 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599f0fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5998101dc0 0x7f5998101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599f0fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59981046f0 0x7f59980ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:39.750 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599f0fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5998107020 0x7f59981002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599f0fe640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599810cf90 con 0x7f5998107020 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599f0fe640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599810ce10 con 0x7f59981046f0 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599f0fe640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f599810d110 con 0x7f5998101dc0 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599d69b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5998107020 0x7f59981002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599d69b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5998107020 0x7f59981002b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55760/0 (socket says 172.21.15.60:55760) 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599d69b640 1 -- 172.21.15.60:0/3548293537 learned_addr learned my addr 172.21.15.60:0/3548293537 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599ce9a640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5998101dc0 0x7f5998101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f5997fff640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59981046f0 0x7f59980ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599d69b640 1 -- 172.21.15.60:0/3548293537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5998101dc0 msgr2=0x7f5998101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599d69b640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5998101dc0 0x7f5998101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599d69b640 1 -- 172.21.15.60:0/3548293537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59981046f0 msgr2=0x7f59980ffd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599d69b640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59981046f0 0x7f59980ffd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599d69b640 1 -- 172.21.15.60:0/3548293537 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5998100b40 con 0x7f5998107020 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f5997fff640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59981046f0 0x7f59980ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:39.751 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.741+0000 7f599d69b640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5998107020 0x7f59981002b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f599000eb10 tx=0x7f599000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:39.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.742+0000 7f5995ffb640 1 -- 172.21.15.60:0/3548293537 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5990004b20 con 0x7f5998107020 2024-03-04T12:25:39.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.742+0000 7f5995ffb640 1 -- 172.21.15.60:0/3548293537 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5990004cc0 con 0x7f5998107020 2024-03-04T12:25:39.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.742+0000 7f599f0fe640 1 -- 172.21.15.60:0/3548293537 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59981a17e0 con 0x7f5998107020 2024-03-04T12:25:39.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.742+0000 7f5995ffb640 1 -- 172.21.15.60:0/3548293537 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5990010760 con 0x7f5998107020 2024-03-04T12:25:39.752 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.742+0000 7f599f0fe640 1 -- 172.21.15.60:0/3548293537 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59981a1cb0 con 0x7f5998107020 2024-03-04T12:25:39.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.743+0000 7f599f0fe640 1 -- 172.21.15.60:0/3548293537 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5998068600 con 0x7f5998107020 2024-03-04T12:25:39.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.744+0000 7f5995ffb640 1 -- 172.21.15.60:0/3548293537 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 18) v1 ==== 136753+0+0 (secure 0 0 0) 0x7f5990005020 con 0x7f5998107020 2024-03-04T12:25:39.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.745+0000 7f5995ffb640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f59880a1d80 0x7f59880a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:39.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.745+0000 7f5995ffb640 1 -- 172.21.15.60:0/3548293537 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f59900ce860 con 0x7f5998107020 2024-03-04T12:25:39.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.745+0000 7f599ce9a640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f59880a1d80 0x7f59880a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:39.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.745+0000 7f599ce9a640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f59880a1d80 0x7f59880a4240 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f598c0095f0 tx=0x7f598c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:39.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.749+0000 7f5995ffb640 1 -- 172.21.15.60:0/3548293537 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f599009b8a0 con 0x7f5998107020 2024-03-04T12:25:39.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.929+0000 7f599f0fe640 1 -- 172.21.15.60:0/3548293537 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5998061f30 con 0x7f5998107020 2024-03-04T12:25:39.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.929+0000 7f5995ffb640 1 -- 172.21.15.60:0/3548293537 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f599009b1a0 con 0x7f5998107020 2024-03-04T12:25:39.937 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:39.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.931+0000 7f599f0fe640 1 -- 172.21.15.60:0/3548293537 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f59880a1d80 msgr2=0x7f59880a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.931+0000 7f599f0fe640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f59880a1d80 0x7f59880a4240 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f598c0095f0 tx=0x7f598c005f90 comp rx=0 tx=0).stop 2024-03-04T12:25:39.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.931+0000 7f599f0fe640 1 -- 172.21.15.60:0/3548293537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5998107020 msgr2=0x7f59981002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:39.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.931+0000 7f599f0fe640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5998107020 0x7f59981002b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f599000eb10 tx=0x7f599000c2d0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.931+0000 7f599f0fe640 1 -- 172.21.15.60:0/3548293537 shutdown_connections 2024-03-04T12:25:39.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.931+0000 7f599f0fe640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f59880a1d80 0x7f59880a4240 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.931+0000 7f599f0fe640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5998107020 0x7f59981002b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.931+0000 7f599f0fe640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59981046f0 0x7f59980ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.931+0000 7f599f0fe640 1 --2- 172.21.15.60:0/3548293537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5998101dc0 0x7f5998101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:39.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.931+0000 7f599f0fe640 1 -- 172.21.15.60:0/3548293537 >> 172.21.15.60:0/3548293537 conn(0x7f59980fb790 msgr2=0x7f5998102860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:39.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.931+0000 7f599f0fe640 1 -- 172.21.15.60:0/3548293537 shutdown_connections 2024-03-04T12:25:39.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:39.931+0000 7f599f0fe640 1 -- 172.21.15.60:0/3548293537 wait complete. 2024-03-04T12:25:39.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:25:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.048+0000 7f7eed916640 1 -- 172.21.15.60:0/2801892523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ee8102f20 msgr2=0x7f7ee81097b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.048+0000 7f7eed916640 1 --2- 172.21.15.60:0/2801892523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ee8102f20 0x7f7ee81097b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009900 tx=0x7f7ed802bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.048+0000 7f7eed916640 1 -- 172.21.15.60:0/2801892523 shutdown_connections 2024-03-04T12:25:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.048+0000 7f7eed916640 1 --2- 172.21.15.60:0/2801892523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ee8102f20 0x7f7ee81097b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.048+0000 7f7eed916640 1 --2- 172.21.15.60:0/2801892523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ee8102560 0x7f7ee81029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.048+0000 7f7eed916640 1 --2- 172.21.15.60:0/2801892523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ee8101370 0x7f7ee8101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.048+0000 7f7eed916640 1 -- 172.21.15.60:0/2801892523 >> 172.21.15.60:0/2801892523 conn(0x7f7ee80fc8c0 msgr2=0x7f7ee80fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.048+0000 7f7eed916640 1 -- 172.21.15.60:0/2801892523 shutdown_connections 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.048+0000 7f7eed916640 1 -- 172.21.15.60:0/2801892523 wait complete. 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7eed916640 1 Processor -- start 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7eed916640 1 -- start start 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7eed916640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ee8101370 0x7f7ee8197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7eed916640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ee8102560 0x7f7ee8197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7eed916640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ee8102f20 0x7f7ee819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7eed916640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee810d000 con 0x7f7ee8102f20 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7eed916640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee810ce80 con 0x7f7ee8101370 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7eed916640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee810d180 con 0x7f7ee8102560 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7ee77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ee8102f20 0x7f7ee819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:40.059 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7ee77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ee8102f20 0x7f7ee819eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55770/0 (socket says 172.21.15.60:55770) 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7ee77fe640 1 -- 172.21.15.60:0/2760828611 learned_addr learned my addr 172.21.15.60:0/2760828611 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7ee67fc640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ee8102560 0x7f7ee8197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7ee6ffd640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ee8101370 0x7f7ee8197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7ee77fe640 1 -- 172.21.15.60:0/2760828611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ee8102560 msgr2=0x7f7ee8197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7ee77fe640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ee8102560 0x7f7ee8197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7ee77fe640 1 -- 172.21.15.60:0/2760828611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ee8101370 msgr2=0x7f7ee8197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7ee77fe640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ee8101370 0x7f7ee8197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7ee77fe640 1 -- 172.21.15.60:0/2760828611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee819f180 con 0x7f7ee8102f20 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.049+0000 7f7ee6ffd640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ee8101370 0x7f7ee8197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.050+0000 7f7ee77fe640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ee8102f20 0x7f7ee819eb10 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7ed802ce70 tx=0x7f7ed8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.050+0000 7f7eec914640 1 -- 172.21.15.60:0/2760828611 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8041070 con 0x7f7ee8102f20 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.050+0000 7f7eec914640 1 -- 172.21.15.60:0/2760828611 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ed80054e0 con 0x7f7ee8102f20 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.050+0000 7f7eed916640 1 -- 172.21.15.60:0/2760828611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee819f410 con 0x7f7ee8102f20 2024-03-04T12:25:40.060 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.050+0000 7f7eec914640 1 -- 172.21.15.60:0/2760828611 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed80344a0 con 0x7f7ee8102f20 2024-03-04T12:25:40.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.050+0000 7f7eed916640 1 -- 172.21.15.60:0/2760828611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee819f950 con 0x7f7ee8102f20 2024-03-04T12:25:40.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.051+0000 7f7eed916640 1 -- 172.21.15.60:0/2760828611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee81085c0 con 0x7f7ee8102f20 2024-03-04T12:25:40.061 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.052+0000 7f7eec914640 1 -- 172.21.15.60:0/2760828611 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f7ed8044070 con 0x7f7ee8102f20 2024-03-04T12:25:40.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.052+0000 7f7eec914640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7ed00a1e90 0x7f7ed00a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:40.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.053+0000 7f7ee6ffd640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7ed00a1e90 0x7f7ed00a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:40.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.053+0000 7f7eec914640 1 -- 172.21.15.60:0/2760828611 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f7ed80eff60 con 0x7f7ee8102f20 2024-03-04T12:25:40.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.053+0000 7f7ee6ffd640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7ed00a1e90 0x7f7ed00a4350 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7ee81023c0 tx=0x7f7ee000a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:40.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.057+0000 7f7eec914640 1 -- 172.21.15.60:0/2760828611 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7ed80bcf50 con 0x7f7ee8102f20 2024-03-04T12:25:40.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.240+0000 7f7eed916640 1 -- 172.21.15.60:0/2760828611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ee8061f30 con 0x7f7ee8102f20 2024-03-04T12:25:40.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.242+0000 7f7eec914640 1 -- 172.21.15.60:0/2760828611 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24562 (secure 0 0 0) 0x7f7ed80bc850 con 0x7f7ee8102f20 2024-03-04T12:25:40.249 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:40.250 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":93,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:25:06.929248+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:25:40.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.244+0000 7f7eed916640 1 -- 172.21.15.60:0/2760828611 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7ed00a1e90 msgr2=0x7f7ed00a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.244+0000 7f7eed916640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7ed00a1e90 0x7f7ed00a4350 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7ee81023c0 tx=0x7f7ee000a5e0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.244+0000 7f7eed916640 1 -- 172.21.15.60:0/2760828611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ee8102f20 msgr2=0x7f7ee819eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.244+0000 7f7eed916640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ee8102f20 0x7f7ee819eb10 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7ed802ce70 tx=0x7f7ed8005e40 comp rx=0 tx=0).stop 2024-03-04T12:25:40.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.244+0000 7f7eed916640 1 -- 172.21.15.60:0/2760828611 shutdown_connections 2024-03-04T12:25:40.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.244+0000 7f7eed916640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7ed00a1e90 0x7f7ed00a4350 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.244+0000 7f7eed916640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7ee8102f20 0x7f7ee819eb10 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.244+0000 7f7eed916640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ee8102560 0x7f7ee8197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.244+0000 7f7eed916640 1 --2- 172.21.15.60:0/2760828611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ee8101370 0x7f7ee8197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.244+0000 7f7eed916640 1 -- 172.21.15.60:0/2760828611 >> 172.21.15.60:0/2760828611 conn(0x7f7ee80fc8c0 msgr2=0x7f7ee8105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:40.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.244+0000 7f7eed916640 1 -- 172.21.15.60:0/2760828611 shutdown_connections 2024-03-04T12:25:40.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.244+0000 7f7eed916640 1 -- 172.21.15.60:0/2760828611 wait complete. 2024-03-04T12:25:40.261 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:40 smithi060 ceph-mon[32745]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-04T12:25:40.261 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:40 smithi060 ceph-mon[32745]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:40.261 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:40 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1325290955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:40.261 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:40 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3193439864' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:40.261 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:40 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/4281224632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:40.261 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:40 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3548293537' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:40.263 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T12:23:10.438928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T12:23:11.472998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T12:23:15.162500+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T12:25:40.263 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:25:40.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:40 smithi086 ceph-mon[36851]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-04T12:25:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:40 smithi086 ceph-mon[36851]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:40 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1325290955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:40 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3193439864' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:40 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/4281224632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:40.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:40 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3548293537' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:40 smithi063 ceph-mon[36717]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-04T12:25:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:40 smithi063 ceph-mon[36717]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:40 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1325290955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:40 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3193439864' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:40 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/4281224632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:40.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:40 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3548293537' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:40.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.363+0000 7f52116e8640 1 -- 172.21.15.60:0/16452852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f520c102ee0 msgr2=0x7f520c109770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.363+0000 7f52116e8640 1 --2- 172.21.15.60:0/16452852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f520c102ee0 0x7f520c109770 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5204009ca0 tx=0x7f520402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.363+0000 7f52116e8640 1 -- 172.21.15.60:0/16452852 shutdown_connections 2024-03-04T12:25:40.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.363+0000 7f52116e8640 1 --2- 172.21.15.60:0/16452852 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f520c102ee0 0x7f520c109770 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.363+0000 7f52116e8640 1 --2- 172.21.15.60:0/16452852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f520c102520 0x7f520c1029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.363+0000 7f52116e8640 1 --2- 172.21.15.60:0/16452852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f520c101330 0x7f520c101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.363+0000 7f52116e8640 1 -- 172.21.15.60:0/16452852 >> 172.21.15.60:0/16452852 conn(0x7f520c0fc880 msgr2=0x7f520c0fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:40.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.364+0000 7f52116e8640 1 -- 172.21.15.60:0/16452852 shutdown_connections 2024-03-04T12:25:40.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.364+0000 7f52116e8640 1 -- 172.21.15.60:0/16452852 wait complete. 2024-03-04T12:25:40.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.364+0000 7f52116e8640 1 Processor -- start 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.364+0000 7f52116e8640 1 -- start start 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.364+0000 7f52116e8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f520c101330 0x7f520c1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.364+0000 7f52116e8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f520c102520 0x7f520c197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.364+0000 7f52116e8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f520c102ee0 0x7f520c19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.364+0000 7f52116e8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c10cfc0 con 0x7f520c102ee0 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.364+0000 7f52116e8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c10ce40 con 0x7f520c101330 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.364+0000 7f52116e8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c10d140 con 0x7f520c102520 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f520b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f520c102ee0 0x7f520c19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f520b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f520c102ee0 0x7f520c19eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55796/0 (socket says 172.21.15.60:55796) 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f520b7fe640 1 -- 172.21.15.60:0/949309776 learned_addr learned my addr 172.21.15.60:0/949309776 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f520a7fc640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f520c102520 0x7f520c197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f520affd640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f520c101330 0x7f520c1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f520b7fe640 1 -- 172.21.15.60:0/949309776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f520c102520 msgr2=0x7f520c197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f520b7fe640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f520c102520 0x7f520c197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f520b7fe640 1 -- 172.21.15.60:0/949309776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f520c101330 msgr2=0x7f520c1975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f520b7fe640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f520c101330 0x7f520c1975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f520b7fe640 1 -- 172.21.15.60:0/949309776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c19f210 con 0x7f520c102ee0 2024-03-04T12:25:40.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f520b7fe640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f520c102ee0 0x7f520c19eba0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f520402c630 tx=0x7f5204032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:40.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f5203fff640 1 -- 172.21.15.60:0/949309776 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204041020 con 0x7f520c102ee0 2024-03-04T12:25:40.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f52116e8640 1 -- 172.21.15.60:0/949309776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520c19f4a0 con 0x7f520c102ee0 2024-03-04T12:25:40.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f5203fff640 1 -- 172.21.15.60:0/949309776 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5204034cd0 con 0x7f520c102ee0 2024-03-04T12:25:40.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f5203fff640 1 -- 172.21.15.60:0/949309776 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5204039040 con 0x7f520c102ee0 2024-03-04T12:25:40.375 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.365+0000 7f52116e8640 1 -- 172.21.15.60:0/949309776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520c19f9e0 con 0x7f520c102ee0 2024-03-04T12:25:40.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.366+0000 7f52116e8640 1 -- 172.21.15.60:0/949309776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520c108650 con 0x7f520c102ee0 2024-03-04T12:25:40.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.367+0000 7f5203fff640 1 -- 172.21.15.60:0/949309776 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f5204044020 con 0x7f520c102ee0 2024-03-04T12:25:40.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.368+0000 7f5203fff640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f51f80a1e10 0x7f51f80a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:40.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.368+0000 7f520affd640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f51f80a1e10 0x7f51f80a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:40.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.368+0000 7f5203fff640 1 -- 172.21.15.60:0/949309776 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f52040f0140 con 0x7f520c102ee0 2024-03-04T12:25:40.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.369+0000 7f520affd640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f51f80a1e10 0x7f51f80a42d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f51f4005d00 tx=0x7f51f4005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:40.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.373+0000 7f5203fff640 1 -- 172.21.15.60:0/949309776 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f520404b020 con 0x7f520c102ee0 2024-03-04T12:25:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.556+0000 7f52116e8640 1 -- 172.21.15.60:0/949309776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f520c02d0b0 con 0x7f520c102ee0 2024-03-04T12:25:40.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.557+0000 7f5203fff640 1 -- 172.21.15.60:0/949309776 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f52040bd190 con 0x7f520c102ee0 2024-03-04T12:25:40.564 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:25:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.559+0000 7f52116e8640 1 -- 172.21.15.60:0/949309776 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f51f80a1e10 msgr2=0x7f51f80a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.559+0000 7f52116e8640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f51f80a1e10 0x7f51f80a42d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f51f4005d00 tx=0x7f51f4005c50 comp rx=0 tx=0).stop 2024-03-04T12:25:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.559+0000 7f52116e8640 1 -- 172.21.15.60:0/949309776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f520c102ee0 msgr2=0x7f520c19eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.559+0000 7f52116e8640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f520c102ee0 0x7f520c19eba0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f520402c630 tx=0x7f5204032ea0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.559+0000 7f52116e8640 1 -- 172.21.15.60:0/949309776 shutdown_connections 2024-03-04T12:25:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.559+0000 7f52116e8640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f51f80a1e10 0x7f51f80a42d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.559+0000 7f52116e8640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f520c102ee0 0x7f520c19eba0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.559+0000 7f52116e8640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f520c102520 0x7f520c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.559+0000 7f52116e8640 1 --2- 172.21.15.60:0/949309776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f520c101330 0x7f520c1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.559+0000 7f52116e8640 1 -- 172.21.15.60:0/949309776 >> 172.21.15.60:0/949309776 conn(0x7f520c0fc880 msgr2=0x7f520c105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.560+0000 7f52116e8640 1 -- 172.21.15.60:0/949309776 shutdown_connections 2024-03-04T12:25:40.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.560+0000 7f52116e8640 1 -- 172.21.15.60:0/949309776 wait complete. 2024-03-04T12:25:40.578 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T12:25:40.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.680+0000 7f4908230640 1 -- 172.21.15.60:0/461281032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4900107030 msgr2=0x7f4900109480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.680+0000 7f4908230640 1 --2- 172.21.15.60:0/461281032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4900107030 0x7f4900109480 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009900 tx=0x7f48f802bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:40.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 -- 172.21.15.60:0/461281032 shutdown_connections 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 --2- 172.21.15.60:0/461281032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4900107030 0x7f4900109480 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 --2- 172.21.15.60:0/461281032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4900104700 0x7f4900106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 --2- 172.21.15.60:0/461281032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900101dd0 0x7f49001041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 -- 172.21.15.60:0/461281032 >> 172.21.15.60:0/461281032 conn(0x7f49000fb7c0 msgr2=0x7f49000fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 -- 172.21.15.60:0/461281032 shutdown_connections 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 -- 172.21.15.60:0/461281032 wait complete. 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 Processor -- start 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 -- start start 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900101dd0 0x7f49001975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4900104700 0x7f4900197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4900107030 0x7f490019eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490010cfe0 con 0x7f4900107030 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490010ce60 con 0x7f4900101dd0 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.681+0000 7f4908230640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490010d160 con 0x7f4900104700 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f49067cd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4900107030 0x7f490019eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:40.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f49067cd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4900107030 0x7f490019eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55812/0 (socket says 172.21.15.60:55812) 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f49067cd640 1 -- 172.21.15.60:0/538512048 learned_addr learned my addr 172.21.15.60:0/538512048 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f49057cb640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4900104700 0x7f4900197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f4905fcc640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900101dd0 0x7f49001975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f49067cd640 1 -- 172.21.15.60:0/538512048 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4900104700 msgr2=0x7f4900197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f49067cd640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4900104700 0x7f4900197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f49067cd640 1 -- 172.21.15.60:0/538512048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900101dd0 msgr2=0x7f49001975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f49067cd640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900101dd0 0x7f49001975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f49067cd640 1 -- 172.21.15.60:0/538512048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490019f200 con 0x7f4900107030 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f49067cd640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4900107030 0x7f490019eb90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f48f802ce70 tx=0x7f48f8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f48feffd640 1 -- 172.21.15.60:0/538512048 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8041070 con 0x7f4900107030 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f48feffd640 1 -- 172.21.15.60:0/538512048 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48f80054e0 con 0x7f4900107030 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f4908230640 1 -- 172.21.15.60:0/538512048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490019f490 con 0x7f4900107030 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f48feffd640 1 -- 172.21.15.60:0/538512048 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f80344a0 con 0x7f4900107030 2024-03-04T12:25:40.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.682+0000 7f4908230640 1 -- 172.21.15.60:0/538512048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490019f9d0 con 0x7f4900107030 2024-03-04T12:25:40.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.683+0000 7f4908230640 1 -- 172.21.15.60:0/538512048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49000fd2b0 con 0x7f4900107030 2024-03-04T12:25:40.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.685+0000 7f48feffd640 1 -- 172.21.15.60:0/538512048 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f48f8044070 con 0x7f4900107030 2024-03-04T12:25:40.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.685+0000 7f48feffd640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f48e80a1e90 0x7f48e80a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:40.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.685+0000 7f48feffd640 1 -- 172.21.15.60:0/538512048 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f48f80effb0 con 0x7f4900107030 2024-03-04T12:25:40.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.685+0000 7f4905fcc640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f48e80a1e90 0x7f48e80a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:40.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.686+0000 7f4905fcc640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f48e80a1e90 0x7f48e80a4350 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f48f0005c40 tx=0x7f48f0005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.692+0000 7f48feffd640 1 -- 172.21.15.60:0/538512048 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f48f80bd000 con 0x7f4900107030 2024-03-04T12:25:40.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.876+0000 7f4908230640 1 -- 172.21.15.60:0/538512048 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4900061f30 con 0x7f4900107030 2024-03-04T12:25:40.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.877+0000 7f48feffd640 1 -- 172.21.15.60:0/538512048 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f48f80bc900 con 0x7f4900107030 2024-03-04T12:25:40.884 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:40.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.879+0000 7f4908230640 1 -- 172.21.15.60:0/538512048 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f48e80a1e90 msgr2=0x7f48e80a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.879+0000 7f4908230640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f48e80a1e90 0x7f48e80a4350 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f48f0005c40 tx=0x7f48f0005bd0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.886 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.879+0000 7f4908230640 1 -- 172.21.15.60:0/538512048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4900107030 msgr2=0x7f490019eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:40.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.879+0000 7f4908230640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4900107030 0x7f490019eb90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f48f802ce70 tx=0x7f48f8005e40 comp rx=0 tx=0).stop 2024-03-04T12:25:40.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.879+0000 7f4908230640 1 -- 172.21.15.60:0/538512048 shutdown_connections 2024-03-04T12:25:40.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.879+0000 7f4908230640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f48e80a1e90 0x7f48e80a4350 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.879+0000 7f4908230640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4900107030 0x7f490019eb90 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.879+0000 7f4908230640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4900104700 0x7f4900197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.879+0000 7f4908230640 1 --2- 172.21.15.60:0/538512048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4900101dd0 0x7f49001975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:40.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.879+0000 7f4908230640 1 -- 172.21.15.60:0/538512048 >> 172.21.15.60:0/538512048 conn(0x7f49000fb7c0 msgr2=0x7f4900102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:40.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.879+0000 7f4908230640 1 -- 172.21.15.60:0/538512048 shutdown_connections 2024-03-04T12:25:40.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.879+0000 7f4908230640 1 -- 172.21.15.60:0/538512048 wait complete. 2024-03-04T12:25:40.897 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T12:25:41.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.999+0000 7fdc45be6640 1 -- 172.21.15.60:0/449226899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc40101340 msgr2=0x7fdc40101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.999+0000 7fdc45be6640 1 --2- 172.21.15.60:0/449226899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc40101340 0x7fdc40101760 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fdc30009900 tx=0x7fdc3002bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:41.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.999+0000 7fdc45be6640 1 -- 172.21.15.60:0/449226899 shutdown_connections 2024-03-04T12:25:41.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.999+0000 7fdc45be6640 1 --2- 172.21.15.60:0/449226899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc40102ef0 0x7fdc40109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.999+0000 7fdc45be6640 1 --2- 172.21.15.60:0/449226899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc40102530 0x7fdc401029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.999+0000 7fdc45be6640 1 --2- 172.21.15.60:0/449226899 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc40101340 0x7fdc40101760 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.999+0000 7fdc45be6640 1 -- 172.21.15.60:0/449226899 >> 172.21.15.60:0/449226899 conn(0x7fdc400fc8b0 msgr2=0x7fdc400fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:41.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.999+0000 7fdc45be6640 1 -- 172.21.15.60:0/449226899 shutdown_connections 2024-03-04T12:25:41.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:40.999+0000 7fdc45be6640 1 -- 172.21.15.60:0/449226899 wait complete. 2024-03-04T12:25:41.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.000+0000 7fdc45be6640 1 Processor -- start 2024-03-04T12:25:41.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.000+0000 7fdc45be6640 1 -- start start 2024-03-04T12:25:41.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.000+0000 7fdc45be6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc40101340 0x7fdc401976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.000+0000 7fdc45be6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc40102530 0x7fdc40197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.000+0000 7fdc45be6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc40102ef0 0x7fdc4019ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.000+0000 7fdc45be6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4010cfd0 con 0x7fdc40102ef0 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.000+0000 7fdc45be6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4010ce50 con 0x7fdc40102530 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.000+0000 7fdc45be6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4010d150 con 0x7fdc40101340 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.000+0000 7fdc3ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc40102ef0 0x7fdc4019ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.000+0000 7fdc3ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc40102ef0 0x7fdc4019ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55838/0 (socket says 172.21.15.60:55838) 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.000+0000 7fdc3ffff640 1 -- 172.21.15.60:0/2151704167 learned_addr learned my addr 172.21.15.60:0/2151704167 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc3effd640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc40102530 0x7fdc40197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc3f7fe640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc40101340 0x7fdc401976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc3ffff640 1 -- 172.21.15.60:0/2151704167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc40101340 msgr2=0x7fdc401976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc3ffff640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc40101340 0x7fdc401976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc3ffff640 1 -- 172.21.15.60:0/2151704167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc40102530 msgr2=0x7fdc40197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc3ffff640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc40102530 0x7fdc40197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc3ffff640 1 -- 172.21.15.60:0/2151704167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4019f2d0 con 0x7fdc40102ef0 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc3effd640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc40102530 0x7fdc40197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc3ffff640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc40102ef0 0x7fdc4019ec60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009fd0 tx=0x7fdc3800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc3cff9640 1 -- 172.21.15.60:0/2151704167 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38017070 con 0x7fdc40102ef0 2024-03-04T12:25:41.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc45be6640 1 -- 172.21.15.60:0/2151704167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4019f5c0 con 0x7fdc40102ef0 2024-03-04T12:25:41.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc3cff9640 1 -- 172.21.15.60:0/2151704167 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc38003e50 con 0x7fdc40102ef0 2024-03-04T12:25:41.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc3cff9640 1 -- 172.21.15.60:0/2151704167 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc380106c0 con 0x7fdc40102ef0 2024-03-04T12:25:41.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.001+0000 7fdc45be6640 1 -- 172.21.15.60:0/2151704167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4019fac0 con 0x7fdc40102ef0 2024-03-04T12:25:41.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.002+0000 7fdc45be6640 1 -- 172.21.15.60:0/2151704167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc401086b0 con 0x7fdc40102ef0 2024-03-04T12:25:41.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.003+0000 7fdc3cff9640 1 -- 172.21.15.60:0/2151704167 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fdc38003970 con 0x7fdc40102ef0 2024-03-04T12:25:41.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.004+0000 7fdc3cff9640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdc280a1e10 0x7fdc280a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.004+0000 7fdc3cff9640 1 -- 172.21.15.60:0/2151704167 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fdc380ce170 con 0x7fdc40102ef0 2024-03-04T12:25:41.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.004+0000 7fdc3f7fe640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdc280a1e10 0x7fdc280a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.004+0000 7fdc3f7fe640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdc280a1e10 0x7fdc280a42d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdc3002c7b0 tx=0x7fdc30005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:41.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.007+0000 7fdc3cff9640 1 -- 172.21.15.60:0/2151704167 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdc3809b050 con 0x7fdc40102ef0 2024-03-04T12:25:41.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:41 smithi060 ceph-mon[32745]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-04T12:25:41.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:41 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2760828611' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:41.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:41 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/949309776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:41.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:41 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/538512048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:41.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.192+0000 7fdc45be6640 1 -- 172.21.15.60:0/2151704167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdc4002ceb0 con 0x7fdc40102ef0 2024-03-04T12:25:41.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.192+0000 7fdc3cff9640 1 -- 172.21.15.60:0/2151704167 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdc3809a950 con 0x7fdc40102ef0 2024-03-04T12:25:41.199 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-03-04T12:25:41.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.194+0000 7fdc45be6640 1 -- 172.21.15.60:0/2151704167 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdc280a1e10 msgr2=0x7fdc280a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.194+0000 7fdc45be6640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdc280a1e10 0x7fdc280a42d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdc3002c7b0 tx=0x7fdc30005fd0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.194+0000 7fdc45be6640 1 -- 172.21.15.60:0/2151704167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc40102ef0 msgr2=0x7fdc4019ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.194+0000 7fdc45be6640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc40102ef0 0x7fdc4019ec60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009fd0 tx=0x7fdc3800b260 comp rx=0 tx=0).stop 2024-03-04T12:25:41.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.194+0000 7fdc45be6640 1 -- 172.21.15.60:0/2151704167 shutdown_connections 2024-03-04T12:25:41.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.194+0000 7fdc45be6640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fdc280a1e10 0x7fdc280a42d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.194+0000 7fdc45be6640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdc40102ef0 0x7fdc4019ec60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.194+0000 7fdc45be6640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc40102530 0x7fdc40197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.194+0000 7fdc45be6640 1 --2- 172.21.15.60:0/2151704167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc40101340 0x7fdc401976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.194+0000 7fdc45be6640 1 -- 172.21.15.60:0/2151704167 >> 172.21.15.60:0/2151704167 conn(0x7fdc400fc8b0 msgr2=0x7fdc40105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:41.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.194+0000 7fdc45be6640 1 -- 172.21.15.60:0/2151704167 shutdown_connections 2024-03-04T12:25:41.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.194+0000 7fdc45be6640 1 -- 172.21.15.60:0/2151704167 wait complete. 2024-03-04T12:25:41.213 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T12:25:41.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.312+0000 7efdcf75b640 1 -- 172.21.15.60:0/563505645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8100250 msgr2=0x7efdc8100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.312+0000 7efdcf75b640 1 --2- 172.21.15.60:0/563505645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8100250 0x7efdc8100670 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efdbc009900 tx=0x7efdbc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:41.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.312+0000 7efdcf75b640 1 -- 172.21.15.60:0/563505645 shutdown_connections 2024-03-04T12:25:41.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.312+0000 7efdcf75b640 1 --2- 172.21.15.60:0/563505645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdc8101e00 0x7efdc8108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.312+0000 7efdcf75b640 1 --2- 172.21.15.60:0/563505645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdc8101440 0x7efdc81018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.312+0000 7efdcf75b640 1 --2- 172.21.15.60:0/563505645 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8100250 0x7efdc8100670 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.312+0000 7efdcf75b640 1 -- 172.21.15.60:0/563505645 >> 172.21.15.60:0/563505645 conn(0x7efdc80fb7c0 msgr2=0x7efdc80fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:41.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.312+0000 7efdcf75b640 1 -- 172.21.15.60:0/563505645 shutdown_connections 2024-03-04T12:25:41.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.312+0000 7efdcf75b640 1 -- 172.21.15.60:0/563505645 wait complete. 2024-03-04T12:25:41.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.312+0000 7efdcf75b640 1 Processor -- start 2024-03-04T12:25:41.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.312+0000 7efdcf75b640 1 -- start start 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcf75b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdc8100250 0x7efdc81976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcf75b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdc8101440 0x7efdc8197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcf75b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8101e00 0x7efdc819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcf75b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc810bee0 con 0x7efdc8101e00 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcf75b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc810bd60 con 0x7efdc8100250 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcf75b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc810c060 con 0x7efdc8101440 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcdcf8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8101e00 0x7efdc819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcdcf8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8101e00 0x7efdc819ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55848/0 (socket says 172.21.15.60:55848) 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcdcf8640 1 -- 172.21.15.60:0/4113103525 learned_addr learned my addr 172.21.15.60:0/4113103525 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcd4f7640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdc8100250 0x7efdc81976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcccf6640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdc8101440 0x7efdc8197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcdcf8640 1 -- 172.21.15.60:0/4113103525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdc8101440 msgr2=0x7efdc8197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcdcf8640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdc8101440 0x7efdc8197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcdcf8640 1 -- 172.21.15.60:0/4113103525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdc8100250 msgr2=0x7efdc81976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcdcf8640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdc8100250 0x7efdc81976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcdcf8640 1 -- 172.21.15.60:0/4113103525 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc819f2b0 con 0x7efdc8101e00 2024-03-04T12:25:41.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcdcf8640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8101e00 0x7efdc819ec40 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efdc000eaa0 tx=0x7efdc000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:41.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdba7fc640 1 -- 172.21.15.60:0/4113103525 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc000cdb0 con 0x7efdc8101e00 2024-03-04T12:25:41.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdba7fc640 1 -- 172.21.15.60:0/4113103525 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdc0005020 con 0x7efdc8101e00 2024-03-04T12:25:41.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdcf75b640 1 -- 172.21.15.60:0/4113103525 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc819f5a0 con 0x7efdc8101e00 2024-03-04T12:25:41.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.313+0000 7efdba7fc640 1 -- 172.21.15.60:0/4113103525 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc00116b0 con 0x7efdc8101e00 2024-03-04T12:25:41.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.314+0000 7efdcf75b640 1 -- 172.21.15.60:0/4113103525 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc819faa0 con 0x7efdc8101e00 2024-03-04T12:25:41.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.314+0000 7efdcf75b640 1 -- 172.21.15.60:0/4113103525 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc819fd60 con 0x7efdc8101e00 2024-03-04T12:25:41.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.316+0000 7efdba7fc640 1 -- 172.21.15.60:0/4113103525 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7efdc0005560 con 0x7efdc8101e00 2024-03-04T12:25:41.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.316+0000 7efdba7fc640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efdb40a1e10 0x7efdb40a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.316+0000 7efdba7fc640 1 -- 172.21.15.60:0/4113103525 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7efdc00ce810 con 0x7efdc8101e00 2024-03-04T12:25:41.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.316+0000 7efdcd4f7640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efdb40a1e10 0x7efdb40a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.324 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.317+0000 7efdcd4f7640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efdb40a1e10 0x7efdb40a42d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efdbc000c00 tx=0x7efdbc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:41.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.320+0000 7efdba7fc640 1 -- 172.21.15.60:0/4113103525 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efdc0016030 con 0x7efdc8101e00 2024-03-04T12:25:41.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:41 smithi086 ceph-mon[36851]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-04T12:25:41.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:41 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2760828611' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:41.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:41 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/949309776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:41.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:41 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/538512048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:41.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:41 smithi063 ceph-mon[36717]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-04T12:25:41.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:41 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2760828611' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:41.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:41 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/949309776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:41.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:41 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/538512048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:41.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.502+0000 7efdcf75b640 1 -- 172.21.15.60:0/4113103525 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7efdc819ff30 con 0x7efdc8101e00 2024-03-04T12:25:41.510 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.503+0000 7efdba7fc640 1 -- 172.21.15.60:0/4113103525 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7efdc009b100 con 0x7efdc8101e00 2024-03-04T12:25:41.510 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:41.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.505+0000 7efdcf75b640 1 -- 172.21.15.60:0/4113103525 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efdb40a1e10 msgr2=0x7efdb40a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.505+0000 7efdcf75b640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efdb40a1e10 0x7efdb40a42d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7efdbc000c00 tx=0x7efdbc005fd0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.505+0000 7efdcf75b640 1 -- 172.21.15.60:0/4113103525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8101e00 msgr2=0x7efdc819ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.505+0000 7efdcf75b640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8101e00 0x7efdc819ec40 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efdc000eaa0 tx=0x7efdc000ef60 comp rx=0 tx=0).stop 2024-03-04T12:25:41.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.505+0000 7efdcf75b640 1 -- 172.21.15.60:0/4113103525 shutdown_connections 2024-03-04T12:25:41.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.505+0000 7efdcf75b640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efdb40a1e10 0x7efdb40a42d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.505+0000 7efdcf75b640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efdc8101e00 0x7efdc819ec40 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.505+0000 7efdcf75b640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efdc8101440 0x7efdc8197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.505+0000 7efdcf75b640 1 --2- 172.21.15.60:0/4113103525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdc8100250 0x7efdc81976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.505+0000 7efdcf75b640 1 -- 172.21.15.60:0/4113103525 >> 172.21.15.60:0/4113103525 conn(0x7efdc80fb7c0 msgr2=0x7efdc80fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:41.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.505+0000 7efdcf75b640 1 -- 172.21.15.60:0/4113103525 shutdown_connections 2024-03-04T12:25:41.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.505+0000 7efdcf75b640 1 -- 172.21.15.60:0/4113103525 wait complete. 2024-03-04T12:25:41.523 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-04T12:25:41.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:25:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.619+0000 7f0e3279f640 1 -- 172.21.15.60:0/2077335153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e2c101cf0 msgr2=0x7f0e2c108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.619+0000 7f0e3279f640 1 --2- 172.21.15.60:0/2077335153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e2c101cf0 0x7f0e2c108580 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009ca0 tx=0x7f0e2402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.619+0000 7f0e3279f640 1 -- 172.21.15.60:0/2077335153 shutdown_connections 2024-03-04T12:25:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.619+0000 7f0e3279f640 1 --2- 172.21.15.60:0/2077335153 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e2c101cf0 0x7f0e2c108580 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.619+0000 7f0e3279f640 1 --2- 172.21.15.60:0/2077335153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e2c101330 0x7f0e2c1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.619+0000 7f0e3279f640 1 --2- 172.21.15.60:0/2077335153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e2c100140 0x7f0e2c100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.619+0000 7f0e3279f640 1 -- 172.21.15.60:0/2077335153 >> 172.21.15.60:0/2077335153 conn(0x7f0e2c0fb6b0 msgr2=0x7f0e2c0fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.619+0000 7f0e3279f640 1 -- 172.21.15.60:0/2077335153 shutdown_connections 2024-03-04T12:25:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.619+0000 7f0e3279f640 1 -- 172.21.15.60:0/2077335153 wait complete. 2024-03-04T12:25:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e3279f640 1 Processor -- start 2024-03-04T12:25:41.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e3279f640 1 -- start start 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e3279f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e2c100140 0x7f0e2c069170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e3279f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e2c101330 0x7f0e2c19c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e3279f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e2c101cf0 0x7f0e2c19e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e3279f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c10bdd0 con 0x7f0e2c101330 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e3279f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c10bc50 con 0x7f0e2c101cf0 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e3279f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c10bf50 con 0x7f0e2c100140 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e30f9c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e2c101330 0x7f0e2c19c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e30f9c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e2c101330 0x7f0e2c19c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55868/0 (socket says 172.21.15.60:55868) 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e30f9c640 1 -- 172.21.15.60:0/3687276221 learned_addr learned my addr 172.21.15.60:0/3687276221 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e31f9e640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e2c101cf0 0x7f0e2c19e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.620+0000 7f0e3179d640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e2c100140 0x7f0e2c069170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.621+0000 7f0e30f9c640 1 -- 172.21.15.60:0/3687276221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e2c100140 msgr2=0x7f0e2c069170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.621+0000 7f0e30f9c640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e2c100140 0x7f0e2c069170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.621+0000 7f0e30f9c640 1 -- 172.21.15.60:0/3687276221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e2c101cf0 msgr2=0x7f0e2c19e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.621+0000 7f0e30f9c640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e2c101cf0 0x7f0e2c19e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.621+0000 7f0e30f9c640 1 -- 172.21.15.60:0/3687276221 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c19f110 con 0x7f0e2c101330 2024-03-04T12:25:41.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.621+0000 7f0e30f9c640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e2c101330 0x7f0e2c19c1e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00c920 tx=0x7f0e1c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:41.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.621+0000 7f0e227fc640 1 -- 172.21.15.60:0/3687276221 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c007c70 con 0x7f0e2c101330 2024-03-04T12:25:41.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.621+0000 7f0e3279f640 1 -- 172.21.15.60:0/3687276221 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c19f3a0 con 0x7f0e2c101330 2024-03-04T12:25:41.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.621+0000 7f0e227fc640 1 -- 172.21.15.60:0/3687276221 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e1c007e10 con 0x7f0e2c101330 2024-03-04T12:25:41.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.621+0000 7f0e227fc640 1 -- 172.21.15.60:0/3687276221 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c018890 con 0x7f0e2c101330 2024-03-04T12:25:41.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.621+0000 7f0e3279f640 1 -- 172.21.15.60:0/3687276221 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c19f8f0 con 0x7f0e2c101330 2024-03-04T12:25:41.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.622+0000 7f0e3279f640 1 -- 172.21.15.60:0/3687276221 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2c107390 con 0x7f0e2c101330 2024-03-04T12:25:41.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.623+0000 7f0e227fc640 1 -- 172.21.15.60:0/3687276221 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f0e1c018a50 con 0x7f0e2c101330 2024-03-04T12:25:41.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.624+0000 7f0e227fc640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0e180a1e00 0x7f0e180a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.624+0000 7f0e3179d640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0e180a1e00 0x7f0e180a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.624+0000 7f0e227fc640 1 -- 172.21.15.60:0/3687276221 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f0e1c0ce630 con 0x7f0e2c101330 2024-03-04T12:25:41.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.624+0000 7f0e3179d640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0e180a1e00 0x7f0e180a42c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0e28005fd0 tx=0x7f0e2800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:41.634 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.627+0000 7f0e227fc640 1 -- 172.21.15.60:0/3687276221 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0e1c014030 con 0x7f0e2c101330 2024-03-04T12:25:41.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.848+0000 7f0e3279f640 1 -- 172.21.15.60:0/3687276221 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e2c02d100 con 0x7f0e2c101330 2024-03-04T12:25:41.855 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.848+0000 7f0e227fc640 1 -- 172.21.15.60:0/3687276221 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11373 (secure 0 0 0) 0x7f0e1c09b5a0 con 0x7f0e2c101330 2024-03-04T12:25:41.856 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:41.856 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:25:37.524434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:25:41.856 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 29 2024-03-04T12:25:41.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.850+0000 7f0e3279f640 1 -- 172.21.15.60:0/3687276221 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0e180a1e00 msgr2=0x7f0e180a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.850+0000 7f0e3279f640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0e180a1e00 0x7f0e180a42c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0e28005fd0 tx=0x7f0e2800b390 comp rx=0 tx=0).stop 2024-03-04T12:25:41.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.850+0000 7f0e3279f640 1 -- 172.21.15.60:0/3687276221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e2c101330 msgr2=0x7f0e2c19c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.850+0000 7f0e3279f640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e2c101330 0x7f0e2c19c1e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00c920 tx=0x7f0e1c00cde0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.850+0000 7f0e3279f640 1 -- 172.21.15.60:0/3687276221 shutdown_connections 2024-03-04T12:25:41.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.850+0000 7f0e3279f640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f0e180a1e00 0x7f0e180a42c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.850+0000 7f0e3279f640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e2c101cf0 0x7f0e2c19e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.850+0000 7f0e3279f640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0e2c101330 0x7f0e2c19c1e0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.850+0000 7f0e3279f640 1 --2- 172.21.15.60:0/3687276221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e2c100140 0x7f0e2c069170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.850+0000 7f0e3279f640 1 -- 172.21.15.60:0/3687276221 >> 172.21.15.60:0/3687276221 conn(0x7f0e2c0fb6b0 msgr2=0x7f0e2c104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:41.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.851+0000 7f0e3279f640 1 -- 172.21.15.60:0/3687276221 shutdown_connections 2024-03-04T12:25:41.859 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.851+0000 7f0e3279f640 1 -- 172.21.15.60:0/3687276221 wait complete. 2024-03-04T12:25:41.869 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-04T12:25:41.869 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-04T12:25:41.869 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T12:25:41.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.965+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1570232033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e201025b0 msgr2=0x7f7e20102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.965+0000 7f7e25b7d640 1 --2- 172.21.15.60:0/1570232033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e201025b0 0x7f7e20102a30 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009900 tx=0x7f7e1402bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:41.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.965+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1570232033 shutdown_connections 2024-03-04T12:25:41.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.965+0000 7f7e25b7d640 1 --2- 172.21.15.60:0/1570232033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e20102f70 0x7f7e20109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.965+0000 7f7e25b7d640 1 --2- 172.21.15.60:0/1570232033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e201025b0 0x7f7e20102a30 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.965+0000 7f7e25b7d640 1 --2- 172.21.15.60:0/1570232033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e201013c0 0x7f7e201017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.965+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1570232033 >> 172.21.15.60:0/1570232033 conn(0x7f7e200fc950 msgr2=0x7f7e200fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.965+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1570232033 shutdown_connections 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.965+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1570232033 wait complete. 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.965+0000 7f7e25b7d640 1 Processor -- start 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.965+0000 7f7e25b7d640 1 -- start start 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e25b7d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e201013c0 0x7f7e201975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e25b7d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e201025b0 0x7f7e20197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e25b7d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e20102f70 0x7f7e2019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e25b7d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2010cfe0 con 0x7f7e201013c0 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e25b7d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2010ce60 con 0x7f7e201025b0 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e25b7d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2010d160 con 0x7f7e20102f70 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e201013c0 0x7f7e201975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e201013c0 0x7f7e201975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55876/0 (socket says 172.21.15.60:55876) 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1f7fe640 1 -- 172.21.15.60:0/1368918907 learned_addr learned my addr 172.21.15.60:0/1368918907 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1effd640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e201025b0 0x7f7e20197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1ffff640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e20102f70 0x7f7e2019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1f7fe640 1 -- 172.21.15.60:0/1368918907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e20102f70 msgr2=0x7f7e2019eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1f7fe640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e20102f70 0x7f7e2019eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1f7fe640 1 -- 172.21.15.60:0/1368918907 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e201025b0 msgr2=0x7f7e20197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:41.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1f7fe640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e201025b0 0x7f7e20197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:41.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1f7fe640 1 -- 172.21.15.60:0/1368918907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2019f210 con 0x7f7e201013c0 2024-03-04T12:25:41.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1f7fe640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e201013c0 0x7f7e201975f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000d950 tx=0x7f7e1000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:41.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1cff9640 1 -- 172.21.15.60:0/1368918907 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e10017020 con 0x7f7e201013c0 2024-03-04T12:25:41.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1368918907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2019f500 con 0x7f7e201013c0 2024-03-04T12:25:41.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1cff9640 1 -- 172.21.15.60:0/1368918907 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e10009c30 con 0x7f7e201013c0 2024-03-04T12:25:41.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e1cff9640 1 -- 172.21.15.60:0/1368918907 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1000f740 con 0x7f7e201013c0 2024-03-04T12:25:41.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.966+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1368918907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2019fa20 con 0x7f7e201013c0 2024-03-04T12:25:41.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.967+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1368918907 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e201086e0 con 0x7f7e201013c0 2024-03-04T12:25:41.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.969+0000 7f7e1cff9640 1 -- 172.21.15.60:0/1368918907 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f7e1001a020 con 0x7f7e201013c0 2024-03-04T12:25:41.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.969+0000 7f7e1cff9640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7e080a1e10 0x7f7e080a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:41.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.969+0000 7f7e1effd640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7e080a1e10 0x7f7e080a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:41.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.969+0000 7f7e1cff9640 1 -- 172.21.15.60:0/1368918907 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f7e100ce0c0 con 0x7f7e201013c0 2024-03-04T12:25:41.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.970+0000 7f7e1effd640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7e080a1e10 0x7f7e080a42d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7e14006010 tx=0x7f7e14000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:41.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:41.973+0000 7f7e1cff9640 1 -- 172.21.15.60:0/1368918907 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7e10021070 con 0x7f7e201013c0 2024-03-04T12:25:42.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.154+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1368918907 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e20061770 con 0x7f7e080a1e10 2024-03-04T12:25:42.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.164+0000 7f7e1cff9640 1 -- 172.21.15.60:0/1368918907 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340806 (secure 0 0 0) 0x7f7e20061770 con 0x7f7e080a1e10 2024-03-04T12:25:42.176 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:42.178 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-04T12:25:42.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.168+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1368918907 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7e080a1e10 msgr2=0x7f7e080a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:42.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.168+0000 7f7e25b7d640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7e080a1e10 0x7f7e080a42d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7e14006010 tx=0x7f7e14000bc0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.168+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1368918907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e201013c0 msgr2=0x7f7e201975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:42.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.168+0000 7f7e25b7d640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e201013c0 0x7f7e201975f0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000d950 tx=0x7f7e1000de10 comp rx=0 tx=0).stop 2024-03-04T12:25:42.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.168+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1368918907 shutdown_connections 2024-03-04T12:25:42.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.168+0000 7f7e25b7d640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f7e080a1e10 0x7f7e080a42d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.168+0000 7f7e25b7d640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e20102f70 0x7f7e2019eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.168+0000 7f7e25b7d640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7e201025b0 0x7f7e20197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.168+0000 7f7e25b7d640 1 --2- 172.21.15.60:0/1368918907 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7e201013c0 0x7f7e201975f0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.168+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1368918907 >> 172.21.15.60:0/1368918907 conn(0x7f7e200fc950 msgr2=0x7f7e201058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:42.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.168+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1368918907 shutdown_connections 2024-03-04T12:25:42.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.168+0000 7f7e25b7d640 1 -- 172.21.15.60:0/1368918907 wait complete. 2024-03-04T12:25:42.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:42 smithi060 ceph-mon[32745]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-04T12:25:42.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:42 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2151704167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:42.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:42 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/4113103525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:42.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:42 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3687276221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:42.186 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-03-04T12:25:40.192424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659222,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1352,"num_read_kb":803,"num_write":440,"num_write_kb":3090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":863,"ondisk_log_size":863,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386748,"kb_used_data":12584,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555076,"statfs":{"total":1151940427776,"available":1151544397824,"internally_reserved":0,"allocated":12886016,"data_stored":8188856,"data_compressed":428748,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":57,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003095"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276572+0000","last_change":"2024-03-04T12:23:12.926673+0000","last_active":"2024-03-04T12:25:07.276572+0000","last_peered":"2024-03-04T12:25:07.276572+0000","last_clean":"2024-03-04T12:25:07.276572+0000","last_became_active":"2024-03-04T12:23:12.926299+0000","last_became_peered":"2024-03-04T12:23:12.926299+0000","last_unstale":"2024-03-04T12:25:07.276572+0000","last_undegraded":"2024-03-04T12:25:07.276572+0000","last_fullsized":"2024-03-04T12:25:07.276572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:19.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940339+0000","last_change":"2024-03-04T12:23:12.921758+0000","last_active":"2024-03-04T12:25:06.940339+0000","last_peered":"2024-03-04T12:25:06.940339+0000","last_clean":"2024-03-04T12:25:06.940339+0000","last_became_active":"2024-03-04T12:23:12.921317+0000","last_became_peered":"2024-03-04T12:23:12.921317+0000","last_unstale":"2024-03-04T12:25:06.940339+0000","last_undegraded":"2024-03-04T12:25:06.940339+0000","last_fullsized":"2024-03-04T12:25:06.940339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:20:25.179004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939582+0000","last_change":"2024-03-04T12:23:12.925771+0000","last_active":"2024-03-04T12:25:06.939582+0000","last_peered":"2024-03-04T12:25:06.939582+0000","last_clean":"2024-03-04T12:25:06.939582+0000","last_became_active":"2024-03-04T12:23:12.925447+0000","last_became_peered":"2024-03-04T12:23:12.925447+0000","last_unstale":"2024-03-04T12:25:06.939582+0000","last_undegraded":"2024-03-04T12:25:06.939582+0000","last_fullsized":"2024-03-04T12:25:06.939582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:12.380831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935876+0000","last_change":"2024-03-04T12:24:05.959806+0000","last_active":"2024-03-04T12:25:06.935876+0000","last_peered":"2024-03-04T12:25:06.935876+0000","last_clean":"2024-03-04T12:25:06.935876+0000","last_became_active":"2024-03-04T12:24:05.959557+0000","last_became_peered":"2024-03-04T12:24:05.959557+0000","last_unstale":"2024-03-04T12:25:06.935876+0000","last_undegraded":"2024-03-04T12:25:06.935876+0000","last_fullsized":"2024-03-04T12:25:06.935876+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:51.945517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938061+0000","last_change":"2024-03-04T12:23:12.921118+0000","last_active":"2024-03-04T12:25:06.938061+0000","last_peered":"2024-03-04T12:25:06.938061+0000","last_clean":"2024-03-04T12:25:06.938061+0000","last_became_active":"2024-03-04T12:23:12.920127+0000","last_became_peered":"2024-03-04T12:23:12.920127+0000","last_unstale":"2024-03-04T12:25:06.938061+0000","last_undegraded":"2024-03-04T12:25:06.938061+0000","last_fullsized":"2024-03-04T12:25:06.938061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:22:10.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.025786+0000","last_change":"2024-03-04T12:23:12.916751+0000","last_active":"2024-03-04T12:25:07.025786+0000","last_peered":"2024-03-04T12:25:07.025786+0000","last_clean":"2024-03-04T12:25:07.025786+0000","last_became_active":"2024-03-04T12:23:12.916404+0000","last_became_peered":"2024-03-04T12:23:12.916404+0000","last_unstale":"2024-03-04T12:25:07.025786+0000","last_undegraded":"2024-03-04T12:25:07.025786+0000","last_fullsized":"2024-03-04T12:25:07.025786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:39.998992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036504+0000","last_change":"2024-03-04T12:24:06.029608+0000","last_active":"2024-03-04T12:25:07.036504+0000","last_peered":"2024-03-04T12:25:07.036504+0000","last_clean":"2024-03-04T12:25:07.036504+0000","last_became_active":"2024-03-04T12:24:06.029021+0000","last_became_peered":"2024-03-04T12:24:06.029021+0000","last_unstale":"2024-03-04T12:25:07.036504+0000","last_undegraded":"2024-03-04T12:25:07.036504+0000","last_fullsized":"2024-03-04T12:25:07.036504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:13.140511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938055+0000","last_change":"2024-03-04T12:23:12.917627+0000","last_active":"2024-03-04T12:25:06.938055+0000","last_peered":"2024-03-04T12:25:06.938055+0000","last_clean":"2024-03-04T12:25:06.938055+0000","last_became_active":"2024-03-04T12:23:12.917279+0000","last_became_peered":"2024-03-04T12:23:12.917279+0000","last_unstale":"2024-03-04T12:25:06.938055+0000","last_undegraded":"2024-03-04T12:25:06.938055+0000","last_fullsized":"2024-03-04T12:25:06.938055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:33:03.487273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.937093+0000","last_change":"2024-03-04T12:23:12.922601+0000","last_active":"2024-03-04T12:25:06.937093+0000","last_peered":"2024-03-04T12:25:06.937093+0000","last_clean":"2024-03-04T12:25:06.937093+0000","last_became_active":"2024-03-04T12:23:12.922037+0000","last_became_peered":"2024-03-04T12:23:12.922037+0000","last_unstale":"2024-03-04T12:25:06.937093+0000","last_undegraded":"2024-03-04T12:25:06.937093+0000","last_fullsized":"2024-03-04T12:25:06.937093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:32.076439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.026254+0000","last_change":"2024-03-04T12:23:12.916308+0000","last_active":"2024-03-04T12:25:07.026254+0000","last_peered":"2024-03-04T12:25:07.026254+0000","last_clean":"2024-03-04T12:25:07.026254+0000","last_became_active":"2024-03-04T12:23:12.915897+0000","last_became_peered":"2024-03-04T12:23:12.915897+0000","last_unstale":"2024-03-04T12:25:07.026254+0000","last_undegraded":"2024-03-04T12:25:07.026254+0000","last_fullsized":"2024-03-04T12:25:07.026254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:15:53.133053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037015+0000","last_change":"2024-03-04T12:23:12.930700+0000","last_active":"2024-03-04T12:25:07.037015+0000","last_peered":"2024-03-04T12:25:07.037015+0000","last_clean":"2024-03-04T12:25:07.037015+0000","last_became_active":"2024-03-04T12:23:12.930328+0000","last_became_peered":"2024-03-04T12:23:12.930328+0000","last_unstale":"2024-03-04T12:25:07.037015+0000","last_undegraded":"2024-03-04T12:25:07.037015+0000","last_fullsized":"2024-03-04T12:25:07.037015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:29.990527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933883+0000","last_change":"2024-03-04T12:23:12.915471+0000","last_active":"2024-03-04T12:25:06.933883+0000","last_peered":"2024-03-04T12:25:06.933883+0000","last_clean":"2024-03-04T12:25:06.933883+0000","last_became_active":"2024-03-04T12:23:12.913627+0000","last_became_peered":"2024-03-04T12:23:12.913627+0000","last_unstale":"2024-03-04T12:25:06.933883+0000","last_undegraded":"2024-03-04T12:25:06.933883+0000","last_fullsized":"2024-03-04T12:25:06.933883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:42:56.762782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932392+0000","last_change":"2024-03-04T12:23:12.899143+0000","last_active":"2024-03-04T12:25:06.932392+0000","last_peered":"2024-03-04T12:25:06.932392+0000","last_clean":"2024-03-04T12:25:06.932392+0000","last_became_active":"2024-03-04T12:23:12.898498+0000","last_became_peered":"2024-03-04T12:23:12.898498+0000","last_unstale":"2024-03-04T12:25:06.932392+0000","last_undegraded":"2024-03-04T12:25:06.932392+0000","last_fullsized":"2024-03-04T12:25:06.932392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:33.840683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939467+0000","last_change":"2024-03-04T12:23:12.920873+0000","last_active":"2024-03-04T12:25:06.939467+0000","last_peered":"2024-03-04T12:25:06.939467+0000","last_clean":"2024-03-04T12:25:06.939467+0000","last_became_active":"2024-03-04T12:23:12.919080+0000","last_became_peered":"2024-03-04T12:23:12.919080+0000","last_unstale":"2024-03-04T12:25:06.939467+0000","last_undegraded":"2024-03-04T12:25:06.939467+0000","last_fullsized":"2024-03-04T12:25:06.939467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:47:26.710624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939195+0000","last_change":"2024-03-04T12:24:05.959024+0000","last_active":"2024-03-04T12:25:06.939195+0000","last_peered":"2024-03-04T12:25:06.939195+0000","last_clean":"2024-03-04T12:25:06.939195+0000","last_became_active":"2024-03-04T12:24:05.958750+0000","last_became_peered":"2024-03-04T12:24:05.958750+0000","last_unstale":"2024-03-04T12:25:06.939195+0000","last_undegraded":"2024-03-04T12:25:06.939195+0000","last_fullsized":"2024-03-04T12:25:06.939195+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:59:33.263357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936333+0000","last_change":"2024-03-04T12:23:12.923860+0000","last_active":"2024-03-04T12:25:06.936333+0000","last_peered":"2024-03-04T12:25:06.936333+0000","last_clean":"2024-03-04T12:25:06.936333+0000","last_became_active":"2024-03-04T12:23:12.923530+0000","last_became_peered":"2024-03-04T12:23:12.923530+0000","last_unstale":"2024-03-04T12:25:06.936333+0000","last_undegraded":"2024-03-04T12:25:06.936333+0000","last_fullsized":"2024-03-04T12:25:06.936333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:30:01.466458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276427+0000","last_change":"2024-03-04T12:23:12.927159+0000","last_active":"2024-03-04T12:25:07.276427+0000","last_peered":"2024-03-04T12:25:07.276427+0000","last_clean":"2024-03-04T12:25:07.276427+0000","last_became_active":"2024-03-04T12:23:12.926800+0000","last_became_peered":"2024-03-04T12:23:12.926800+0000","last_unstale":"2024-03-04T12:25:07.276427+0000","last_undegraded":"2024-03-04T12:25:07.276427+0000","last_fullsized":"2024-03-04T12:25:07.276427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:52.633648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936123+0000","last_change":"2024-03-04T12:23:12.920607+0000","last_active":"2024-03-04T12:25:06.936123+0000","last_peered":"2024-03-04T12:25:06.936123+0000","last_clean":"2024-03-04T12:25:06.936123+0000","last_became_active":"2024-03-04T12:23:12.920257+0000","last_became_peered":"2024-03-04T12:23:12.920257+0000","last_unstale":"2024-03-04T12:25:06.936123+0000","last_undegraded":"2024-03-04T12:25:06.936123+0000","last_fullsized":"2024-03-04T12:25:06.936123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:39.353673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939723+0000","last_change":"2024-03-04T12:23:12.906635+0000","last_active":"2024-03-04T12:25:06.939723+0000","last_peered":"2024-03-04T12:25:06.939723+0000","last_clean":"2024-03-04T12:25:06.939723+0000","last_became_active":"2024-03-04T12:23:12.906243+0000","last_became_peered":"2024-03-04T12:23:12.906243+0000","last_unstale":"2024-03-04T12:25:06.939723+0000","last_undegraded":"2024-03-04T12:25:06.939723+0000","last_fullsized":"2024-03-04T12:25:06.939723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:44.375934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939135+0000","last_change":"2024-03-04T12:23:16.951685+0000","last_active":"2024-03-04T12:25:06.939135+0000","last_peered":"2024-03-04T12:25:06.939135+0000","last_clean":"2024-03-04T12:25:06.939135+0000","last_became_active":"2024-03-04T12:23:16.951329+0000","last_became_peered":"2024-03-04T12:23:16.951329+0000","last_unstale":"2024-03-04T12:25:06.939135+0000","last_undegraded":"2024-03-04T12:25:06.939135+0000","last_fullsized":"2024-03-04T12:25:06.939135+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:03:27.233466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":160,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:34.071338+0000","last_change":"2024-03-04T12:23:11.900681+0000","last_active":"2024-03-04T12:25:34.071338+0000","last_peered":"2024-03-04T12:25:34.071338+0000","last_clean":"2024-03-04T12:25:34.071338+0000","last_became_active":"2024-03-04T12:23:11.900365+0000","last_became_peered":"2024-03-04T12:23:11.900365+0000","last_unstale":"2024-03-04T12:25:34.071338+0000","last_undegraded":"2024-03-04T12:25:34.071338+0000","last_fullsized":"2024-03-04T12:25:34.071338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:53.052513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":115,"num_read_kb":115,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936030+0000","last_change":"2024-03-04T12:24:06.030707+0000","last_active":"2024-03-04T12:25:06.936030+0000","last_peered":"2024-03-04T12:25:06.936030+0000","last_clean":"2024-03-04T12:25:06.936030+0000","last_became_active":"2024-03-04T12:24:06.030306+0000","last_became_peered":"2024-03-04T12:24:06.030306+0000","last_unstale":"2024-03-04T12:25:06.936030+0000","last_undegraded":"2024-03-04T12:25:06.936030+0000","last_fullsized":"2024-03-04T12:25:06.936030+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:59:18.939734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933263+0000","last_change":"2024-03-04T12:23:16.953704+0000","last_active":"2024-03-04T12:25:06.933263+0000","last_peered":"2024-03-04T12:25:06.933263+0000","last_clean":"2024-03-04T12:25:06.933263+0000","last_became_active":"2024-03-04T12:23:16.953177+0000","last_became_peered":"2024-03-04T12:23:16.953177+0000","last_unstale":"2024-03-04T12:25:06.933263+0000","last_undegraded":"2024-03-04T12:25:06.933263+0000","last_fullsized":"2024-03-04T12:25:06.933263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:37:00.469767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939293+0000","last_change":"2024-03-04T12:23:11.904271+0000","last_active":"2024-03-04T12:25:06.939293+0000","last_peered":"2024-03-04T12:25:06.939293+0000","last_clean":"2024-03-04T12:25:06.939293+0000","last_became_active":"2024-03-04T12:23:11.903963+0000","last_became_peered":"2024-03-04T12:23:11.903963+0000","last_unstale":"2024-03-04T12:25:06.939293+0000","last_undegraded":"2024-03-04T12:25:06.939293+0000","last_fullsized":"2024-03-04T12:25:06.939293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:44.774621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939180+0000","last_change":"2024-03-04T12:23:12.925750+0000","last_active":"2024-03-04T12:25:06.939180+0000","last_peered":"2024-03-04T12:25:06.939180+0000","last_clean":"2024-03-04T12:25:06.939180+0000","last_became_active":"2024-03-04T12:23:12.925380+0000","last_became_peered":"2024-03-04T12:23:12.925380+0000","last_unstale":"2024-03-04T12:25:06.939180+0000","last_undegraded":"2024-03-04T12:25:06.939180+0000","last_fullsized":"2024-03-04T12:25:06.939180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:14.665127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933357+0000","last_change":"2024-03-04T12:23:16.952303+0000","last_active":"2024-03-04T12:25:06.933357+0000","last_peered":"2024-03-04T12:25:06.933357+0000","last_clean":"2024-03-04T12:25:06.933357+0000","last_became_active":"2024-03-04T12:23:16.951666+0000","last_became_peered":"2024-03-04T12:23:16.951666+0000","last_unstale":"2024-03-04T12:25:06.933357+0000","last_undegraded":"2024-03-04T12:25:06.933357+0000","last_fullsized":"2024-03-04T12:25:06.933357+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:41.326808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938665+0000","last_change":"2024-03-04T12:23:11.903871+0000","last_active":"2024-03-04T12:25:06.938665+0000","last_peered":"2024-03-04T12:25:06.938665+0000","last_clean":"2024-03-04T12:25:06.938665+0000","last_became_active":"2024-03-04T12:23:11.902735+0000","last_became_peered":"2024-03-04T12:23:11.902735+0000","last_unstale":"2024-03-04T12:25:06.938665+0000","last_undegraded":"2024-03-04T12:25:06.938665+0000","last_fullsized":"2024-03-04T12:25:06.938665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:45:09.897740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.937164+0000","last_change":"2024-03-04T12:23:12.924504+0000","last_active":"2024-03-04T12:25:06.937164+0000","last_peered":"2024-03-04T12:25:06.937164+0000","last_clean":"2024-03-04T12:25:06.937164+0000","last_became_active":"2024-03-04T12:23:12.922934+0000","last_became_peered":"2024-03-04T12:23:12.922934+0000","last_unstale":"2024-03-04T12:25:06.937164+0000","last_undegraded":"2024-03-04T12:25:06.937164+0000","last_fullsized":"2024-03-04T12:25:06.937164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:28.974601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933262+0000","last_change":"2024-03-04T12:23:16.952324+0000","last_active":"2024-03-04T12:25:06.933262+0000","last_peered":"2024-03-04T12:25:06.933262+0000","last_clean":"2024-03-04T12:25:06.933262+0000","last_became_active":"2024-03-04T12:23:16.951145+0000","last_became_peered":"2024-03-04T12:23:16.951145+0000","last_unstale":"2024-03-04T12:25:06.933262+0000","last_undegraded":"2024-03-04T12:25:06.933262+0000","last_fullsized":"2024-03-04T12:25:06.933262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:20:47.029523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938577+0000","last_change":"2024-03-04T12:23:11.904391+0000","last_active":"2024-03-04T12:25:06.938577+0000","last_peered":"2024-03-04T12:25:06.938577+0000","last_clean":"2024-03-04T12:25:06.938577+0000","last_became_active":"2024-03-04T12:23:11.903244+0000","last_became_peered":"2024-03-04T12:23:11.903244+0000","last_unstale":"2024-03-04T12:25:06.938577+0000","last_undegraded":"2024-03-04T12:25:06.938577+0000","last_fullsized":"2024-03-04T12:25:06.938577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:27.590418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933368+0000","last_change":"2024-03-04T12:23:12.925623+0000","last_active":"2024-03-04T12:25:06.933368+0000","last_peered":"2024-03-04T12:25:06.933368+0000","last_clean":"2024-03-04T12:25:06.933368+0000","last_became_active":"2024-03-04T12:23:12.925243+0000","last_became_peered":"2024-03-04T12:23:12.925243+0000","last_unstale":"2024-03-04T12:25:06.933368+0000","last_undegraded":"2024-03-04T12:25:06.933368+0000","last_fullsized":"2024-03-04T12:25:06.933368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:11:04.142655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939469+0000","last_change":"2024-03-04T12:23:16.946064+0000","last_active":"2024-03-04T12:25:06.939469+0000","last_peered":"2024-03-04T12:25:06.939469+0000","last_clean":"2024-03-04T12:25:06.939469+0000","last_became_active":"2024-03-04T12:23:16.945342+0000","last_became_peered":"2024-03-04T12:23:16.945342+0000","last_unstale":"2024-03-04T12:25:06.939469+0000","last_undegraded":"2024-03-04T12:25:06.939469+0000","last_fullsized":"2024-03-04T12:25:06.939469+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:43:37.500216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:37.071102+0000","last_change":"2024-03-04T12:23:11.899478+0000","last_active":"2024-03-04T12:25:37.071102+0000","last_peered":"2024-03-04T12:25:37.071102+0000","last_clean":"2024-03-04T12:25:37.071102+0000","last_became_active":"2024-03-04T12:23:11.899153+0000","last_became_peered":"2024-03-04T12:23:11.899153+0000","last_unstale":"2024-03-04T12:25:37.071102+0000","last_undegraded":"2024-03-04T12:25:37.071102+0000","last_fullsized":"2024-03-04T12:25:37.071102+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:10.709054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938359+0000","last_change":"2024-03-04T12:23:12.908560+0000","last_active":"2024-03-04T12:25:06.938359+0000","last_peered":"2024-03-04T12:25:06.938359+0000","last_clean":"2024-03-04T12:25:06.938359+0000","last_became_active":"2024-03-04T12:23:12.908221+0000","last_became_peered":"2024-03-04T12:23:12.908221+0000","last_unstale":"2024-03-04T12:25:06.938359+0000","last_undegraded":"2024-03-04T12:25:06.938359+0000","last_fullsized":"2024-03-04T12:25:06.938359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:28:06.545945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939489+0000","last_change":"2024-03-04T12:24:06.031257+0000","last_active":"2024-03-04T12:25:06.939489+0000","last_peered":"2024-03-04T12:25:06.939489+0000","last_clean":"2024-03-04T12:25:06.939489+0000","last_became_active":"2024-03-04T12:24:06.030409+0000","last_became_peered":"2024-03-04T12:24:06.030409+0000","last_unstale":"2024-03-04T12:25:06.939489+0000","last_undegraded":"2024-03-04T12:25:06.939489+0000","last_fullsized":"2024-03-04T12:25:06.939489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:31.403224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.025691+0000","last_change":"2024-03-04T12:23:11.898360+0000","last_active":"2024-03-04T12:25:07.025691+0000","last_peered":"2024-03-04T12:25:07.025691+0000","last_clean":"2024-03-04T12:25:07.025691+0000","last_became_active":"2024-03-04T12:23:11.897988+0000","last_became_peered":"2024-03-04T12:23:11.897988+0000","last_unstale":"2024-03-04T12:25:07.025691+0000","last_undegraded":"2024-03-04T12:25:07.025691+0000","last_fullsized":"2024-03-04T12:25:07.025691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:17.448032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933598+0000","last_change":"2024-03-04T12:23:12.918392+0000","last_active":"2024-03-04T12:25:06.933598+0000","last_peered":"2024-03-04T12:25:06.933598+0000","last_clean":"2024-03-04T12:25:06.933598+0000","last_became_active":"2024-03-04T12:23:12.917656+0000","last_became_peered":"2024-03-04T12:23:12.917656+0000","last_unstale":"2024-03-04T12:25:06.933598+0000","last_undegraded":"2024-03-04T12:25:06.933598+0000","last_fullsized":"2024-03-04T12:25:06.933598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:44:14.545925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276449+0000","last_change":"2024-03-04T12:23:16.941841+0000","last_active":"2024-03-04T12:25:07.276449+0000","last_peered":"2024-03-04T12:25:07.276449+0000","last_clean":"2024-03-04T12:25:07.276449+0000","last_became_active":"2024-03-04T12:23:16.941449+0000","last_became_peered":"2024-03-04T12:23:16.941449+0000","last_unstale":"2024-03-04T12:25:07.276449+0000","last_undegraded":"2024-03-04T12:25:07.276449+0000","last_fullsized":"2024-03-04T12:25:07.276449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:50:35.451834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932169+0000","last_change":"2024-03-04T12:23:11.906723+0000","last_active":"2024-03-04T12:25:06.932169+0000","last_peered":"2024-03-04T12:25:06.932169+0000","last_clean":"2024-03-04T12:25:06.932169+0000","last_became_active":"2024-03-04T12:23:11.906357+0000","last_became_peered":"2024-03-04T12:23:11.906357+0000","last_unstale":"2024-03-04T12:25:06.932169+0000","last_undegraded":"2024-03-04T12:25:06.932169+0000","last_fullsized":"2024-03-04T12:25:06.932169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:09.592236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938664+0000","last_change":"2024-03-04T12:23:12.916435+0000","last_active":"2024-03-04T12:25:06.938664+0000","last_peered":"2024-03-04T12:25:06.938664+0000","last_clean":"2024-03-04T12:25:06.938664+0000","last_became_active":"2024-03-04T12:23:12.915607+0000","last_became_peered":"2024-03-04T12:23:12.915607+0000","last_unstale":"2024-03-04T12:25:06.938664+0000","last_undegraded":"2024-03-04T12:25:06.938664+0000","last_fullsized":"2024-03-04T12:25:06.938664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:41:32.900344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940234+0000","last_change":"2024-03-04T12:23:16.944342+0000","last_active":"2024-03-04T12:25:06.940234+0000","last_peered":"2024-03-04T12:25:06.940234+0000","last_clean":"2024-03-04T12:25:06.940234+0000","last_became_active":"2024-03-04T12:23:16.943683+0000","last_became_peered":"2024-03-04T12:23:16.943683+0000","last_unstale":"2024-03-04T12:25:06.940234+0000","last_undegraded":"2024-03-04T12:25:06.940234+0000","last_fullsized":"2024-03-04T12:25:06.940234+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:19.658860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933018+0000","last_change":"2024-03-04T12:23:11.899194+0000","last_active":"2024-03-04T12:25:06.933018+0000","last_peered":"2024-03-04T12:25:06.933018+0000","last_clean":"2024-03-04T12:25:06.933018+0000","last_became_active":"2024-03-04T12:23:11.898109+0000","last_became_peered":"2024-03-04T12:23:11.898109+0000","last_unstale":"2024-03-04T12:25:06.933018+0000","last_undegraded":"2024-03-04T12:25:06.933018+0000","last_fullsized":"2024-03-04T12:25:06.933018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:08.808202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935567+0000","last_change":"2024-03-04T12:23:12.916549+0000","last_active":"2024-03-04T12:25:06.935567+0000","last_peered":"2024-03-04T12:25:06.935567+0000","last_clean":"2024-03-04T12:25:06.935567+0000","last_became_active":"2024-03-04T12:23:12.916204+0000","last_became_peered":"2024-03-04T12:23:12.916204+0000","last_unstale":"2024-03-04T12:25:06.935567+0000","last_undegraded":"2024-03-04T12:25:06.935567+0000","last_fullsized":"2024-03-04T12:25:06.935567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:56.412600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938437+0000","last_change":"2024-03-04T12:23:16.945903+0000","last_active":"2024-03-04T12:25:06.938437+0000","last_peered":"2024-03-04T12:25:06.938437+0000","last_clean":"2024-03-04T12:25:06.938437+0000","last_became_active":"2024-03-04T12:23:16.945535+0000","last_became_peered":"2024-03-04T12:23:16.945535+0000","last_unstale":"2024-03-04T12:25:06.938437+0000","last_undegraded":"2024-03-04T12:25:06.938437+0000","last_fullsized":"2024-03-04T12:25:06.938437+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:45:01.178370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'6","reported_seq":272,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:37.082131+0000","last_change":"2024-03-04T12:23:11.901570+0000","last_active":"2024-03-04T12:25:37.082131+0000","last_peered":"2024-03-04T12:25:37.082131+0000","last_clean":"2024-03-04T12:25:37.082131+0000","last_became_active":"2024-03-04T12:23:11.900702+0000","last_became_peered":"2024-03-04T12:23:11.900702+0000","last_unstale":"2024-03-04T12:25:37.082131+0000","last_undegraded":"2024-03-04T12:25:37.082131+0000","last_fullsized":"2024-03-04T12:25:37.082131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:18.758453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936880+0000","last_change":"2024-03-04T12:23:12.919571+0000","last_active":"2024-03-04T12:25:06.936880+0000","last_peered":"2024-03-04T12:25:06.936880+0000","last_clean":"2024-03-04T12:25:06.936880+0000","last_became_active":"2024-03-04T12:23:12.919143+0000","last_became_peered":"2024-03-04T12:23:12.919143+0000","last_unstale":"2024-03-04T12:25:06.936880+0000","last_undegraded":"2024-03-04T12:25:06.936880+0000","last_fullsized":"2024-03-04T12:25:06.936880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:25.459417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935505+0000","last_change":"2024-03-04T12:23:16.946380+0000","last_active":"2024-03-04T12:25:06.935505+0000","last_peered":"2024-03-04T12:25:06.935505+0000","last_clean":"2024-03-04T12:25:06.935505+0000","last_became_active":"2024-03-04T12:23:16.945983+0000","last_became_peered":"2024-03-04T12:23:16.945983+0000","last_unstale":"2024-03-04T12:25:06.935505+0000","last_undegraded":"2024-03-04T12:25:06.935505+0000","last_fullsized":"2024-03-04T12:25:06.935505+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:39:43.297619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036989+0000","last_change":"2024-03-04T12:23:11.897176+0000","last_active":"2024-03-04T12:25:07.036989+0000","last_peered":"2024-03-04T12:25:07.036989+0000","last_clean":"2024-03-04T12:25:07.036989+0000","last_became_active":"2024-03-04T12:23:11.896747+0000","last_became_peered":"2024-03-04T12:23:11.896747+0000","last_unstale":"2024-03-04T12:25:07.036989+0000","last_undegraded":"2024-03-04T12:25:07.036989+0000","last_fullsized":"2024-03-04T12:25:07.036989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:02.206704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939064+0000","last_change":"2024-03-04T12:23:12.922021+0000","last_active":"2024-03-04T12:25:06.939064+0000","last_peered":"2024-03-04T12:25:06.939064+0000","last_clean":"2024-03-04T12:25:06.939064+0000","last_became_active":"2024-03-04T12:23:12.918724+0000","last_became_peered":"2024-03-04T12:23:12.918724+0000","last_unstale":"2024-03-04T12:25:06.939064+0000","last_undegraded":"2024-03-04T12:25:06.939064+0000","last_fullsized":"2024-03-04T12:25:06.939064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:49:09.017979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.934143+0000","last_change":"2024-03-04T12:23:16.943271+0000","last_active":"2024-03-04T12:25:06.934143+0000","last_peered":"2024-03-04T12:25:06.934143+0000","last_clean":"2024-03-04T12:25:06.934143+0000","last_became_active":"2024-03-04T12:23:16.942400+0000","last_became_peered":"2024-03-04T12:23:16.942400+0000","last_unstale":"2024-03-04T12:25:06.934143+0000","last_undegraded":"2024-03-04T12:25:06.934143+0000","last_fullsized":"2024-03-04T12:25:06.934143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:06.010960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933398+0000","last_change":"2024-03-04T12:23:11.896706+0000","last_active":"2024-03-04T12:25:06.933398+0000","last_peered":"2024-03-04T12:25:06.933398+0000","last_clean":"2024-03-04T12:25:06.933398+0000","last_became_active":"2024-03-04T12:23:11.896100+0000","last_became_peered":"2024-03-04T12:23:11.896100+0000","last_unstale":"2024-03-04T12:25:06.933398+0000","last_undegraded":"2024-03-04T12:25:06.933398+0000","last_fullsized":"2024-03-04T12:25:06.933398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:42:19.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.937310+0000","last_change":"2024-03-04T12:23:12.923631+0000","last_active":"2024-03-04T12:25:06.937310+0000","last_peered":"2024-03-04T12:25:06.937310+0000","last_clean":"2024-03-04T12:25:06.937310+0000","last_became_active":"2024-03-04T12:23:12.922487+0000","last_became_peered":"2024-03-04T12:23:12.922487+0000","last_unstale":"2024-03-04T12:25:06.937310+0000","last_undegraded":"2024-03-04T12:25:06.937310+0000","last_fullsized":"2024-03-04T12:25:06.937310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:30:33.923064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.025491+0000","last_change":"2024-03-04T12:24:06.032075+0000","last_active":"2024-03-04T12:25:07.025491+0000","last_peered":"2024-03-04T12:25:07.025491+0000","last_clean":"2024-03-04T12:25:07.025491+0000","last_became_active":"2024-03-04T12:24:06.031639+0000","last_became_peered":"2024-03-04T12:24:06.031639+0000","last_unstale":"2024-03-04T12:25:07.025491+0000","last_undegraded":"2024-03-04T12:25:07.025491+0000","last_fullsized":"2024-03-04T12:25:07.025491+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:44:48.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'39","reported_seq":316,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:37.070833+0000","last_change":"2024-03-04T12:23:11.898903+0000","last_active":"2024-03-04T12:25:37.070833+0000","last_peered":"2024-03-04T12:25:37.070833+0000","last_clean":"2024-03-04T12:25:37.070833+0000","last_became_active":"2024-03-04T12:23:11.898477+0000","last_became_peered":"2024-03-04T12:23:11.898477+0000","last_unstale":"2024-03-04T12:25:37.070833+0000","last_undegraded":"2024-03-04T12:25:37.070833+0000","last_fullsized":"2024-03-04T12:25:37.070833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:36:58.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60826,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":245,"num_read_kb":182,"num_write":41,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938545+0000","last_change":"2024-03-04T12:23:12.908497+0000","last_active":"2024-03-04T12:25:06.938545+0000","last_peered":"2024-03-04T12:25:06.938545+0000","last_clean":"2024-03-04T12:25:06.938545+0000","last_became_active":"2024-03-04T12:23:12.907847+0000","last_became_peered":"2024-03-04T12:23:12.907847+0000","last_unstale":"2024-03-04T12:25:06.938545+0000","last_undegraded":"2024-03-04T12:25:06.938545+0000","last_fullsized":"2024-03-04T12:25:06.938545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:51:44.592851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933682+0000","last_change":"2024-03-04T12:23:16.951844+0000","last_active":"2024-03-04T12:25:06.933682+0000","last_peered":"2024-03-04T12:25:06.933682+0000","last_clean":"2024-03-04T12:25:06.933682+0000","last_became_active":"2024-03-04T12:23:16.951491+0000","last_became_peered":"2024-03-04T12:23:16.951491+0000","last_unstale":"2024-03-04T12:25:06.933682+0000","last_undegraded":"2024-03-04T12:25:06.933682+0000","last_fullsized":"2024-03-04T12:25:06.933682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:54.098239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939315+0000","last_change":"2024-03-04T12:23:11.899592+0000","last_active":"2024-03-04T12:25:06.939315+0000","last_peered":"2024-03-04T12:25:06.939315+0000","last_clean":"2024-03-04T12:25:06.939315+0000","last_became_active":"2024-03-04T12:23:11.899275+0000","last_became_peered":"2024-03-04T12:23:11.899275+0000","last_unstale":"2024-03-04T12:25:06.939315+0000","last_undegraded":"2024-03-04T12:25:06.939315+0000","last_fullsized":"2024-03-04T12:25:06.939315+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:02.184268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932222+0000","last_change":"2024-03-04T12:23:12.910084+0000","last_active":"2024-03-04T12:25:06.932222+0000","last_peered":"2024-03-04T12:25:06.932222+0000","last_clean":"2024-03-04T12:25:06.932222+0000","last_became_active":"2024-03-04T12:23:12.909719+0000","last_became_peered":"2024-03-04T12:23:12.909719+0000","last_unstale":"2024-03-04T12:25:06.932222+0000","last_undegraded":"2024-03-04T12:25:06.932222+0000","last_fullsized":"2024-03-04T12:25:06.932222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:16:41.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276279+0000","last_change":"2024-03-04T12:24:06.029579+0000","last_active":"2024-03-04T12:25:07.276279+0000","last_peered":"2024-03-04T12:25:07.276279+0000","last_clean":"2024-03-04T12:25:07.276279+0000","last_became_active":"2024-03-04T12:24:06.029042+0000","last_became_peered":"2024-03-04T12:24:06.029042+0000","last_unstale":"2024-03-04T12:25:07.276279+0000","last_undegraded":"2024-03-04T12:25:07.276279+0000","last_fullsized":"2024-03-04T12:25:07.276279+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:59:35.867212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933484+0000","last_change":"2024-03-04T12:23:11.897506+0000","last_active":"2024-03-04T12:25:06.933484+0000","last_peered":"2024-03-04T12:25:06.933484+0000","last_clean":"2024-03-04T12:25:06.933484+0000","last_became_active":"2024-03-04T12:23:11.897090+0000","last_became_peered":"2024-03-04T12:23:11.897090+0000","last_unstale":"2024-03-04T12:25:06.933484+0000","last_undegraded":"2024-03-04T12:25:06.933484+0000","last_fullsized":"2024-03-04T12:25:06.933484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:10:27.868713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037179+0000","last_change":"2024-03-04T12:23:12.921825+0000","last_active":"2024-03-04T12:25:07.037179+0000","last_peered":"2024-03-04T12:25:07.037179+0000","last_clean":"2024-03-04T12:25:07.037179+0000","last_became_active":"2024-03-04T12:23:12.921491+0000","last_became_peered":"2024-03-04T12:23:12.921491+0000","last_unstale":"2024-03-04T12:25:07.037179+0000","last_undegraded":"2024-03-04T12:25:07.037179+0000","last_fullsized":"2024-03-04T12:25:07.037179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:17:08.203502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935989+0000","last_change":"2024-03-04T12:23:16.951920+0000","last_active":"2024-03-04T12:25:06.935989+0000","last_peered":"2024-03-04T12:25:06.935989+0000","last_clean":"2024-03-04T12:25:06.935989+0000","last_became_active":"2024-03-04T12:23:16.951507+0000","last_became_peered":"2024-03-04T12:23:16.951507+0000","last_unstale":"2024-03-04T12:25:06.935989+0000","last_undegraded":"2024-03-04T12:25:06.935989+0000","last_fullsized":"2024-03-04T12:25:06.935989+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:27.673692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":161,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:36.068598+0000","last_change":"2024-03-04T12:23:11.897722+0000","last_active":"2024-03-04T12:25:36.068598+0000","last_peered":"2024-03-04T12:25:36.068598+0000","last_clean":"2024-03-04T12:25:36.068598+0000","last_became_active":"2024-03-04T12:23:11.897276+0000","last_became_peered":"2024-03-04T12:23:11.897276+0000","last_unstale":"2024-03-04T12:25:36.068598+0000","last_undegraded":"2024-03-04T12:25:36.068598+0000","last_fullsized":"2024-03-04T12:25:36.068598+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:37.968804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036780+0000","last_change":"2024-03-04T12:23:12.914773+0000","last_active":"2024-03-04T12:25:07.036780+0000","last_peered":"2024-03-04T12:25:07.036780+0000","last_clean":"2024-03-04T12:25:07.036780+0000","last_became_active":"2024-03-04T12:23:12.914431+0000","last_became_peered":"2024-03-04T12:23:12.914431+0000","last_unstale":"2024-03-04T12:25:07.036780+0000","last_undegraded":"2024-03-04T12:25:07.036780+0000","last_fullsized":"2024-03-04T12:25:07.036780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:50.234491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932697+0000","last_change":"2024-03-04T12:23:16.952405+0000","last_active":"2024-03-04T12:25:06.932697+0000","last_peered":"2024-03-04T12:25:06.932697+0000","last_clean":"2024-03-04T12:25:06.932697+0000","last_became_active":"2024-03-04T12:23:16.951915+0000","last_became_peered":"2024-03-04T12:23:16.951915+0000","last_unstale":"2024-03-04T12:25:06.932697+0000","last_undegraded":"2024-03-04T12:25:06.932697+0000","last_fullsized":"2024-03-04T12:25:06.932697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:38:03.765883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938109+0000","last_change":"2024-03-04T12:23:11.903267+0000","last_active":"2024-03-04T12:25:06.938109+0000","last_peered":"2024-03-04T12:25:06.938109+0000","last_clean":"2024-03-04T12:25:06.938109+0000","last_became_active":"2024-03-04T12:23:11.902870+0000","last_became_peered":"2024-03-04T12:23:11.902870+0000","last_unstale":"2024-03-04T12:25:06.938109+0000","last_undegraded":"2024-03-04T12:25:06.938109+0000","last_fullsized":"2024-03-04T12:25:06.938109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:33:43.679869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933267+0000","last_change":"2024-03-04T12:23:12.925303+0000","last_active":"2024-03-04T12:25:06.933267+0000","last_peered":"2024-03-04T12:25:06.933267+0000","last_clean":"2024-03-04T12:25:06.933267+0000","last_became_active":"2024-03-04T12:23:12.924945+0000","last_became_peered":"2024-03-04T12:23:12.924945+0000","last_unstale":"2024-03-04T12:25:06.933267+0000","last_undegraded":"2024-03-04T12:25:06.933267+0000","last_fullsized":"2024-03-04T12:25:06.933267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:49.680925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933148+0000","last_change":"2024-03-04T12:23:16.951457+0000","last_active":"2024-03-04T12:25:06.933148+0000","last_peered":"2024-03-04T12:25:06.933148+0000","last_clean":"2024-03-04T12:25:06.933148+0000","last_became_active":"2024-03-04T12:23:16.950835+0000","last_became_peered":"2024-03-04T12:23:16.950835+0000","last_unstale":"2024-03-04T12:25:06.933148+0000","last_undegraded":"2024-03-04T12:25:06.933148+0000","last_fullsized":"2024-03-04T12:25:06.933148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:39.337275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:29.397242+0000","last_change":"2024-03-04T12:23:11.903718+0000","last_active":"2024-03-04T12:25:29.397242+0000","last_peered":"2024-03-04T12:25:29.397242+0000","last_clean":"2024-03-04T12:25:29.397242+0000","last_became_active":"2024-03-04T12:23:11.902814+0000","last_became_peered":"2024-03-04T12:23:11.902814+0000","last_unstale":"2024-03-04T12:25:29.397242+0000","last_undegraded":"2024-03-04T12:25:29.397242+0000","last_fullsized":"2024-03-04T12:25:29.397242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:40:30.978576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276785+0000","last_change":"2024-03-04T12:24:05.958392+0000","last_active":"2024-03-04T12:25:07.276785+0000","last_peered":"2024-03-04T12:25:07.276785+0000","last_clean":"2024-03-04T12:25:07.276785+0000","last_became_active":"2024-03-04T12:24:05.958012+0000","last_became_peered":"2024-03-04T12:24:05.958012+0000","last_unstale":"2024-03-04T12:25:07.276785+0000","last_undegraded":"2024-03-04T12:25:07.276785+0000","last_fullsized":"2024-03-04T12:25:07.276785+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:43:50.400090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938459+0000","last_change":"2024-03-04T12:23:16.950046+0000","last_active":"2024-03-04T12:25:06.938459+0000","last_peered":"2024-03-04T12:25:06.938459+0000","last_clean":"2024-03-04T12:25:06.938459+0000","last_became_active":"2024-03-04T12:23:16.949632+0000","last_became_peered":"2024-03-04T12:23:16.949632+0000","last_unstale":"2024-03-04T12:25:06.938459+0000","last_undegraded":"2024-03-04T12:25:06.938459+0000","last_fullsized":"2024-03-04T12:25:06.938459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:25.817212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935925+0000","last_change":"2024-03-04T12:23:11.897016+0000","last_active":"2024-03-04T12:25:06.935925+0000","last_peered":"2024-03-04T12:25:06.935925+0000","last_clean":"2024-03-04T12:25:06.935925+0000","last_became_active":"2024-03-04T12:23:11.896668+0000","last_became_peered":"2024-03-04T12:23:11.896668+0000","last_unstale":"2024-03-04T12:25:06.935925+0000","last_undegraded":"2024-03-04T12:25:06.935925+0000","last_fullsized":"2024-03-04T12:25:06.935925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:50:01.089658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.026051+0000","last_change":"2024-03-04T12:23:12.908169+0000","last_active":"2024-03-04T12:25:07.026051+0000","last_peered":"2024-03-04T12:25:07.026051+0000","last_clean":"2024-03-04T12:25:07.026051+0000","last_became_active":"2024-03-04T12:23:12.907786+0000","last_became_peered":"2024-03-04T12:23:12.907786+0000","last_unstale":"2024-03-04T12:25:07.026051+0000","last_undegraded":"2024-03-04T12:25:07.026051+0000","last_fullsized":"2024-03-04T12:25:07.026051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:10.602216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932425+0000","last_change":"2024-03-04T12:23:16.944232+0000","last_active":"2024-03-04T12:25:06.932425+0000","last_peered":"2024-03-04T12:25:06.932425+0000","last_clean":"2024-03-04T12:25:06.932425+0000","last_became_active":"2024-03-04T12:23:16.943855+0000","last_became_peered":"2024-03-04T12:23:16.943855+0000","last_unstale":"2024-03-04T12:25:06.932425+0000","last_undegraded":"2024-03-04T12:25:06.932425+0000","last_fullsized":"2024-03-04T12:25:06.932425+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:37:12.083622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":155,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:37.075695+0000","last_change":"2024-03-04T12:23:11.899742+0000","last_active":"2024-03-04T12:25:37.075695+0000","last_peered":"2024-03-04T12:25:37.075695+0000","last_clean":"2024-03-04T12:25:37.075695+0000","last_became_active":"2024-03-04T12:23:11.898445+0000","last_became_peered":"2024-03-04T12:23:11.898445+0000","last_unstale":"2024-03-04T12:25:37.075695+0000","last_undegraded":"2024-03-04T12:25:37.075695+0000","last_fullsized":"2024-03-04T12:25:37.075695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:13.128348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.025878+0000","last_change":"2024-03-04T12:23:12.921321+0000","last_active":"2024-03-04T12:25:07.025878+0000","last_peered":"2024-03-04T12:25:07.025878+0000","last_clean":"2024-03-04T12:25:07.025878+0000","last_became_active":"2024-03-04T12:23:12.918119+0000","last_became_peered":"2024-03-04T12:23:12.918119+0000","last_unstale":"2024-03-04T12:25:07.025878+0000","last_undegraded":"2024-03-04T12:25:07.025878+0000","last_fullsized":"2024-03-04T12:25:07.025878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:31.693583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932634+0000","last_change":"2024-03-04T12:23:16.943887+0000","last_active":"2024-03-04T12:25:06.932634+0000","last_peered":"2024-03-04T12:25:06.932634+0000","last_clean":"2024-03-04T12:25:06.932634+0000","last_became_active":"2024-03-04T12:23:16.943487+0000","last_became_peered":"2024-03-04T12:23:16.943487+0000","last_unstale":"2024-03-04T12:25:06.932634+0000","last_undegraded":"2024-03-04T12:25:06.932634+0000","last_fullsized":"2024-03-04T12:25:06.932634+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:12.451176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933359+0000","last_change":"2024-03-04T12:23:12.918363+0000","last_active":"2024-03-04T12:25:06.933359+0000","last_peered":"2024-03-04T12:25:06.933359+0000","last_clean":"2024-03-04T12:25:06.933359+0000","last_became_active":"2024-03-04T12:23:12.917563+0000","last_became_peered":"2024-03-04T12:23:12.917563+0000","last_unstale":"2024-03-04T12:25:06.933359+0000","last_undegraded":"2024-03-04T12:25:06.933359+0000","last_fullsized":"2024-03-04T12:25:06.933359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:26:35.297513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'7","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:38.071274+0000","last_change":"2024-03-04T12:23:11.904072+0000","last_active":"2024-03-04T12:25:38.071274+0000","last_peered":"2024-03-04T12:25:38.071274+0000","last_clean":"2024-03-04T12:25:38.071274+0000","last_became_active":"2024-03-04T12:23:11.903097+0000","last_became_peered":"2024-03-04T12:23:11.903097+0000","last_unstale":"2024-03-04T12:25:38.071274+0000","last_undegraded":"2024-03-04T12:25:38.071274+0000","last_fullsized":"2024-03-04T12:25:38.071274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:36.606667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036799+0000","last_change":"2024-03-04T12:23:16.951019+0000","last_active":"2024-03-04T12:25:07.036799+0000","last_peered":"2024-03-04T12:25:07.036799+0000","last_clean":"2024-03-04T12:25:07.036799+0000","last_became_active":"2024-03-04T12:23:16.950524+0000","last_became_peered":"2024-03-04T12:23:16.950524+0000","last_unstale":"2024-03-04T12:25:07.036799+0000","last_undegraded":"2024-03-04T12:25:07.036799+0000","last_fullsized":"2024-03-04T12:25:07.036799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:48.376539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938226+0000","last_change":"2024-03-04T12:23:12.927615+0000","last_active":"2024-03-04T12:25:06.938226+0000","last_peered":"2024-03-04T12:25:06.938226+0000","last_clean":"2024-03-04T12:25:06.938226+0000","last_became_active":"2024-03-04T12:23:12.927190+0000","last_became_peered":"2024-03-04T12:23:12.927190+0000","last_unstale":"2024-03-04T12:25:06.938226+0000","last_undegraded":"2024-03-04T12:25:06.938226+0000","last_fullsized":"2024-03-04T12:25:06.938226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:54:19.019030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933773+0000","last_change":"2024-03-04T12:23:11.893542+0000","last_active":"2024-03-04T12:25:06.933773+0000","last_peered":"2024-03-04T12:25:06.933773+0000","last_clean":"2024-03-04T12:25:06.933773+0000","last_became_active":"2024-03-04T12:23:11.893166+0000","last_became_peered":"2024-03-04T12:23:11.893166+0000","last_unstale":"2024-03-04T12:25:06.933773+0000","last_undegraded":"2024-03-04T12:25:06.933773+0000","last_fullsized":"2024-03-04T12:25:06.933773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:32.117863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036450+0000","last_change":"2024-03-04T12:21:55.865103+0000","last_active":"2024-03-04T12:25:07.036450+0000","last_peered":"2024-03-04T12:25:07.036450+0000","last_clean":"2024-03-04T12:25:07.036450+0000","last_became_active":"2024-03-04T12:21:55.864553+0000","last_became_peered":"2024-03-04T12:21:55.864553+0000","last_unstale":"2024-03-04T12:25:07.036450+0000","last_undegraded":"2024-03-04T12:25:07.036450+0000","last_fullsized":"2024-03-04T12:25:07.036450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_clean_scrub_stamp":"2024-03-04T12:18:28.442423+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:31.052782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933731+0000","last_change":"2024-03-04T12:23:16.945852+0000","last_active":"2024-03-04T12:25:06.933731+0000","last_peered":"2024-03-04T12:25:06.933731+0000","last_clean":"2024-03-04T12:25:06.933731+0000","last_became_active":"2024-03-04T12:23:16.945496+0000","last_became_peered":"2024-03-04T12:23:16.945496+0000","last_unstale":"2024-03-04T12:25:06.933731+0000","last_undegraded":"2024-03-04T12:25:06.933731+0000","last_fullsized":"2024-03-04T12:25:06.933731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:35:10.880757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932050+0000","last_change":"2024-03-04T12:23:12.919219+0000","last_active":"2024-03-04T12:25:06.932050+0000","last_peered":"2024-03-04T12:25:06.932050+0000","last_clean":"2024-03-04T12:25:06.932050+0000","last_became_active":"2024-03-04T12:23:12.918897+0000","last_became_peered":"2024-03-04T12:23:12.918897+0000","last_unstale":"2024-03-04T12:25:06.932050+0000","last_undegraded":"2024-03-04T12:25:06.932050+0000","last_fullsized":"2024-03-04T12:25:06.932050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:24:32.594172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036668+0000","last_change":"2024-03-04T12:23:11.894544+0000","last_active":"2024-03-04T12:25:07.036668+0000","last_peered":"2024-03-04T12:25:07.036668+0000","last_clean":"2024-03-04T12:25:07.036668+0000","last_became_active":"2024-03-04T12:23:11.894188+0000","last_became_peered":"2024-03-04T12:23:11.894188+0000","last_unstale":"2024-03-04T12:25:07.036668+0000","last_undegraded":"2024-03-04T12:25:07.036668+0000","last_fullsized":"2024-03-04T12:25:07.036668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:28:14.687530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276193+0000","last_change":"2024-03-04T12:23:16.945510+0000","last_active":"2024-03-04T12:25:07.276193+0000","last_peered":"2024-03-04T12:25:07.276193+0000","last_clean":"2024-03-04T12:25:07.276193+0000","last_became_active":"2024-03-04T12:23:16.945140+0000","last_became_peered":"2024-03-04T12:23:16.945140+0000","last_unstale":"2024-03-04T12:25:07.276193+0000","last_undegraded":"2024-03-04T12:25:07.276193+0000","last_fullsized":"2024-03-04T12:25:07.276193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:44.102503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938493+0000","last_change":"2024-03-04T12:23:11.904515+0000","last_active":"2024-03-04T12:25:06.938493+0000","last_peered":"2024-03-04T12:25:06.938493+0000","last_clean":"2024-03-04T12:25:06.938493+0000","last_became_active":"2024-03-04T12:23:11.904149+0000","last_became_peered":"2024-03-04T12:23:11.904149+0000","last_unstale":"2024-03-04T12:25:06.938493+0000","last_undegraded":"2024-03-04T12:25:06.938493+0000","last_fullsized":"2024-03-04T12:25:06.938493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:04:51.382626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932543+0000","last_change":"2024-03-04T12:23:12.908749+0000","last_active":"2024-03-04T12:25:06.932543+0000","last_peered":"2024-03-04T12:25:06.932543+0000","last_clean":"2024-03-04T12:25:06.932543+0000","last_became_active":"2024-03-04T12:23:12.908405+0000","last_became_peered":"2024-03-04T12:23:12.908405+0000","last_unstale":"2024-03-04T12:25:06.932543+0000","last_undegraded":"2024-03-04T12:25:06.932543+0000","last_fullsized":"2024-03-04T12:25:06.932543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:10.472953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938204+0000","last_change":"2024-03-04T12:23:16.951145+0000","last_active":"2024-03-04T12:25:06.938204+0000","last_peered":"2024-03-04T12:25:06.938204+0000","last_clean":"2024-03-04T12:25:06.938204+0000","last_became_active":"2024-03-04T12:23:16.950702+0000","last_became_peered":"2024-03-04T12:23:16.950702+0000","last_unstale":"2024-03-04T12:25:06.938204+0000","last_undegraded":"2024-03-04T12:25:06.938204+0000","last_fullsized":"2024-03-04T12:25:06.938204+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:42.720616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'12","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.284439+0000","last_change":"2024-03-04T12:25:07.284006+0000","last_active":"2024-03-04T12:25:07.284439+0000","last_peered":"2024-03-04T12:25:07.284439+0000","last_clean":"2024-03-04T12:25:07.284439+0000","last_became_active":"2024-03-04T12:25:07.277446+0000","last_became_peered":"2024-03-04T12:25:07.277446+0000","last_unstale":"2024-03-04T12:25:07.284439+0000","last_undegraded":"2024-03-04T12:25:07.284439+0000","last_fullsized":"2024-03-04T12:25:07.284439+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:08:40.719222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938663+0000","last_change":"2024-03-04T12:23:12.925763+0000","last_active":"2024-03-04T12:25:06.938663+0000","last_peered":"2024-03-04T12:25:06.938663+0000","last_clean":"2024-03-04T12:25:06.938663+0000","last_became_active":"2024-03-04T12:23:12.925429+0000","last_became_peered":"2024-03-04T12:23:12.925429+0000","last_unstale":"2024-03-04T12:25:06.938663+0000","last_undegraded":"2024-03-04T12:25:06.938663+0000","last_fullsized":"2024-03-04T12:25:06.938663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:46.814459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939067+0000","last_change":"2024-03-04T12:23:16.942624+0000","last_active":"2024-03-04T12:25:06.939067+0000","last_peered":"2024-03-04T12:25:06.939067+0000","last_clean":"2024-03-04T12:25:06.939067+0000","last_became_active":"2024-03-04T12:23:16.942287+0000","last_became_peered":"2024-03-04T12:23:16.942287+0000","last_unstale":"2024-03-04T12:25:06.939067+0000","last_undegraded":"2024-03-04T12:25:06.939067+0000","last_fullsized":"2024-03-04T12:25:06.939067+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:59.479611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933762+0000","last_change":"2024-03-04T12:23:11.891504+0000","last_active":"2024-03-04T12:25:06.933762+0000","last_peered":"2024-03-04T12:25:06.933762+0000","last_clean":"2024-03-04T12:25:06.933762+0000","last_became_active":"2024-03-04T12:23:11.891134+0000","last_became_peered":"2024-03-04T12:23:11.891134+0000","last_unstale":"2024-03-04T12:25:06.933762+0000","last_undegraded":"2024-03-04T12:25:06.933762+0000","last_fullsized":"2024-03-04T12:25:06.933762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:41.481003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940405+0000","last_change":"2024-03-04T12:23:12.928989+0000","last_active":"2024-03-04T12:25:06.940405+0000","last_peered":"2024-03-04T12:25:06.940405+0000","last_clean":"2024-03-04T12:25:06.940405+0000","last_became_active":"2024-03-04T12:23:12.928609+0000","last_became_peered":"2024-03-04T12:23:12.928609+0000","last_unstale":"2024-03-04T12:25:06.940405+0000","last_undegraded":"2024-03-04T12:25:06.940405+0000","last_fullsized":"2024-03-04T12:25:06.940405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:39.979546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933694+0000","last_change":"2024-03-04T12:23:16.949590+0000","last_active":"2024-03-04T12:25:06.933694+0000","last_peered":"2024-03-04T12:25:06.933694+0000","last_clean":"2024-03-04T12:25:06.933694+0000","last_became_active":"2024-03-04T12:23:16.949086+0000","last_became_peered":"2024-03-04T12:23:16.949086+0000","last_unstale":"2024-03-04T12:25:06.933694+0000","last_undegraded":"2024-03-04T12:25:06.933694+0000","last_fullsized":"2024-03-04T12:25:06.933694+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:07:09.819097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.286558+0000","last_change":"2024-03-04T12:25:07.286087+0000","last_active":"2024-03-04T12:25:07.286558+0000","last_peered":"2024-03-04T12:25:07.286558+0000","last_clean":"2024-03-04T12:25:07.286558+0000","last_became_active":"2024-03-04T12:25:07.277551+0000","last_became_peered":"2024-03-04T12:25:07.277551+0000","last_unstale":"2024-03-04T12:25:07.286558+0000","last_undegraded":"2024-03-04T12:25:07.286558+0000","last_fullsized":"2024-03-04T12:25:07.286558+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:23.167325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938356+0000","last_change":"2024-03-04T12:23:12.907403+0000","last_active":"2024-03-04T12:25:06.938356+0000","last_peered":"2024-03-04T12:25:06.938356+0000","last_clean":"2024-03-04T12:25:06.938356+0000","last_became_active":"2024-03-04T12:23:12.906674+0000","last_became_peered":"2024-03-04T12:23:12.906674+0000","last_unstale":"2024-03-04T12:25:06.938356+0000","last_undegraded":"2024-03-04T12:25:06.938356+0000","last_fullsized":"2024-03-04T12:25:06.938356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:49:01.133676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936150+0000","last_change":"2024-03-04T12:23:16.955113+0000","last_active":"2024-03-04T12:25:06.936150+0000","last_peered":"2024-03-04T12:25:06.936150+0000","last_clean":"2024-03-04T12:25:06.936150+0000","last_became_active":"2024-03-04T12:23:16.951624+0000","last_became_peered":"2024-03-04T12:23:16.951624+0000","last_unstale":"2024-03-04T12:25:06.936150+0000","last_undegraded":"2024-03-04T12:25:06.936150+0000","last_fullsized":"2024-03-04T12:25:06.936150+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:22:23.935460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938252+0000","last_change":"2024-03-04T12:23:11.900213+0000","last_active":"2024-03-04T12:25:06.938252+0000","last_peered":"2024-03-04T12:25:06.938252+0000","last_clean":"2024-03-04T12:25:06.938252+0000","last_became_active":"2024-03-04T12:23:11.899832+0000","last_became_peered":"2024-03-04T12:23:11.899832+0000","last_unstale":"2024-03-04T12:25:06.938252+0000","last_undegraded":"2024-03-04T12:25:06.938252+0000","last_fullsized":"2024-03-04T12:25:06.938252+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:55:24.553309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276791+0000","last_change":"2024-03-04T12:23:12.913315+0000","last_active":"2024-03-04T12:25:07.276791+0000","last_peered":"2024-03-04T12:25:07.276791+0000","last_clean":"2024-03-04T12:25:07.276791+0000","last_became_active":"2024-03-04T12:23:12.912850+0000","last_became_peered":"2024-03-04T12:23:12.912850+0000","last_unstale":"2024-03-04T12:25:07.276791+0000","last_undegraded":"2024-03-04T12:25:07.276791+0000","last_fullsized":"2024-03-04T12:25:07.276791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:55.140866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933591+0000","last_change":"2024-03-04T12:23:16.943295+0000","last_active":"2024-03-04T12:25:06.933591+0000","last_peered":"2024-03-04T12:25:06.933591+0000","last_clean":"2024-03-04T12:25:06.933591+0000","last_became_active":"2024-03-04T12:23:16.942853+0000","last_became_peered":"2024-03-04T12:23:16.942853+0000","last_unstale":"2024-03-04T12:25:06.933591+0000","last_undegraded":"2024-03-04T12:25:06.933591+0000","last_fullsized":"2024-03-04T12:25:06.933591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:05:51.162179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:22.480402+0000","last_change":"2024-03-04T12:23:11.897870+0000","last_active":"2024-03-04T12:25:22.480402+0000","last_peered":"2024-03-04T12:25:22.480402+0000","last_clean":"2024-03-04T12:25:22.480402+0000","last_became_active":"2024-03-04T12:23:11.897443+0000","last_became_peered":"2024-03-04T12:23:11.897443+0000","last_unstale":"2024-03-04T12:25:22.480402+0000","last_undegraded":"2024-03-04T12:25:22.480402+0000","last_fullsized":"2024-03-04T12:25:22.480402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:49:37.875541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938638+0000","last_change":"2024-03-04T12:23:12.916441+0000","last_active":"2024-03-04T12:25:06.938638+0000","last_peered":"2024-03-04T12:25:06.938638+0000","last_clean":"2024-03-04T12:25:06.938638+0000","last_became_active":"2024-03-04T12:23:12.916078+0000","last_became_peered":"2024-03-04T12:23:12.916078+0000","last_unstale":"2024-03-04T12:25:06.938638+0000","last_undegraded":"2024-03-04T12:25:06.938638+0000","last_fullsized":"2024-03-04T12:25:06.938638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:45:23.663627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933671+0000","last_change":"2024-03-04T12:23:16.953341+0000","last_active":"2024-03-04T12:25:06.933671+0000","last_peered":"2024-03-04T12:25:06.933671+0000","last_clean":"2024-03-04T12:25:06.933671+0000","last_became_active":"2024-03-04T12:23:16.952878+0000","last_became_peered":"2024-03-04T12:23:16.952878+0000","last_unstale":"2024-03-04T12:25:06.933671+0000","last_undegraded":"2024-03-04T12:25:06.933671+0000","last_fullsized":"2024-03-04T12:25:06.933671+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:08:55.575263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938122+0000","last_change":"2024-03-04T12:23:11.903182+0000","last_active":"2024-03-04T12:25:06.938122+0000","last_peered":"2024-03-04T12:25:06.938122+0000","last_clean":"2024-03-04T12:25:06.938122+0000","last_became_active":"2024-03-04T12:23:11.902709+0000","last_became_peered":"2024-03-04T12:23:11.902709+0000","last_unstale":"2024-03-04T12:25:06.938122+0000","last_undegraded":"2024-03-04T12:25:06.938122+0000","last_fullsized":"2024-03-04T12:25:06.938122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:13:33.438929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037059+0000","last_change":"2024-03-04T12:23:12.931339+0000","last_active":"2024-03-04T12:25:07.037059+0000","last_peered":"2024-03-04T12:25:07.037059+0000","last_clean":"2024-03-04T12:25:07.037059+0000","last_became_active":"2024-03-04T12:23:12.930213+0000","last_became_peered":"2024-03-04T12:23:12.930213+0000","last_unstale":"2024-03-04T12:25:07.037059+0000","last_undegraded":"2024-03-04T12:25:07.037059+0000","last_fullsized":"2024-03-04T12:25:07.037059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:55:23.084139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936423+0000","last_change":"2024-03-04T12:23:16.948723+0000","last_active":"2024-03-04T12:25:06.936423+0000","last_peered":"2024-03-04T12:25:06.936423+0000","last_clean":"2024-03-04T12:25:06.936423+0000","last_became_active":"2024-03-04T12:23:16.948256+0000","last_became_peered":"2024-03-04T12:23:16.948256+0000","last_unstale":"2024-03-04T12:25:06.936423+0000","last_undegraded":"2024-03-04T12:25:06.936423+0000","last_fullsized":"2024-03-04T12:25:06.936423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:22.060517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":155,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:36.081533+0000","last_change":"2024-03-04T12:23:11.898475+0000","last_active":"2024-03-04T12:25:36.081533+0000","last_peered":"2024-03-04T12:25:36.081533+0000","last_clean":"2024-03-04T12:25:36.081533+0000","last_became_active":"2024-03-04T12:23:11.898117+0000","last_became_peered":"2024-03-04T12:23:11.898117+0000","last_unstale":"2024-03-04T12:25:36.081533+0000","last_undegraded":"2024-03-04T12:25:36.081533+0000","last_fullsized":"2024-03-04T12:25:36.081533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:21:25.119591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037220+0000","last_change":"2024-03-04T12:23:12.929772+0000","last_active":"2024-03-04T12:25:07.037220+0000","last_peered":"2024-03-04T12:25:07.037220+0000","last_clean":"2024-03-04T12:25:07.037220+0000","last_became_active":"2024-03-04T12:23:12.929405+0000","last_became_peered":"2024-03-04T12:23:12.929405+0000","last_unstale":"2024-03-04T12:25:07.037220+0000","last_undegraded":"2024-03-04T12:25:07.037220+0000","last_fullsized":"2024-03-04T12:25:07.037220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:22:42.874989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940260+0000","last_change":"2024-03-04T12:23:16.941715+0000","last_active":"2024-03-04T12:25:06.940260+0000","last_peered":"2024-03-04T12:25:06.940260+0000","last_clean":"2024-03-04T12:25:06.940260+0000","last_became_active":"2024-03-04T12:23:16.941298+0000","last_became_peered":"2024-03-04T12:23:16.941298+0000","last_unstale":"2024-03-04T12:25:06.940260+0000","last_undegraded":"2024-03-04T12:25:06.940260+0000","last_fullsized":"2024-03-04T12:25:06.940260+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:09:02.019531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933616+0000","last_change":"2024-03-04T12:23:11.896413+0000","last_active":"2024-03-04T12:25:06.933616+0000","last_peered":"2024-03-04T12:25:06.933616+0000","last_clean":"2024-03-04T12:25:06.933616+0000","last_became_active":"2024-03-04T12:23:11.895679+0000","last_became_peered":"2024-03-04T12:23:11.895679+0000","last_unstale":"2024-03-04T12:25:06.933616+0000","last_undegraded":"2024-03-04T12:25:06.933616+0000","last_fullsized":"2024-03-04T12:25:06.933616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:25:41.967530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036828+0000","last_change":"2024-03-04T12:23:12.920468+0000","last_active":"2024-03-04T12:25:07.036828+0000","last_peered":"2024-03-04T12:25:07.036828+0000","last_clean":"2024-03-04T12:25:07.036828+0000","last_became_active":"2024-03-04T12:23:12.920113+0000","last_became_peered":"2024-03-04T12:23:12.920113+0000","last_unstale":"2024-03-04T12:25:07.036828+0000","last_undegraded":"2024-03-04T12:25:07.036828+0000","last_fullsized":"2024-03-04T12:25:07.036828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:00:02.283518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.937863+0000","last_change":"2024-03-04T12:24:06.030444+0000","last_active":"2024-03-04T12:25:06.937863+0000","last_peered":"2024-03-04T12:25:06.937863+0000","last_clean":"2024-03-04T12:25:06.937863+0000","last_became_active":"2024-03-04T12:24:06.029934+0000","last_became_peered":"2024-03-04T12:24:06.029934+0000","last_unstale":"2024-03-04T12:25:06.937863+0000","last_undegraded":"2024-03-04T12:25:06.937863+0000","last_fullsized":"2024-03-04T12:25:06.937863+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:08:16.820509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932650+0000","last_change":"2024-03-04T12:23:11.903323+0000","last_active":"2024-03-04T12:25:06.932650+0000","last_peered":"2024-03-04T12:25:06.932650+0000","last_clean":"2024-03-04T12:25:06.932650+0000","last_became_active":"2024-03-04T12:23:11.902987+0000","last_became_peered":"2024-03-04T12:23:11.902987+0000","last_unstale":"2024-03-04T12:25:06.932650+0000","last_undegraded":"2024-03-04T12:25:06.932650+0000","last_fullsized":"2024-03-04T12:25:06.932650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:14.044391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037158+0000","last_change":"2024-03-04T12:23:12.913491+0000","last_active":"2024-03-04T12:25:07.037158+0000","last_peered":"2024-03-04T12:25:07.037158+0000","last_clean":"2024-03-04T12:25:07.037158+0000","last_became_active":"2024-03-04T12:23:12.912103+0000","last_became_peered":"2024-03-04T12:23:12.912103+0000","last_unstale":"2024-03-04T12:25:07.037158+0000","last_undegraded":"2024-03-04T12:25:07.037158+0000","last_fullsized":"2024-03-04T12:25:07.037158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:05:38.033596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276868+0000","last_change":"2024-03-04T12:23:12.908437+0000","last_active":"2024-03-04T12:25:07.276868+0000","last_peered":"2024-03-04T12:25:07.276868+0000","last_clean":"2024-03-04T12:25:07.276868+0000","last_became_active":"2024-03-04T12:23:12.908071+0000","last_became_peered":"2024-03-04T12:23:12.908071+0000","last_unstale":"2024-03-04T12:25:07.276868+0000","last_undegraded":"2024-03-04T12:25:07.276868+0000","last_fullsized":"2024-03-04T12:25:07.276868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:20.112645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939915+0000","last_change":"2024-03-04T12:23:12.921645+0000","last_active":"2024-03-04T12:25:06.939915+0000","last_peered":"2024-03-04T12:25:06.939915+0000","last_clean":"2024-03-04T12:25:06.939915+0000","last_became_active":"2024-03-04T12:23:12.921275+0000","last_became_peered":"2024-03-04T12:23:12.921275+0000","last_unstale":"2024-03-04T12:25:06.939915+0000","last_undegraded":"2024-03-04T12:25:06.939915+0000","last_fullsized":"2024-03-04T12:25:06.939915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:56:17.740782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939311+0000","last_change":"2024-03-04T12:23:12.919813+0000","last_active":"2024-03-04T12:25:06.939311+0000","last_peered":"2024-03-04T12:25:06.939311+0000","last_clean":"2024-03-04T12:25:06.939311+0000","last_became_active":"2024-03-04T12:23:12.919469+0000","last_became_peered":"2024-03-04T12:23:12.919469+0000","last_unstale":"2024-03-04T12:25:06.939311+0000","last_undegraded":"2024-03-04T12:25:06.939311+0000","last_fullsized":"2024-03-04T12:25:06.939311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:46.720268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940192+0000","last_change":"2024-03-04T12:23:12.918476+0000","last_active":"2024-03-04T12:25:06.940192+0000","last_peered":"2024-03-04T12:25:06.940192+0000","last_clean":"2024-03-04T12:25:06.940192+0000","last_became_active":"2024-03-04T12:23:12.917743+0000","last_became_peered":"2024-03-04T12:23:12.917743+0000","last_unstale":"2024-03-04T12:25:06.940192+0000","last_undegraded":"2024-03-04T12:25:06.940192+0000","last_fullsized":"2024-03-04T12:25:06.940192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:20.246769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932605+0000","last_change":"2024-03-04T12:23:12.918254+0000","last_active":"2024-03-04T12:25:06.932605+0000","last_peered":"2024-03-04T12:25:06.932605+0000","last_clean":"2024-03-04T12:25:06.932605+0000","last_became_active":"2024-03-04T12:23:12.917922+0000","last_became_peered":"2024-03-04T12:23:12.917922+0000","last_unstale":"2024-03-04T12:25:06.932605+0000","last_undegraded":"2024-03-04T12:25:06.932605+0000","last_fullsized":"2024-03-04T12:25:06.932605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:48:20.847984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276507+0000","last_change":"2024-03-04T12:23:12.910368+0000","last_active":"2024-03-04T12:25:07.276507+0000","last_peered":"2024-03-04T12:25:07.276507+0000","last_clean":"2024-03-04T12:25:07.276507+0000","last_became_active":"2024-03-04T12:23:12.910017+0000","last_became_peered":"2024-03-04T12:23:12.910017+0000","last_unstale":"2024-03-04T12:25:07.276507+0000","last_undegraded":"2024-03-04T12:25:07.276507+0000","last_fullsized":"2024-03-04T12:25:07.276507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:15:22.093644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939872+0000","last_change":"2024-03-04T12:23:12.921021+0000","last_active":"2024-03-04T12:25:06.939872+0000","last_peered":"2024-03-04T12:25:06.939872+0000","last_clean":"2024-03-04T12:25:06.939872+0000","last_became_active":"2024-03-04T12:23:12.920600+0000","last_became_peered":"2024-03-04T12:23:12.920600+0000","last_unstale":"2024-03-04T12:25:06.939872+0000","last_undegraded":"2024-03-04T12:25:06.939872+0000","last_fullsized":"2024-03-04T12:25:06.939872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:00:31.706900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939264+0000","last_change":"2024-03-04T12:24:05.957607+0000","last_active":"2024-03-04T12:25:06.939264+0000","last_peered":"2024-03-04T12:25:06.939264+0000","last_clean":"2024-03-04T12:25:06.939264+0000","last_became_active":"2024-03-04T12:24:05.957335+0000","last_became_peered":"2024-03-04T12:24:05.957335+0000","last_unstale":"2024-03-04T12:25:06.939264+0000","last_undegraded":"2024-03-04T12:25:06.939264+0000","last_fullsized":"2024-03-04T12:25:06.939264+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:12.926219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933250+0000","last_change":"2024-03-04T12:23:12.911394+0000","last_active":"2024-03-04T12:25:06.933250+0000","last_peered":"2024-03-04T12:25:06.933250+0000","last_clean":"2024-03-04T12:25:06.933250+0000","last_became_active":"2024-03-04T12:23:12.911061+0000","last_became_peered":"2024-03-04T12:23:12.911061+0000","last_unstale":"2024-03-04T12:25:06.933250+0000","last_undegraded":"2024-03-04T12:25:06.933250+0000","last_fullsized":"2024-03-04T12:25:06.933250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:39.880383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938372+0000","last_change":"2024-03-04T12:23:12.911356+0000","last_active":"2024-03-04T12:25:06.938372+0000","last_peered":"2024-03-04T12:25:06.938372+0000","last_clean":"2024-03-04T12:25:06.938372+0000","last_became_active":"2024-03-04T12:23:12.911029+0000","last_became_peered":"2024-03-04T12:23:12.911029+0000","last_unstale":"2024-03-04T12:25:06.938372+0000","last_undegraded":"2024-03-04T12:25:06.938372+0000","last_fullsized":"2024-03-04T12:25:06.938372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:52.611625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936612+0000","last_change":"2024-03-04T12:23:12.924921+0000","last_active":"2024-03-04T12:25:06.936612+0000","last_peered":"2024-03-04T12:25:06.936612+0000","last_clean":"2024-03-04T12:25:06.936612+0000","last_became_active":"2024-03-04T12:23:12.924558+0000","last_became_peered":"2024-03-04T12:23:12.924558+0000","last_unstale":"2024-03-04T12:25:06.936612+0000","last_undegraded":"2024-03-04T12:25:06.936612+0000","last_fullsized":"2024-03-04T12:25:06.936612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:10:53.238440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932437+0000","last_change":"2024-03-04T12:23:12.913106+0000","last_active":"2024-03-04T12:25:06.932437+0000","last_peered":"2024-03-04T12:25:06.932437+0000","last_clean":"2024-03-04T12:25:06.932437+0000","last_became_active":"2024-03-04T12:23:12.912741+0000","last_became_peered":"2024-03-04T12:23:12.912741+0000","last_unstale":"2024-03-04T12:25:06.932437+0000","last_undegraded":"2024-03-04T12:25:06.932437+0000","last_fullsized":"2024-03-04T12:25:06.932437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:46:21.406383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276375+0000","last_change":"2024-03-04T12:23:12.918082+0000","last_active":"2024-03-04T12:25:07.276375+0000","last_peered":"2024-03-04T12:25:07.276375+0000","last_clean":"2024-03-04T12:25:07.276375+0000","last_became_active":"2024-03-04T12:23:12.917754+0000","last_became_peered":"2024-03-04T12:23:12.917754+0000","last_unstale":"2024-03-04T12:25:07.276375+0000","last_undegraded":"2024-03-04T12:25:07.276375+0000","last_fullsized":"2024-03-04T12:25:07.276375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:56.601779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286257,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68620,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1216,"num_read_kb":684,"num_write":257,"num_write_kb":150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":210956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383724,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":520215,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28164,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716988,"statfs":{"total":95995035648,"available":95966195712,"internally_reserved":0,"allocated":1175552,"data_stored":1168389,"data_compressed":63026,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":524241,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":520523,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1118001,"data_compressed":63026,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515649,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":577505,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1052672,"data_stored":517105,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":578565,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":515357,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":520255,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1110016,"data_stored":1113051,"data_compressed":63026,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T12:25:42.191 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T12:25:42.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.321+0000 7fad9a705640 1 -- 172.21.15.60:0/2235811814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad94101450 msgr2=0x7fad941018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:42.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.321+0000 7fad9a705640 1 --2- 172.21.15.60:0/2235811814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad94101450 0x7fad941018d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fad88009900 tx=0x7fad8802bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:42.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 -- 172.21.15.60:0/2235811814 shutdown_connections 2024-03-04T12:25:42.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 --2- 172.21.15.60:0/2235811814 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad94101e10 0x7fad941086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 --2- 172.21.15.60:0/2235811814 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad94101450 0x7fad941018d0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 --2- 172.21.15.60:0/2235811814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad94100260 0x7fad94100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 -- 172.21.15.60:0/2235811814 >> 172.21.15.60:0/2235811814 conn(0x7fad940fb7d0 msgr2=0x7fad940fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:42.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 -- 172.21.15.60:0/2235811814 shutdown_connections 2024-03-04T12:25:42.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 -- 172.21.15.60:0/2235811814 wait complete. 2024-03-04T12:25:42.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 Processor -- start 2024-03-04T12:25:42.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 -- start start 2024-03-04T12:25:42.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad94100260 0x7fad94197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:42.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad94101450 0x7fad94197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:42.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad94101e10 0x7fad9419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:42.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9410bf40 con 0x7fad94100260 2024-03-04T12:25:42.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9410bdc0 con 0x7fad94101450 2024-03-04T12:25:42.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.322+0000 7fad9a705640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9410c0c0 con 0x7fad94101e10 2024-03-04T12:25:42.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad93fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad94100260 0x7fad94197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:42.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad93fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad94100260 0x7fad94197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55892/0 (socket says 172.21.15.60:55892) 2024-03-04T12:25:42.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad93fff640 1 -- 172.21.15.60:0/1708740825 learned_addr learned my addr 172.21.15.60:0/1708740825 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:42.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad98ca2640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad94101e10 0x7fad9419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:42.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad937fe640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad94101450 0x7fad94197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:42.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad93fff640 1 -- 172.21.15.60:0/1708740825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad94101e10 msgr2=0x7fad9419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad93fff640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad94101e10 0x7fad9419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad93fff640 1 -- 172.21.15.60:0/1708740825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad94101450 msgr2=0x7fad94197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad93fff640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad94101450 0x7fad94197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad93fff640 1 -- 172.21.15.60:0/1708740825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9419f2b0 con 0x7fad94100260 2024-03-04T12:25:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad937fe640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad94101450 0x7fad94197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad93fff640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad94100260 0x7fad94197690 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fad8400d950 tx=0x7fad8400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad917fa640 1 -- 172.21.15.60:0/1708740825 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad84017070 con 0x7fad94100260 2024-03-04T12:25:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad9a705640 1 -- 172.21.15.60:0/1708740825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9419f5a0 con 0x7fad94100260 2024-03-04T12:25:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad917fa640 1 -- 172.21.15.60:0/1708740825 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad84009c30 con 0x7fad94100260 2024-03-04T12:25:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad917fa640 1 -- 172.21.15.60:0/1708740825 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8400f600 con 0x7fad94100260 2024-03-04T12:25:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.323+0000 7fad9a705640 1 -- 172.21.15.60:0/1708740825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9419faf0 con 0x7fad94100260 2024-03-04T12:25:42.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.324+0000 7fad9a705640 1 -- 172.21.15.60:0/1708740825 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9419fd50 con 0x7fad94100260 2024-03-04T12:25:42.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.325+0000 7fad917fa640 1 -- 172.21.15.60:0/1708740825 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fad8401a070 con 0x7fad94100260 2024-03-04T12:25:42.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.326+0000 7fad917fa640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fad7c0a1e10 0x7fad7c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:42.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.326+0000 7fad937fe640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fad7c0a1e10 0x7fad7c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:42.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.326+0000 7fad917fa640 1 -- 172.21.15.60:0/1708740825 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fad840cdd70 con 0x7fad94100260 2024-03-04T12:25:42.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.327+0000 7fad937fe640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fad7c0a1e10 0x7fad7c0a42d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fad880096c0 tx=0x7fad8802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:42.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.331+0000 7fad917fa640 1 -- 172.21.15.60:0/1708740825 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fad84009de0 con 0x7fad94100260 2024-03-04T12:25:42.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:42 smithi086 ceph-mon[36851]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-04T12:25:42.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:42 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2151704167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:42.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:42 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/4113103525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:42.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:42 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3687276221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:42.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:42 smithi063 ceph-mon[36717]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-04T12:25:42.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:42 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2151704167' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:42.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:42 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/4113103525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:42.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:42 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3687276221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:42.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.510+0000 7fad9a705640 1 -- 172.21.15.60:0/1708740825 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad94068d10 con 0x7fad7c0a1e10 2024-03-04T12:25:42.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.520+0000 7fad917fa640 1 -- 172.21.15.60:0/1708740825 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340807 (secure 0 0 0) 0x7fad94068d10 con 0x7fad7c0a1e10 2024-03-04T12:25:42.531 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:42.533 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-04T12:25:42.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.523+0000 7fad9a705640 1 -- 172.21.15.60:0/1708740825 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fad7c0a1e10 msgr2=0x7fad7c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:42.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.523+0000 7fad9a705640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fad7c0a1e10 0x7fad7c0a42d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fad880096c0 tx=0x7fad8802df90 comp rx=0 tx=0).stop 2024-03-04T12:25:42.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.523+0000 7fad9a705640 1 -- 172.21.15.60:0/1708740825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad94100260 msgr2=0x7fad94197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:42.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.523+0000 7fad9a705640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad94100260 0x7fad94197690 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fad8400d950 tx=0x7fad8400de10 comp rx=0 tx=0).stop 2024-03-04T12:25:42.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.523+0000 7fad9a705640 1 -- 172.21.15.60:0/1708740825 shutdown_connections 2024-03-04T12:25:42.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.523+0000 7fad9a705640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fad7c0a1e10 0x7fad7c0a42d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.523+0000 7fad9a705640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad94101e10 0x7fad9419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.523+0000 7fad9a705640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad94101450 0x7fad94197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.523+0000 7fad9a705640 1 --2- 172.21.15.60:0/1708740825 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad94100260 0x7fad94197690 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.523+0000 7fad9a705640 1 -- 172.21.15.60:0/1708740825 >> 172.21.15.60:0/1708740825 conn(0x7fad940fb7d0 msgr2=0x7fad94068600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:42.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.523+0000 7fad9a705640 1 -- 172.21.15.60:0/1708740825 shutdown_connections 2024-03-04T12:25:42.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.523+0000 7fad9a705640 1 -- 172.21.15.60:0/1708740825 wait complete. 2024-03-04T12:25:42.541 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-03-04T12:25:42.192927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659222,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1357,"num_read_kb":808,"num_write":444,"num_write_kb":3092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386748,"kb_used_data":12584,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555076,"statfs":{"total":1151940427776,"available":1151544397824,"internally_reserved":0,"allocated":12886016,"data_stored":8188856,"data_compressed":428748,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":57,"num_write":13,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003123"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276572+0000","last_change":"2024-03-04T12:23:12.926673+0000","last_active":"2024-03-04T12:25:07.276572+0000","last_peered":"2024-03-04T12:25:07.276572+0000","last_clean":"2024-03-04T12:25:07.276572+0000","last_became_active":"2024-03-04T12:23:12.926299+0000","last_became_peered":"2024-03-04T12:23:12.926299+0000","last_unstale":"2024-03-04T12:25:07.276572+0000","last_undegraded":"2024-03-04T12:25:07.276572+0000","last_fullsized":"2024-03-04T12:25:07.276572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:19.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940339+0000","last_change":"2024-03-04T12:23:12.921758+0000","last_active":"2024-03-04T12:25:06.940339+0000","last_peered":"2024-03-04T12:25:06.940339+0000","last_clean":"2024-03-04T12:25:06.940339+0000","last_became_active":"2024-03-04T12:23:12.921317+0000","last_became_peered":"2024-03-04T12:23:12.921317+0000","last_unstale":"2024-03-04T12:25:06.940339+0000","last_undegraded":"2024-03-04T12:25:06.940339+0000","last_fullsized":"2024-03-04T12:25:06.940339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:20:25.179004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939582+0000","last_change":"2024-03-04T12:23:12.925771+0000","last_active":"2024-03-04T12:25:06.939582+0000","last_peered":"2024-03-04T12:25:06.939582+0000","last_clean":"2024-03-04T12:25:06.939582+0000","last_became_active":"2024-03-04T12:23:12.925447+0000","last_became_peered":"2024-03-04T12:23:12.925447+0000","last_unstale":"2024-03-04T12:25:06.939582+0000","last_undegraded":"2024-03-04T12:25:06.939582+0000","last_fullsized":"2024-03-04T12:25:06.939582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:12.380831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935876+0000","last_change":"2024-03-04T12:24:05.959806+0000","last_active":"2024-03-04T12:25:06.935876+0000","last_peered":"2024-03-04T12:25:06.935876+0000","last_clean":"2024-03-04T12:25:06.935876+0000","last_became_active":"2024-03-04T12:24:05.959557+0000","last_became_peered":"2024-03-04T12:24:05.959557+0000","last_unstale":"2024-03-04T12:25:06.935876+0000","last_undegraded":"2024-03-04T12:25:06.935876+0000","last_fullsized":"2024-03-04T12:25:06.935876+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:51.945517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938061+0000","last_change":"2024-03-04T12:23:12.921118+0000","last_active":"2024-03-04T12:25:06.938061+0000","last_peered":"2024-03-04T12:25:06.938061+0000","last_clean":"2024-03-04T12:25:06.938061+0000","last_became_active":"2024-03-04T12:23:12.920127+0000","last_became_peered":"2024-03-04T12:23:12.920127+0000","last_unstale":"2024-03-04T12:25:06.938061+0000","last_undegraded":"2024-03-04T12:25:06.938061+0000","last_fullsized":"2024-03-04T12:25:06.938061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:22:10.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.025786+0000","last_change":"2024-03-04T12:23:12.916751+0000","last_active":"2024-03-04T12:25:07.025786+0000","last_peered":"2024-03-04T12:25:07.025786+0000","last_clean":"2024-03-04T12:25:07.025786+0000","last_became_active":"2024-03-04T12:23:12.916404+0000","last_became_peered":"2024-03-04T12:23:12.916404+0000","last_unstale":"2024-03-04T12:25:07.025786+0000","last_undegraded":"2024-03-04T12:25:07.025786+0000","last_fullsized":"2024-03-04T12:25:07.025786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:39.998992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036504+0000","last_change":"2024-03-04T12:24:06.029608+0000","last_active":"2024-03-04T12:25:07.036504+0000","last_peered":"2024-03-04T12:25:07.036504+0000","last_clean":"2024-03-04T12:25:07.036504+0000","last_became_active":"2024-03-04T12:24:06.029021+0000","last_became_peered":"2024-03-04T12:24:06.029021+0000","last_unstale":"2024-03-04T12:25:07.036504+0000","last_undegraded":"2024-03-04T12:25:07.036504+0000","last_fullsized":"2024-03-04T12:25:07.036504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:13.140511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938055+0000","last_change":"2024-03-04T12:23:12.917627+0000","last_active":"2024-03-04T12:25:06.938055+0000","last_peered":"2024-03-04T12:25:06.938055+0000","last_clean":"2024-03-04T12:25:06.938055+0000","last_became_active":"2024-03-04T12:23:12.917279+0000","last_became_peered":"2024-03-04T12:23:12.917279+0000","last_unstale":"2024-03-04T12:25:06.938055+0000","last_undegraded":"2024-03-04T12:25:06.938055+0000","last_fullsized":"2024-03-04T12:25:06.938055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:33:03.487273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.937093+0000","last_change":"2024-03-04T12:23:12.922601+0000","last_active":"2024-03-04T12:25:06.937093+0000","last_peered":"2024-03-04T12:25:06.937093+0000","last_clean":"2024-03-04T12:25:06.937093+0000","last_became_active":"2024-03-04T12:23:12.922037+0000","last_became_peered":"2024-03-04T12:23:12.922037+0000","last_unstale":"2024-03-04T12:25:06.937093+0000","last_undegraded":"2024-03-04T12:25:06.937093+0000","last_fullsized":"2024-03-04T12:25:06.937093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:32.076439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.026254+0000","last_change":"2024-03-04T12:23:12.916308+0000","last_active":"2024-03-04T12:25:07.026254+0000","last_peered":"2024-03-04T12:25:07.026254+0000","last_clean":"2024-03-04T12:25:07.026254+0000","last_became_active":"2024-03-04T12:23:12.915897+0000","last_became_peered":"2024-03-04T12:23:12.915897+0000","last_unstale":"2024-03-04T12:25:07.026254+0000","last_undegraded":"2024-03-04T12:25:07.026254+0000","last_fullsized":"2024-03-04T12:25:07.026254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:15:53.133053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037015+0000","last_change":"2024-03-04T12:23:12.930700+0000","last_active":"2024-03-04T12:25:07.037015+0000","last_peered":"2024-03-04T12:25:07.037015+0000","last_clean":"2024-03-04T12:25:07.037015+0000","last_became_active":"2024-03-04T12:23:12.930328+0000","last_became_peered":"2024-03-04T12:23:12.930328+0000","last_unstale":"2024-03-04T12:25:07.037015+0000","last_undegraded":"2024-03-04T12:25:07.037015+0000","last_fullsized":"2024-03-04T12:25:07.037015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:29.990527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933883+0000","last_change":"2024-03-04T12:23:12.915471+0000","last_active":"2024-03-04T12:25:06.933883+0000","last_peered":"2024-03-04T12:25:06.933883+0000","last_clean":"2024-03-04T12:25:06.933883+0000","last_became_active":"2024-03-04T12:23:12.913627+0000","last_became_peered":"2024-03-04T12:23:12.913627+0000","last_unstale":"2024-03-04T12:25:06.933883+0000","last_undegraded":"2024-03-04T12:25:06.933883+0000","last_fullsized":"2024-03-04T12:25:06.933883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:42:56.762782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932392+0000","last_change":"2024-03-04T12:23:12.899143+0000","last_active":"2024-03-04T12:25:06.932392+0000","last_peered":"2024-03-04T12:25:06.932392+0000","last_clean":"2024-03-04T12:25:06.932392+0000","last_became_active":"2024-03-04T12:23:12.898498+0000","last_became_peered":"2024-03-04T12:23:12.898498+0000","last_unstale":"2024-03-04T12:25:06.932392+0000","last_undegraded":"2024-03-04T12:25:06.932392+0000","last_fullsized":"2024-03-04T12:25:06.932392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:33.840683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939467+0000","last_change":"2024-03-04T12:23:12.920873+0000","last_active":"2024-03-04T12:25:06.939467+0000","last_peered":"2024-03-04T12:25:06.939467+0000","last_clean":"2024-03-04T12:25:06.939467+0000","last_became_active":"2024-03-04T12:23:12.919080+0000","last_became_peered":"2024-03-04T12:23:12.919080+0000","last_unstale":"2024-03-04T12:25:06.939467+0000","last_undegraded":"2024-03-04T12:25:06.939467+0000","last_fullsized":"2024-03-04T12:25:06.939467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:47:26.710624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939195+0000","last_change":"2024-03-04T12:24:05.959024+0000","last_active":"2024-03-04T12:25:06.939195+0000","last_peered":"2024-03-04T12:25:06.939195+0000","last_clean":"2024-03-04T12:25:06.939195+0000","last_became_active":"2024-03-04T12:24:05.958750+0000","last_became_peered":"2024-03-04T12:24:05.958750+0000","last_unstale":"2024-03-04T12:25:06.939195+0000","last_undegraded":"2024-03-04T12:25:06.939195+0000","last_fullsized":"2024-03-04T12:25:06.939195+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:59:33.263357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936333+0000","last_change":"2024-03-04T12:23:12.923860+0000","last_active":"2024-03-04T12:25:06.936333+0000","last_peered":"2024-03-04T12:25:06.936333+0000","last_clean":"2024-03-04T12:25:06.936333+0000","last_became_active":"2024-03-04T12:23:12.923530+0000","last_became_peered":"2024-03-04T12:23:12.923530+0000","last_unstale":"2024-03-04T12:25:06.936333+0000","last_undegraded":"2024-03-04T12:25:06.936333+0000","last_fullsized":"2024-03-04T12:25:06.936333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:30:01.466458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276427+0000","last_change":"2024-03-04T12:23:12.927159+0000","last_active":"2024-03-04T12:25:07.276427+0000","last_peered":"2024-03-04T12:25:07.276427+0000","last_clean":"2024-03-04T12:25:07.276427+0000","last_became_active":"2024-03-04T12:23:12.926800+0000","last_became_peered":"2024-03-04T12:23:12.926800+0000","last_unstale":"2024-03-04T12:25:07.276427+0000","last_undegraded":"2024-03-04T12:25:07.276427+0000","last_fullsized":"2024-03-04T12:25:07.276427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:52.633648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936123+0000","last_change":"2024-03-04T12:23:12.920607+0000","last_active":"2024-03-04T12:25:06.936123+0000","last_peered":"2024-03-04T12:25:06.936123+0000","last_clean":"2024-03-04T12:25:06.936123+0000","last_became_active":"2024-03-04T12:23:12.920257+0000","last_became_peered":"2024-03-04T12:23:12.920257+0000","last_unstale":"2024-03-04T12:25:06.936123+0000","last_undegraded":"2024-03-04T12:25:06.936123+0000","last_fullsized":"2024-03-04T12:25:06.936123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:39.353673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939723+0000","last_change":"2024-03-04T12:23:12.906635+0000","last_active":"2024-03-04T12:25:06.939723+0000","last_peered":"2024-03-04T12:25:06.939723+0000","last_clean":"2024-03-04T12:25:06.939723+0000","last_became_active":"2024-03-04T12:23:12.906243+0000","last_became_peered":"2024-03-04T12:23:12.906243+0000","last_unstale":"2024-03-04T12:25:06.939723+0000","last_undegraded":"2024-03-04T12:25:06.939723+0000","last_fullsized":"2024-03-04T12:25:06.939723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:44.375934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939135+0000","last_change":"2024-03-04T12:23:16.951685+0000","last_active":"2024-03-04T12:25:06.939135+0000","last_peered":"2024-03-04T12:25:06.939135+0000","last_clean":"2024-03-04T12:25:06.939135+0000","last_became_active":"2024-03-04T12:23:16.951329+0000","last_became_peered":"2024-03-04T12:23:16.951329+0000","last_unstale":"2024-03-04T12:25:06.939135+0000","last_undegraded":"2024-03-04T12:25:06.939135+0000","last_fullsized":"2024-03-04T12:25:06.939135+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:03:27.233466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":165,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:39.077764+0000","last_change":"2024-03-04T12:23:11.900681+0000","last_active":"2024-03-04T12:25:39.077764+0000","last_peered":"2024-03-04T12:25:39.077764+0000","last_clean":"2024-03-04T12:25:39.077764+0000","last_became_active":"2024-03-04T12:23:11.900365+0000","last_became_peered":"2024-03-04T12:23:11.900365+0000","last_unstale":"2024-03-04T12:25:39.077764+0000","last_undegraded":"2024-03-04T12:25:39.077764+0000","last_fullsized":"2024-03-04T12:25:39.077764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:53.052513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936030+0000","last_change":"2024-03-04T12:24:06.030707+0000","last_active":"2024-03-04T12:25:06.936030+0000","last_peered":"2024-03-04T12:25:06.936030+0000","last_clean":"2024-03-04T12:25:06.936030+0000","last_became_active":"2024-03-04T12:24:06.030306+0000","last_became_peered":"2024-03-04T12:24:06.030306+0000","last_unstale":"2024-03-04T12:25:06.936030+0000","last_undegraded":"2024-03-04T12:25:06.936030+0000","last_fullsized":"2024-03-04T12:25:06.936030+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:59:18.939734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933263+0000","last_change":"2024-03-04T12:23:16.953704+0000","last_active":"2024-03-04T12:25:06.933263+0000","last_peered":"2024-03-04T12:25:06.933263+0000","last_clean":"2024-03-04T12:25:06.933263+0000","last_became_active":"2024-03-04T12:23:16.953177+0000","last_became_peered":"2024-03-04T12:23:16.953177+0000","last_unstale":"2024-03-04T12:25:06.933263+0000","last_undegraded":"2024-03-04T12:25:06.933263+0000","last_fullsized":"2024-03-04T12:25:06.933263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:37:00.469767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939293+0000","last_change":"2024-03-04T12:23:11.904271+0000","last_active":"2024-03-04T12:25:06.939293+0000","last_peered":"2024-03-04T12:25:06.939293+0000","last_clean":"2024-03-04T12:25:06.939293+0000","last_became_active":"2024-03-04T12:23:11.903963+0000","last_became_peered":"2024-03-04T12:23:11.903963+0000","last_unstale":"2024-03-04T12:25:06.939293+0000","last_undegraded":"2024-03-04T12:25:06.939293+0000","last_fullsized":"2024-03-04T12:25:06.939293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:44.774621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939180+0000","last_change":"2024-03-04T12:23:12.925750+0000","last_active":"2024-03-04T12:25:06.939180+0000","last_peered":"2024-03-04T12:25:06.939180+0000","last_clean":"2024-03-04T12:25:06.939180+0000","last_became_active":"2024-03-04T12:23:12.925380+0000","last_became_peered":"2024-03-04T12:23:12.925380+0000","last_unstale":"2024-03-04T12:25:06.939180+0000","last_undegraded":"2024-03-04T12:25:06.939180+0000","last_fullsized":"2024-03-04T12:25:06.939180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:14.665127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933357+0000","last_change":"2024-03-04T12:23:16.952303+0000","last_active":"2024-03-04T12:25:06.933357+0000","last_peered":"2024-03-04T12:25:06.933357+0000","last_clean":"2024-03-04T12:25:06.933357+0000","last_became_active":"2024-03-04T12:23:16.951666+0000","last_became_peered":"2024-03-04T12:23:16.951666+0000","last_unstale":"2024-03-04T12:25:06.933357+0000","last_undegraded":"2024-03-04T12:25:06.933357+0000","last_fullsized":"2024-03-04T12:25:06.933357+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:41.326808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938665+0000","last_change":"2024-03-04T12:23:11.903871+0000","last_active":"2024-03-04T12:25:06.938665+0000","last_peered":"2024-03-04T12:25:06.938665+0000","last_clean":"2024-03-04T12:25:06.938665+0000","last_became_active":"2024-03-04T12:23:11.902735+0000","last_became_peered":"2024-03-04T12:23:11.902735+0000","last_unstale":"2024-03-04T12:25:06.938665+0000","last_undegraded":"2024-03-04T12:25:06.938665+0000","last_fullsized":"2024-03-04T12:25:06.938665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:45:09.897740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.937164+0000","last_change":"2024-03-04T12:23:12.924504+0000","last_active":"2024-03-04T12:25:06.937164+0000","last_peered":"2024-03-04T12:25:06.937164+0000","last_clean":"2024-03-04T12:25:06.937164+0000","last_became_active":"2024-03-04T12:23:12.922934+0000","last_became_peered":"2024-03-04T12:23:12.922934+0000","last_unstale":"2024-03-04T12:25:06.937164+0000","last_undegraded":"2024-03-04T12:25:06.937164+0000","last_fullsized":"2024-03-04T12:25:06.937164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:28.974601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933262+0000","last_change":"2024-03-04T12:23:16.952324+0000","last_active":"2024-03-04T12:25:06.933262+0000","last_peered":"2024-03-04T12:25:06.933262+0000","last_clean":"2024-03-04T12:25:06.933262+0000","last_became_active":"2024-03-04T12:23:16.951145+0000","last_became_peered":"2024-03-04T12:23:16.951145+0000","last_unstale":"2024-03-04T12:25:06.933262+0000","last_undegraded":"2024-03-04T12:25:06.933262+0000","last_fullsized":"2024-03-04T12:25:06.933262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:20:47.029523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938577+0000","last_change":"2024-03-04T12:23:11.904391+0000","last_active":"2024-03-04T12:25:06.938577+0000","last_peered":"2024-03-04T12:25:06.938577+0000","last_clean":"2024-03-04T12:25:06.938577+0000","last_became_active":"2024-03-04T12:23:11.903244+0000","last_became_peered":"2024-03-04T12:23:11.903244+0000","last_unstale":"2024-03-04T12:25:06.938577+0000","last_undegraded":"2024-03-04T12:25:06.938577+0000","last_fullsized":"2024-03-04T12:25:06.938577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:27.590418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933368+0000","last_change":"2024-03-04T12:23:12.925623+0000","last_active":"2024-03-04T12:25:06.933368+0000","last_peered":"2024-03-04T12:25:06.933368+0000","last_clean":"2024-03-04T12:25:06.933368+0000","last_became_active":"2024-03-04T12:23:12.925243+0000","last_became_peered":"2024-03-04T12:23:12.925243+0000","last_unstale":"2024-03-04T12:25:06.933368+0000","last_undegraded":"2024-03-04T12:25:06.933368+0000","last_fullsized":"2024-03-04T12:25:06.933368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:11:04.142655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939469+0000","last_change":"2024-03-04T12:23:16.946064+0000","last_active":"2024-03-04T12:25:06.939469+0000","last_peered":"2024-03-04T12:25:06.939469+0000","last_clean":"2024-03-04T12:25:06.939469+0000","last_became_active":"2024-03-04T12:23:16.945342+0000","last_became_peered":"2024-03-04T12:23:16.945342+0000","last_unstale":"2024-03-04T12:25:06.939469+0000","last_undegraded":"2024-03-04T12:25:06.939469+0000","last_fullsized":"2024-03-04T12:25:06.939469+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:43:37.500216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:37.071102+0000","last_change":"2024-03-04T12:23:11.899478+0000","last_active":"2024-03-04T12:25:37.071102+0000","last_peered":"2024-03-04T12:25:37.071102+0000","last_clean":"2024-03-04T12:25:37.071102+0000","last_became_active":"2024-03-04T12:23:11.899153+0000","last_became_peered":"2024-03-04T12:23:11.899153+0000","last_unstale":"2024-03-04T12:25:37.071102+0000","last_undegraded":"2024-03-04T12:25:37.071102+0000","last_fullsized":"2024-03-04T12:25:37.071102+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:10.709054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938359+0000","last_change":"2024-03-04T12:23:12.908560+0000","last_active":"2024-03-04T12:25:06.938359+0000","last_peered":"2024-03-04T12:25:06.938359+0000","last_clean":"2024-03-04T12:25:06.938359+0000","last_became_active":"2024-03-04T12:23:12.908221+0000","last_became_peered":"2024-03-04T12:23:12.908221+0000","last_unstale":"2024-03-04T12:25:06.938359+0000","last_undegraded":"2024-03-04T12:25:06.938359+0000","last_fullsized":"2024-03-04T12:25:06.938359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:28:06.545945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939489+0000","last_change":"2024-03-04T12:24:06.031257+0000","last_active":"2024-03-04T12:25:06.939489+0000","last_peered":"2024-03-04T12:25:06.939489+0000","last_clean":"2024-03-04T12:25:06.939489+0000","last_became_active":"2024-03-04T12:24:06.030409+0000","last_became_peered":"2024-03-04T12:24:06.030409+0000","last_unstale":"2024-03-04T12:25:06.939489+0000","last_undegraded":"2024-03-04T12:25:06.939489+0000","last_fullsized":"2024-03-04T12:25:06.939489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:31.403224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.025691+0000","last_change":"2024-03-04T12:23:11.898360+0000","last_active":"2024-03-04T12:25:07.025691+0000","last_peered":"2024-03-04T12:25:07.025691+0000","last_clean":"2024-03-04T12:25:07.025691+0000","last_became_active":"2024-03-04T12:23:11.897988+0000","last_became_peered":"2024-03-04T12:23:11.897988+0000","last_unstale":"2024-03-04T12:25:07.025691+0000","last_undegraded":"2024-03-04T12:25:07.025691+0000","last_fullsized":"2024-03-04T12:25:07.025691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:17.448032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933598+0000","last_change":"2024-03-04T12:23:12.918392+0000","last_active":"2024-03-04T12:25:06.933598+0000","last_peered":"2024-03-04T12:25:06.933598+0000","last_clean":"2024-03-04T12:25:06.933598+0000","last_became_active":"2024-03-04T12:23:12.917656+0000","last_became_peered":"2024-03-04T12:23:12.917656+0000","last_unstale":"2024-03-04T12:25:06.933598+0000","last_undegraded":"2024-03-04T12:25:06.933598+0000","last_fullsized":"2024-03-04T12:25:06.933598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:44:14.545925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276449+0000","last_change":"2024-03-04T12:23:16.941841+0000","last_active":"2024-03-04T12:25:07.276449+0000","last_peered":"2024-03-04T12:25:07.276449+0000","last_clean":"2024-03-04T12:25:07.276449+0000","last_became_active":"2024-03-04T12:23:16.941449+0000","last_became_peered":"2024-03-04T12:23:16.941449+0000","last_unstale":"2024-03-04T12:25:07.276449+0000","last_undegraded":"2024-03-04T12:25:07.276449+0000","last_fullsized":"2024-03-04T12:25:07.276449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:50:35.451834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932169+0000","last_change":"2024-03-04T12:23:11.906723+0000","last_active":"2024-03-04T12:25:06.932169+0000","last_peered":"2024-03-04T12:25:06.932169+0000","last_clean":"2024-03-04T12:25:06.932169+0000","last_became_active":"2024-03-04T12:23:11.906357+0000","last_became_peered":"2024-03-04T12:23:11.906357+0000","last_unstale":"2024-03-04T12:25:06.932169+0000","last_undegraded":"2024-03-04T12:25:06.932169+0000","last_fullsized":"2024-03-04T12:25:06.932169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:09.592236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938664+0000","last_change":"2024-03-04T12:23:12.916435+0000","last_active":"2024-03-04T12:25:06.938664+0000","last_peered":"2024-03-04T12:25:06.938664+0000","last_clean":"2024-03-04T12:25:06.938664+0000","last_became_active":"2024-03-04T12:23:12.915607+0000","last_became_peered":"2024-03-04T12:23:12.915607+0000","last_unstale":"2024-03-04T12:25:06.938664+0000","last_undegraded":"2024-03-04T12:25:06.938664+0000","last_fullsized":"2024-03-04T12:25:06.938664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:41:32.900344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940234+0000","last_change":"2024-03-04T12:23:16.944342+0000","last_active":"2024-03-04T12:25:06.940234+0000","last_peered":"2024-03-04T12:25:06.940234+0000","last_clean":"2024-03-04T12:25:06.940234+0000","last_became_active":"2024-03-04T12:23:16.943683+0000","last_became_peered":"2024-03-04T12:23:16.943683+0000","last_unstale":"2024-03-04T12:25:06.940234+0000","last_undegraded":"2024-03-04T12:25:06.940234+0000","last_fullsized":"2024-03-04T12:25:06.940234+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:19.658860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933018+0000","last_change":"2024-03-04T12:23:11.899194+0000","last_active":"2024-03-04T12:25:06.933018+0000","last_peered":"2024-03-04T12:25:06.933018+0000","last_clean":"2024-03-04T12:25:06.933018+0000","last_became_active":"2024-03-04T12:23:11.898109+0000","last_became_peered":"2024-03-04T12:23:11.898109+0000","last_unstale":"2024-03-04T12:25:06.933018+0000","last_undegraded":"2024-03-04T12:25:06.933018+0000","last_fullsized":"2024-03-04T12:25:06.933018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:08.808202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935619+0000","last_change":"2024-03-04T12:23:12.916549+0000","last_active":"2024-03-04T12:25:06.935619+0000","last_peered":"2024-03-04T12:25:06.935619+0000","last_clean":"2024-03-04T12:25:06.935619+0000","last_became_active":"2024-03-04T12:23:12.916204+0000","last_became_peered":"2024-03-04T12:23:12.916204+0000","last_unstale":"2024-03-04T12:25:06.935619+0000","last_undegraded":"2024-03-04T12:25:06.935619+0000","last_fullsized":"2024-03-04T12:25:06.935619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:56.412600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938437+0000","last_change":"2024-03-04T12:23:16.945903+0000","last_active":"2024-03-04T12:25:06.938437+0000","last_peered":"2024-03-04T12:25:06.938437+0000","last_clean":"2024-03-04T12:25:06.938437+0000","last_became_active":"2024-03-04T12:23:16.945535+0000","last_became_peered":"2024-03-04T12:23:16.945535+0000","last_unstale":"2024-03-04T12:25:06.938437+0000","last_undegraded":"2024-03-04T12:25:06.938437+0000","last_fullsized":"2024-03-04T12:25:06.938437+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:45:01.178370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'6","reported_seq":272,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:37.082131+0000","last_change":"2024-03-04T12:23:11.901570+0000","last_active":"2024-03-04T12:25:37.082131+0000","last_peered":"2024-03-04T12:25:37.082131+0000","last_clean":"2024-03-04T12:25:37.082131+0000","last_became_active":"2024-03-04T12:23:11.900702+0000","last_became_peered":"2024-03-04T12:23:11.900702+0000","last_unstale":"2024-03-04T12:25:37.082131+0000","last_undegraded":"2024-03-04T12:25:37.082131+0000","last_fullsized":"2024-03-04T12:25:37.082131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:18.758453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936880+0000","last_change":"2024-03-04T12:23:12.919571+0000","last_active":"2024-03-04T12:25:06.936880+0000","last_peered":"2024-03-04T12:25:06.936880+0000","last_clean":"2024-03-04T12:25:06.936880+0000","last_became_active":"2024-03-04T12:23:12.919143+0000","last_became_peered":"2024-03-04T12:23:12.919143+0000","last_unstale":"2024-03-04T12:25:06.936880+0000","last_undegraded":"2024-03-04T12:25:06.936880+0000","last_fullsized":"2024-03-04T12:25:06.936880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:25.459417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935561+0000","last_change":"2024-03-04T12:23:16.946380+0000","last_active":"2024-03-04T12:25:06.935561+0000","last_peered":"2024-03-04T12:25:06.935561+0000","last_clean":"2024-03-04T12:25:06.935561+0000","last_became_active":"2024-03-04T12:23:16.945983+0000","last_became_peered":"2024-03-04T12:23:16.945983+0000","last_unstale":"2024-03-04T12:25:06.935561+0000","last_undegraded":"2024-03-04T12:25:06.935561+0000","last_fullsized":"2024-03-04T12:25:06.935561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:39:43.297619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036989+0000","last_change":"2024-03-04T12:23:11.897176+0000","last_active":"2024-03-04T12:25:07.036989+0000","last_peered":"2024-03-04T12:25:07.036989+0000","last_clean":"2024-03-04T12:25:07.036989+0000","last_became_active":"2024-03-04T12:23:11.896747+0000","last_became_peered":"2024-03-04T12:23:11.896747+0000","last_unstale":"2024-03-04T12:25:07.036989+0000","last_undegraded":"2024-03-04T12:25:07.036989+0000","last_fullsized":"2024-03-04T12:25:07.036989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:02.206704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939064+0000","last_change":"2024-03-04T12:23:12.922021+0000","last_active":"2024-03-04T12:25:06.939064+0000","last_peered":"2024-03-04T12:25:06.939064+0000","last_clean":"2024-03-04T12:25:06.939064+0000","last_became_active":"2024-03-04T12:23:12.918724+0000","last_became_peered":"2024-03-04T12:23:12.918724+0000","last_unstale":"2024-03-04T12:25:06.939064+0000","last_undegraded":"2024-03-04T12:25:06.939064+0000","last_fullsized":"2024-03-04T12:25:06.939064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:49:09.017979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.934143+0000","last_change":"2024-03-04T12:23:16.943271+0000","last_active":"2024-03-04T12:25:06.934143+0000","last_peered":"2024-03-04T12:25:06.934143+0000","last_clean":"2024-03-04T12:25:06.934143+0000","last_became_active":"2024-03-04T12:23:16.942400+0000","last_became_peered":"2024-03-04T12:23:16.942400+0000","last_unstale":"2024-03-04T12:25:06.934143+0000","last_undegraded":"2024-03-04T12:25:06.934143+0000","last_fullsized":"2024-03-04T12:25:06.934143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:06.010960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933398+0000","last_change":"2024-03-04T12:23:11.896706+0000","last_active":"2024-03-04T12:25:06.933398+0000","last_peered":"2024-03-04T12:25:06.933398+0000","last_clean":"2024-03-04T12:25:06.933398+0000","last_became_active":"2024-03-04T12:23:11.896100+0000","last_became_peered":"2024-03-04T12:23:11.896100+0000","last_unstale":"2024-03-04T12:25:06.933398+0000","last_undegraded":"2024-03-04T12:25:06.933398+0000","last_fullsized":"2024-03-04T12:25:06.933398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:42:19.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.937310+0000","last_change":"2024-03-04T12:23:12.923631+0000","last_active":"2024-03-04T12:25:06.937310+0000","last_peered":"2024-03-04T12:25:06.937310+0000","last_clean":"2024-03-04T12:25:06.937310+0000","last_became_active":"2024-03-04T12:23:12.922487+0000","last_became_peered":"2024-03-04T12:23:12.922487+0000","last_unstale":"2024-03-04T12:25:06.937310+0000","last_undegraded":"2024-03-04T12:25:06.937310+0000","last_fullsized":"2024-03-04T12:25:06.937310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:30:33.923064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.025491+0000","last_change":"2024-03-04T12:24:06.032075+0000","last_active":"2024-03-04T12:25:07.025491+0000","last_peered":"2024-03-04T12:25:07.025491+0000","last_clean":"2024-03-04T12:25:07.025491+0000","last_became_active":"2024-03-04T12:24:06.031639+0000","last_became_peered":"2024-03-04T12:24:06.031639+0000","last_unstale":"2024-03-04T12:25:07.025491+0000","last_undegraded":"2024-03-04T12:25:07.025491+0000","last_fullsized":"2024-03-04T12:25:07.025491+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:44:48.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'39","reported_seq":316,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:37.070833+0000","last_change":"2024-03-04T12:23:11.898903+0000","last_active":"2024-03-04T12:25:37.070833+0000","last_peered":"2024-03-04T12:25:37.070833+0000","last_clean":"2024-03-04T12:25:37.070833+0000","last_became_active":"2024-03-04T12:23:11.898477+0000","last_became_peered":"2024-03-04T12:23:11.898477+0000","last_unstale":"2024-03-04T12:25:37.070833+0000","last_undegraded":"2024-03-04T12:25:37.070833+0000","last_fullsized":"2024-03-04T12:25:37.070833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:36:58.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60826,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":245,"num_read_kb":182,"num_write":41,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938545+0000","last_change":"2024-03-04T12:23:12.908497+0000","last_active":"2024-03-04T12:25:06.938545+0000","last_peered":"2024-03-04T12:25:06.938545+0000","last_clean":"2024-03-04T12:25:06.938545+0000","last_became_active":"2024-03-04T12:23:12.907847+0000","last_became_peered":"2024-03-04T12:23:12.907847+0000","last_unstale":"2024-03-04T12:25:06.938545+0000","last_undegraded":"2024-03-04T12:25:06.938545+0000","last_fullsized":"2024-03-04T12:25:06.938545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:51:44.592851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933682+0000","last_change":"2024-03-04T12:23:16.951844+0000","last_active":"2024-03-04T12:25:06.933682+0000","last_peered":"2024-03-04T12:25:06.933682+0000","last_clean":"2024-03-04T12:25:06.933682+0000","last_became_active":"2024-03-04T12:23:16.951491+0000","last_became_peered":"2024-03-04T12:23:16.951491+0000","last_unstale":"2024-03-04T12:25:06.933682+0000","last_undegraded":"2024-03-04T12:25:06.933682+0000","last_fullsized":"2024-03-04T12:25:06.933682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:54.098239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939315+0000","last_change":"2024-03-04T12:23:11.899592+0000","last_active":"2024-03-04T12:25:06.939315+0000","last_peered":"2024-03-04T12:25:06.939315+0000","last_clean":"2024-03-04T12:25:06.939315+0000","last_became_active":"2024-03-04T12:23:11.899275+0000","last_became_peered":"2024-03-04T12:23:11.899275+0000","last_unstale":"2024-03-04T12:25:06.939315+0000","last_undegraded":"2024-03-04T12:25:06.939315+0000","last_fullsized":"2024-03-04T12:25:06.939315+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:02.184268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932222+0000","last_change":"2024-03-04T12:23:12.910084+0000","last_active":"2024-03-04T12:25:06.932222+0000","last_peered":"2024-03-04T12:25:06.932222+0000","last_clean":"2024-03-04T12:25:06.932222+0000","last_became_active":"2024-03-04T12:23:12.909719+0000","last_became_peered":"2024-03-04T12:23:12.909719+0000","last_unstale":"2024-03-04T12:25:06.932222+0000","last_undegraded":"2024-03-04T12:25:06.932222+0000","last_fullsized":"2024-03-04T12:25:06.932222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:16:41.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276279+0000","last_change":"2024-03-04T12:24:06.029579+0000","last_active":"2024-03-04T12:25:07.276279+0000","last_peered":"2024-03-04T12:25:07.276279+0000","last_clean":"2024-03-04T12:25:07.276279+0000","last_became_active":"2024-03-04T12:24:06.029042+0000","last_became_peered":"2024-03-04T12:24:06.029042+0000","last_unstale":"2024-03-04T12:25:07.276279+0000","last_undegraded":"2024-03-04T12:25:07.276279+0000","last_fullsized":"2024-03-04T12:25:07.276279+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:59:35.867212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933484+0000","last_change":"2024-03-04T12:23:11.897506+0000","last_active":"2024-03-04T12:25:06.933484+0000","last_peered":"2024-03-04T12:25:06.933484+0000","last_clean":"2024-03-04T12:25:06.933484+0000","last_became_active":"2024-03-04T12:23:11.897090+0000","last_became_peered":"2024-03-04T12:23:11.897090+0000","last_unstale":"2024-03-04T12:25:06.933484+0000","last_undegraded":"2024-03-04T12:25:06.933484+0000","last_fullsized":"2024-03-04T12:25:06.933484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:10:27.868713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037179+0000","last_change":"2024-03-04T12:23:12.921825+0000","last_active":"2024-03-04T12:25:07.037179+0000","last_peered":"2024-03-04T12:25:07.037179+0000","last_clean":"2024-03-04T12:25:07.037179+0000","last_became_active":"2024-03-04T12:23:12.921491+0000","last_became_peered":"2024-03-04T12:23:12.921491+0000","last_unstale":"2024-03-04T12:25:07.037179+0000","last_undegraded":"2024-03-04T12:25:07.037179+0000","last_fullsized":"2024-03-04T12:25:07.037179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:17:08.203502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935989+0000","last_change":"2024-03-04T12:23:16.951920+0000","last_active":"2024-03-04T12:25:06.935989+0000","last_peered":"2024-03-04T12:25:06.935989+0000","last_clean":"2024-03-04T12:25:06.935989+0000","last_became_active":"2024-03-04T12:23:16.951507+0000","last_became_peered":"2024-03-04T12:23:16.951507+0000","last_unstale":"2024-03-04T12:25:06.935989+0000","last_undegraded":"2024-03-04T12:25:06.935989+0000","last_fullsized":"2024-03-04T12:25:06.935989+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:27.673692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":161,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:36.068598+0000","last_change":"2024-03-04T12:23:11.897722+0000","last_active":"2024-03-04T12:25:36.068598+0000","last_peered":"2024-03-04T12:25:36.068598+0000","last_clean":"2024-03-04T12:25:36.068598+0000","last_became_active":"2024-03-04T12:23:11.897276+0000","last_became_peered":"2024-03-04T12:23:11.897276+0000","last_unstale":"2024-03-04T12:25:36.068598+0000","last_undegraded":"2024-03-04T12:25:36.068598+0000","last_fullsized":"2024-03-04T12:25:36.068598+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:37.968804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036780+0000","last_change":"2024-03-04T12:23:12.914773+0000","last_active":"2024-03-04T12:25:07.036780+0000","last_peered":"2024-03-04T12:25:07.036780+0000","last_clean":"2024-03-04T12:25:07.036780+0000","last_became_active":"2024-03-04T12:23:12.914431+0000","last_became_peered":"2024-03-04T12:23:12.914431+0000","last_unstale":"2024-03-04T12:25:07.036780+0000","last_undegraded":"2024-03-04T12:25:07.036780+0000","last_fullsized":"2024-03-04T12:25:07.036780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:50.234491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932697+0000","last_change":"2024-03-04T12:23:16.952405+0000","last_active":"2024-03-04T12:25:06.932697+0000","last_peered":"2024-03-04T12:25:06.932697+0000","last_clean":"2024-03-04T12:25:06.932697+0000","last_became_active":"2024-03-04T12:23:16.951915+0000","last_became_peered":"2024-03-04T12:23:16.951915+0000","last_unstale":"2024-03-04T12:25:06.932697+0000","last_undegraded":"2024-03-04T12:25:06.932697+0000","last_fullsized":"2024-03-04T12:25:06.932697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:38:03.765883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938109+0000","last_change":"2024-03-04T12:23:11.903267+0000","last_active":"2024-03-04T12:25:06.938109+0000","last_peered":"2024-03-04T12:25:06.938109+0000","last_clean":"2024-03-04T12:25:06.938109+0000","last_became_active":"2024-03-04T12:23:11.902870+0000","last_became_peered":"2024-03-04T12:23:11.902870+0000","last_unstale":"2024-03-04T12:25:06.938109+0000","last_undegraded":"2024-03-04T12:25:06.938109+0000","last_fullsized":"2024-03-04T12:25:06.938109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:33:43.679869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933267+0000","last_change":"2024-03-04T12:23:12.925303+0000","last_active":"2024-03-04T12:25:06.933267+0000","last_peered":"2024-03-04T12:25:06.933267+0000","last_clean":"2024-03-04T12:25:06.933267+0000","last_became_active":"2024-03-04T12:23:12.924945+0000","last_became_peered":"2024-03-04T12:23:12.924945+0000","last_unstale":"2024-03-04T12:25:06.933267+0000","last_undegraded":"2024-03-04T12:25:06.933267+0000","last_fullsized":"2024-03-04T12:25:06.933267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:49.680925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933148+0000","last_change":"2024-03-04T12:23:16.951457+0000","last_active":"2024-03-04T12:25:06.933148+0000","last_peered":"2024-03-04T12:25:06.933148+0000","last_clean":"2024-03-04T12:25:06.933148+0000","last_became_active":"2024-03-04T12:23:16.950835+0000","last_became_peered":"2024-03-04T12:23:16.950835+0000","last_unstale":"2024-03-04T12:25:06.933148+0000","last_undegraded":"2024-03-04T12:25:06.933148+0000","last_fullsized":"2024-03-04T12:25:06.933148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:39.337275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:29.397242+0000","last_change":"2024-03-04T12:23:11.903718+0000","last_active":"2024-03-04T12:25:29.397242+0000","last_peered":"2024-03-04T12:25:29.397242+0000","last_clean":"2024-03-04T12:25:29.397242+0000","last_became_active":"2024-03-04T12:23:11.902814+0000","last_became_peered":"2024-03-04T12:23:11.902814+0000","last_unstale":"2024-03-04T12:25:29.397242+0000","last_undegraded":"2024-03-04T12:25:29.397242+0000","last_fullsized":"2024-03-04T12:25:29.397242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:40:30.978576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276785+0000","last_change":"2024-03-04T12:24:05.958392+0000","last_active":"2024-03-04T12:25:07.276785+0000","last_peered":"2024-03-04T12:25:07.276785+0000","last_clean":"2024-03-04T12:25:07.276785+0000","last_became_active":"2024-03-04T12:24:05.958012+0000","last_became_peered":"2024-03-04T12:24:05.958012+0000","last_unstale":"2024-03-04T12:25:07.276785+0000","last_undegraded":"2024-03-04T12:25:07.276785+0000","last_fullsized":"2024-03-04T12:25:07.276785+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:43:50.400090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938459+0000","last_change":"2024-03-04T12:23:16.950046+0000","last_active":"2024-03-04T12:25:06.938459+0000","last_peered":"2024-03-04T12:25:06.938459+0000","last_clean":"2024-03-04T12:25:06.938459+0000","last_became_active":"2024-03-04T12:23:16.949632+0000","last_became_peered":"2024-03-04T12:23:16.949632+0000","last_unstale":"2024-03-04T12:25:06.938459+0000","last_undegraded":"2024-03-04T12:25:06.938459+0000","last_fullsized":"2024-03-04T12:25:06.938459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:25.817212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935925+0000","last_change":"2024-03-04T12:23:11.897016+0000","last_active":"2024-03-04T12:25:06.935925+0000","last_peered":"2024-03-04T12:25:06.935925+0000","last_clean":"2024-03-04T12:25:06.935925+0000","last_became_active":"2024-03-04T12:23:11.896668+0000","last_became_peered":"2024-03-04T12:23:11.896668+0000","last_unstale":"2024-03-04T12:25:06.935925+0000","last_undegraded":"2024-03-04T12:25:06.935925+0000","last_fullsized":"2024-03-04T12:25:06.935925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:50:01.089658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.026051+0000","last_change":"2024-03-04T12:23:12.908169+0000","last_active":"2024-03-04T12:25:07.026051+0000","last_peered":"2024-03-04T12:25:07.026051+0000","last_clean":"2024-03-04T12:25:07.026051+0000","last_became_active":"2024-03-04T12:23:12.907786+0000","last_became_peered":"2024-03-04T12:23:12.907786+0000","last_unstale":"2024-03-04T12:25:07.026051+0000","last_undegraded":"2024-03-04T12:25:07.026051+0000","last_fullsized":"2024-03-04T12:25:07.026051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:10.602216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932425+0000","last_change":"2024-03-04T12:23:16.944232+0000","last_active":"2024-03-04T12:25:06.932425+0000","last_peered":"2024-03-04T12:25:06.932425+0000","last_clean":"2024-03-04T12:25:06.932425+0000","last_became_active":"2024-03-04T12:23:16.943855+0000","last_became_peered":"2024-03-04T12:23:16.943855+0000","last_unstale":"2024-03-04T12:25:06.932425+0000","last_undegraded":"2024-03-04T12:25:06.932425+0000","last_fullsized":"2024-03-04T12:25:06.932425+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:37:12.083622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":155,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:37.075695+0000","last_change":"2024-03-04T12:23:11.899742+0000","last_active":"2024-03-04T12:25:37.075695+0000","last_peered":"2024-03-04T12:25:37.075695+0000","last_clean":"2024-03-04T12:25:37.075695+0000","last_became_active":"2024-03-04T12:23:11.898445+0000","last_became_peered":"2024-03-04T12:23:11.898445+0000","last_unstale":"2024-03-04T12:25:37.075695+0000","last_undegraded":"2024-03-04T12:25:37.075695+0000","last_fullsized":"2024-03-04T12:25:37.075695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:13.128348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.025878+0000","last_change":"2024-03-04T12:23:12.921321+0000","last_active":"2024-03-04T12:25:07.025878+0000","last_peered":"2024-03-04T12:25:07.025878+0000","last_clean":"2024-03-04T12:25:07.025878+0000","last_became_active":"2024-03-04T12:23:12.918119+0000","last_became_peered":"2024-03-04T12:23:12.918119+0000","last_unstale":"2024-03-04T12:25:07.025878+0000","last_undegraded":"2024-03-04T12:25:07.025878+0000","last_fullsized":"2024-03-04T12:25:07.025878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:31.693583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932634+0000","last_change":"2024-03-04T12:23:16.943887+0000","last_active":"2024-03-04T12:25:06.932634+0000","last_peered":"2024-03-04T12:25:06.932634+0000","last_clean":"2024-03-04T12:25:06.932634+0000","last_became_active":"2024-03-04T12:23:16.943487+0000","last_became_peered":"2024-03-04T12:23:16.943487+0000","last_unstale":"2024-03-04T12:25:06.932634+0000","last_undegraded":"2024-03-04T12:25:06.932634+0000","last_fullsized":"2024-03-04T12:25:06.932634+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:12.451176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933359+0000","last_change":"2024-03-04T12:23:12.918363+0000","last_active":"2024-03-04T12:25:06.933359+0000","last_peered":"2024-03-04T12:25:06.933359+0000","last_clean":"2024-03-04T12:25:06.933359+0000","last_became_active":"2024-03-04T12:23:12.917563+0000","last_became_peered":"2024-03-04T12:23:12.917563+0000","last_unstale":"2024-03-04T12:25:06.933359+0000","last_undegraded":"2024-03-04T12:25:06.933359+0000","last_fullsized":"2024-03-04T12:25:06.933359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:26:35.297513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'7","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:38.071274+0000","last_change":"2024-03-04T12:23:11.904072+0000","last_active":"2024-03-04T12:25:38.071274+0000","last_peered":"2024-03-04T12:25:38.071274+0000","last_clean":"2024-03-04T12:25:38.071274+0000","last_became_active":"2024-03-04T12:23:11.903097+0000","last_became_peered":"2024-03-04T12:23:11.903097+0000","last_unstale":"2024-03-04T12:25:38.071274+0000","last_undegraded":"2024-03-04T12:25:38.071274+0000","last_fullsized":"2024-03-04T12:25:38.071274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:36.606667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036799+0000","last_change":"2024-03-04T12:23:16.951019+0000","last_active":"2024-03-04T12:25:07.036799+0000","last_peered":"2024-03-04T12:25:07.036799+0000","last_clean":"2024-03-04T12:25:07.036799+0000","last_became_active":"2024-03-04T12:23:16.950524+0000","last_became_peered":"2024-03-04T12:23:16.950524+0000","last_unstale":"2024-03-04T12:25:07.036799+0000","last_undegraded":"2024-03-04T12:25:07.036799+0000","last_fullsized":"2024-03-04T12:25:07.036799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:48.376539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938226+0000","last_change":"2024-03-04T12:23:12.927615+0000","last_active":"2024-03-04T12:25:06.938226+0000","last_peered":"2024-03-04T12:25:06.938226+0000","last_clean":"2024-03-04T12:25:06.938226+0000","last_became_active":"2024-03-04T12:23:12.927190+0000","last_became_peered":"2024-03-04T12:23:12.927190+0000","last_unstale":"2024-03-04T12:25:06.938226+0000","last_undegraded":"2024-03-04T12:25:06.938226+0000","last_fullsized":"2024-03-04T12:25:06.938226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:54:19.019030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933773+0000","last_change":"2024-03-04T12:23:11.893542+0000","last_active":"2024-03-04T12:25:06.933773+0000","last_peered":"2024-03-04T12:25:06.933773+0000","last_clean":"2024-03-04T12:25:06.933773+0000","last_became_active":"2024-03-04T12:23:11.893166+0000","last_became_peered":"2024-03-04T12:23:11.893166+0000","last_unstale":"2024-03-04T12:25:06.933773+0000","last_undegraded":"2024-03-04T12:25:06.933773+0000","last_fullsized":"2024-03-04T12:25:06.933773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:32.117863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036450+0000","last_change":"2024-03-04T12:21:55.865103+0000","last_active":"2024-03-04T12:25:07.036450+0000","last_peered":"2024-03-04T12:25:07.036450+0000","last_clean":"2024-03-04T12:25:07.036450+0000","last_became_active":"2024-03-04T12:21:55.864553+0000","last_became_peered":"2024-03-04T12:21:55.864553+0000","last_unstale":"2024-03-04T12:25:07.036450+0000","last_undegraded":"2024-03-04T12:25:07.036450+0000","last_fullsized":"2024-03-04T12:25:07.036450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_clean_scrub_stamp":"2024-03-04T12:18:28.442423+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:31.052782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933731+0000","last_change":"2024-03-04T12:23:16.945852+0000","last_active":"2024-03-04T12:25:06.933731+0000","last_peered":"2024-03-04T12:25:06.933731+0000","last_clean":"2024-03-04T12:25:06.933731+0000","last_became_active":"2024-03-04T12:23:16.945496+0000","last_became_peered":"2024-03-04T12:23:16.945496+0000","last_unstale":"2024-03-04T12:25:06.933731+0000","last_undegraded":"2024-03-04T12:25:06.933731+0000","last_fullsized":"2024-03-04T12:25:06.933731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:35:10.880757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932050+0000","last_change":"2024-03-04T12:23:12.919219+0000","last_active":"2024-03-04T12:25:06.932050+0000","last_peered":"2024-03-04T12:25:06.932050+0000","last_clean":"2024-03-04T12:25:06.932050+0000","last_became_active":"2024-03-04T12:23:12.918897+0000","last_became_peered":"2024-03-04T12:23:12.918897+0000","last_unstale":"2024-03-04T12:25:06.932050+0000","last_undegraded":"2024-03-04T12:25:06.932050+0000","last_fullsized":"2024-03-04T12:25:06.932050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:24:32.594172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036668+0000","last_change":"2024-03-04T12:23:11.894544+0000","last_active":"2024-03-04T12:25:07.036668+0000","last_peered":"2024-03-04T12:25:07.036668+0000","last_clean":"2024-03-04T12:25:07.036668+0000","last_became_active":"2024-03-04T12:23:11.894188+0000","last_became_peered":"2024-03-04T12:23:11.894188+0000","last_unstale":"2024-03-04T12:25:07.036668+0000","last_undegraded":"2024-03-04T12:25:07.036668+0000","last_fullsized":"2024-03-04T12:25:07.036668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:28:14.687530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276193+0000","last_change":"2024-03-04T12:23:16.945510+0000","last_active":"2024-03-04T12:25:07.276193+0000","last_peered":"2024-03-04T12:25:07.276193+0000","last_clean":"2024-03-04T12:25:07.276193+0000","last_became_active":"2024-03-04T12:23:16.945140+0000","last_became_peered":"2024-03-04T12:23:16.945140+0000","last_unstale":"2024-03-04T12:25:07.276193+0000","last_undegraded":"2024-03-04T12:25:07.276193+0000","last_fullsized":"2024-03-04T12:25:07.276193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:44.102503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938493+0000","last_change":"2024-03-04T12:23:11.904515+0000","last_active":"2024-03-04T12:25:06.938493+0000","last_peered":"2024-03-04T12:25:06.938493+0000","last_clean":"2024-03-04T12:25:06.938493+0000","last_became_active":"2024-03-04T12:23:11.904149+0000","last_became_peered":"2024-03-04T12:23:11.904149+0000","last_unstale":"2024-03-04T12:25:06.938493+0000","last_undegraded":"2024-03-04T12:25:06.938493+0000","last_fullsized":"2024-03-04T12:25:06.938493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:04:51.382626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932543+0000","last_change":"2024-03-04T12:23:12.908749+0000","last_active":"2024-03-04T12:25:06.932543+0000","last_peered":"2024-03-04T12:25:06.932543+0000","last_clean":"2024-03-04T12:25:06.932543+0000","last_became_active":"2024-03-04T12:23:12.908405+0000","last_became_peered":"2024-03-04T12:23:12.908405+0000","last_unstale":"2024-03-04T12:25:06.932543+0000","last_undegraded":"2024-03-04T12:25:06.932543+0000","last_fullsized":"2024-03-04T12:25:06.932543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:10.472953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938204+0000","last_change":"2024-03-04T12:23:16.951145+0000","last_active":"2024-03-04T12:25:06.938204+0000","last_peered":"2024-03-04T12:25:06.938204+0000","last_clean":"2024-03-04T12:25:06.938204+0000","last_became_active":"2024-03-04T12:23:16.950702+0000","last_became_peered":"2024-03-04T12:23:16.950702+0000","last_unstale":"2024-03-04T12:25:06.938204+0000","last_undegraded":"2024-03-04T12:25:06.938204+0000","last_fullsized":"2024-03-04T12:25:06.938204+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:42.720616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'14","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:36.932982+0000","last_change":"2024-03-04T12:25:07.284006+0000","last_active":"2024-03-04T12:25:36.932982+0000","last_peered":"2024-03-04T12:25:36.932982+0000","last_clean":"2024-03-04T12:25:36.932982+0000","last_became_active":"2024-03-04T12:25:07.277446+0000","last_became_peered":"2024-03-04T12:25:07.277446+0000","last_unstale":"2024-03-04T12:25:36.932982+0000","last_undegraded":"2024-03-04T12:25:36.932982+0000","last_fullsized":"2024-03-04T12:25:36.932982+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:08:40.719222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938663+0000","last_change":"2024-03-04T12:23:12.925763+0000","last_active":"2024-03-04T12:25:06.938663+0000","last_peered":"2024-03-04T12:25:06.938663+0000","last_clean":"2024-03-04T12:25:06.938663+0000","last_became_active":"2024-03-04T12:23:12.925429+0000","last_became_peered":"2024-03-04T12:23:12.925429+0000","last_unstale":"2024-03-04T12:25:06.938663+0000","last_undegraded":"2024-03-04T12:25:06.938663+0000","last_fullsized":"2024-03-04T12:25:06.938663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:46.814459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939067+0000","last_change":"2024-03-04T12:23:16.942624+0000","last_active":"2024-03-04T12:25:06.939067+0000","last_peered":"2024-03-04T12:25:06.939067+0000","last_clean":"2024-03-04T12:25:06.939067+0000","last_became_active":"2024-03-04T12:23:16.942287+0000","last_became_peered":"2024-03-04T12:23:16.942287+0000","last_unstale":"2024-03-04T12:25:06.939067+0000","last_undegraded":"2024-03-04T12:25:06.939067+0000","last_fullsized":"2024-03-04T12:25:06.939067+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:59.479611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933762+0000","last_change":"2024-03-04T12:23:11.891504+0000","last_active":"2024-03-04T12:25:06.933762+0000","last_peered":"2024-03-04T12:25:06.933762+0000","last_clean":"2024-03-04T12:25:06.933762+0000","last_became_active":"2024-03-04T12:23:11.891134+0000","last_became_peered":"2024-03-04T12:23:11.891134+0000","last_unstale":"2024-03-04T12:25:06.933762+0000","last_undegraded":"2024-03-04T12:25:06.933762+0000","last_fullsized":"2024-03-04T12:25:06.933762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:41.481003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940405+0000","last_change":"2024-03-04T12:23:12.928989+0000","last_active":"2024-03-04T12:25:06.940405+0000","last_peered":"2024-03-04T12:25:06.940405+0000","last_clean":"2024-03-04T12:25:06.940405+0000","last_became_active":"2024-03-04T12:23:12.928609+0000","last_became_peered":"2024-03-04T12:23:12.928609+0000","last_unstale":"2024-03-04T12:25:06.940405+0000","last_undegraded":"2024-03-04T12:25:06.940405+0000","last_fullsized":"2024-03-04T12:25:06.940405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:39.979546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933694+0000","last_change":"2024-03-04T12:23:16.949590+0000","last_active":"2024-03-04T12:25:06.933694+0000","last_peered":"2024-03-04T12:25:06.933694+0000","last_clean":"2024-03-04T12:25:06.933694+0000","last_became_active":"2024-03-04T12:23:16.949086+0000","last_became_peered":"2024-03-04T12:23:16.949086+0000","last_unstale":"2024-03-04T12:25:06.933694+0000","last_undegraded":"2024-03-04T12:25:06.933694+0000","last_fullsized":"2024-03-04T12:25:06.933694+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:07:09.819097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.286558+0000","last_change":"2024-03-04T12:25:07.286087+0000","last_active":"2024-03-04T12:25:07.286558+0000","last_peered":"2024-03-04T12:25:07.286558+0000","last_clean":"2024-03-04T12:25:07.286558+0000","last_became_active":"2024-03-04T12:25:07.277551+0000","last_became_peered":"2024-03-04T12:25:07.277551+0000","last_unstale":"2024-03-04T12:25:07.286558+0000","last_undegraded":"2024-03-04T12:25:07.286558+0000","last_fullsized":"2024-03-04T12:25:07.286558+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:23.167325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938356+0000","last_change":"2024-03-04T12:23:12.907403+0000","last_active":"2024-03-04T12:25:06.938356+0000","last_peered":"2024-03-04T12:25:06.938356+0000","last_clean":"2024-03-04T12:25:06.938356+0000","last_became_active":"2024-03-04T12:23:12.906674+0000","last_became_peered":"2024-03-04T12:23:12.906674+0000","last_unstale":"2024-03-04T12:25:06.938356+0000","last_undegraded":"2024-03-04T12:25:06.938356+0000","last_fullsized":"2024-03-04T12:25:06.938356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:49:01.133676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936150+0000","last_change":"2024-03-04T12:23:16.955113+0000","last_active":"2024-03-04T12:25:06.936150+0000","last_peered":"2024-03-04T12:25:06.936150+0000","last_clean":"2024-03-04T12:25:06.936150+0000","last_became_active":"2024-03-04T12:23:16.951624+0000","last_became_peered":"2024-03-04T12:23:16.951624+0000","last_unstale":"2024-03-04T12:25:06.936150+0000","last_undegraded":"2024-03-04T12:25:06.936150+0000","last_fullsized":"2024-03-04T12:25:06.936150+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:22:23.935460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938252+0000","last_change":"2024-03-04T12:23:11.900213+0000","last_active":"2024-03-04T12:25:06.938252+0000","last_peered":"2024-03-04T12:25:06.938252+0000","last_clean":"2024-03-04T12:25:06.938252+0000","last_became_active":"2024-03-04T12:23:11.899832+0000","last_became_peered":"2024-03-04T12:23:11.899832+0000","last_unstale":"2024-03-04T12:25:06.938252+0000","last_undegraded":"2024-03-04T12:25:06.938252+0000","last_fullsized":"2024-03-04T12:25:06.938252+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:55:24.553309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276791+0000","last_change":"2024-03-04T12:23:12.913315+0000","last_active":"2024-03-04T12:25:07.276791+0000","last_peered":"2024-03-04T12:25:07.276791+0000","last_clean":"2024-03-04T12:25:07.276791+0000","last_became_active":"2024-03-04T12:23:12.912850+0000","last_became_peered":"2024-03-04T12:23:12.912850+0000","last_unstale":"2024-03-04T12:25:07.276791+0000","last_undegraded":"2024-03-04T12:25:07.276791+0000","last_fullsized":"2024-03-04T12:25:07.276791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:55.140866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933591+0000","last_change":"2024-03-04T12:23:16.943295+0000","last_active":"2024-03-04T12:25:06.933591+0000","last_peered":"2024-03-04T12:25:06.933591+0000","last_clean":"2024-03-04T12:25:06.933591+0000","last_became_active":"2024-03-04T12:23:16.942853+0000","last_became_peered":"2024-03-04T12:23:16.942853+0000","last_unstale":"2024-03-04T12:25:06.933591+0000","last_undegraded":"2024-03-04T12:25:06.933591+0000","last_fullsized":"2024-03-04T12:25:06.933591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:05:51.162179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:22.480402+0000","last_change":"2024-03-04T12:23:11.897870+0000","last_active":"2024-03-04T12:25:22.480402+0000","last_peered":"2024-03-04T12:25:22.480402+0000","last_clean":"2024-03-04T12:25:22.480402+0000","last_became_active":"2024-03-04T12:23:11.897443+0000","last_became_peered":"2024-03-04T12:23:11.897443+0000","last_unstale":"2024-03-04T12:25:22.480402+0000","last_undegraded":"2024-03-04T12:25:22.480402+0000","last_fullsized":"2024-03-04T12:25:22.480402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:49:37.875541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938638+0000","last_change":"2024-03-04T12:23:12.916441+0000","last_active":"2024-03-04T12:25:06.938638+0000","last_peered":"2024-03-04T12:25:06.938638+0000","last_clean":"2024-03-04T12:25:06.938638+0000","last_became_active":"2024-03-04T12:23:12.916078+0000","last_became_peered":"2024-03-04T12:23:12.916078+0000","last_unstale":"2024-03-04T12:25:06.938638+0000","last_undegraded":"2024-03-04T12:25:06.938638+0000","last_fullsized":"2024-03-04T12:25:06.938638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:45:23.663627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933671+0000","last_change":"2024-03-04T12:23:16.953341+0000","last_active":"2024-03-04T12:25:06.933671+0000","last_peered":"2024-03-04T12:25:06.933671+0000","last_clean":"2024-03-04T12:25:06.933671+0000","last_became_active":"2024-03-04T12:23:16.952878+0000","last_became_peered":"2024-03-04T12:23:16.952878+0000","last_unstale":"2024-03-04T12:25:06.933671+0000","last_undegraded":"2024-03-04T12:25:06.933671+0000","last_fullsized":"2024-03-04T12:25:06.933671+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:08:55.575263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938122+0000","last_change":"2024-03-04T12:23:11.903182+0000","last_active":"2024-03-04T12:25:06.938122+0000","last_peered":"2024-03-04T12:25:06.938122+0000","last_clean":"2024-03-04T12:25:06.938122+0000","last_became_active":"2024-03-04T12:23:11.902709+0000","last_became_peered":"2024-03-04T12:23:11.902709+0000","last_unstale":"2024-03-04T12:25:06.938122+0000","last_undegraded":"2024-03-04T12:25:06.938122+0000","last_fullsized":"2024-03-04T12:25:06.938122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:13:33.438929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037059+0000","last_change":"2024-03-04T12:23:12.931339+0000","last_active":"2024-03-04T12:25:07.037059+0000","last_peered":"2024-03-04T12:25:07.037059+0000","last_clean":"2024-03-04T12:25:07.037059+0000","last_became_active":"2024-03-04T12:23:12.930213+0000","last_became_peered":"2024-03-04T12:23:12.930213+0000","last_unstale":"2024-03-04T12:25:07.037059+0000","last_undegraded":"2024-03-04T12:25:07.037059+0000","last_fullsized":"2024-03-04T12:25:07.037059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:55:23.084139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936423+0000","last_change":"2024-03-04T12:23:16.948723+0000","last_active":"2024-03-04T12:25:06.936423+0000","last_peered":"2024-03-04T12:25:06.936423+0000","last_clean":"2024-03-04T12:25:06.936423+0000","last_became_active":"2024-03-04T12:23:16.948256+0000","last_became_peered":"2024-03-04T12:23:16.948256+0000","last_unstale":"2024-03-04T12:25:06.936423+0000","last_undegraded":"2024-03-04T12:25:06.936423+0000","last_fullsized":"2024-03-04T12:25:06.936423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:22.060517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":155,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:36.081533+0000","last_change":"2024-03-04T12:23:11.898475+0000","last_active":"2024-03-04T12:25:36.081533+0000","last_peered":"2024-03-04T12:25:36.081533+0000","last_clean":"2024-03-04T12:25:36.081533+0000","last_became_active":"2024-03-04T12:23:11.898117+0000","last_became_peered":"2024-03-04T12:23:11.898117+0000","last_unstale":"2024-03-04T12:25:36.081533+0000","last_undegraded":"2024-03-04T12:25:36.081533+0000","last_fullsized":"2024-03-04T12:25:36.081533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:21:25.119591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037220+0000","last_change":"2024-03-04T12:23:12.929772+0000","last_active":"2024-03-04T12:25:07.037220+0000","last_peered":"2024-03-04T12:25:07.037220+0000","last_clean":"2024-03-04T12:25:07.037220+0000","last_became_active":"2024-03-04T12:23:12.929405+0000","last_became_peered":"2024-03-04T12:23:12.929405+0000","last_unstale":"2024-03-04T12:25:07.037220+0000","last_undegraded":"2024-03-04T12:25:07.037220+0000","last_fullsized":"2024-03-04T12:25:07.037220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:22:42.874989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940260+0000","last_change":"2024-03-04T12:23:16.941715+0000","last_active":"2024-03-04T12:25:06.940260+0000","last_peered":"2024-03-04T12:25:06.940260+0000","last_clean":"2024-03-04T12:25:06.940260+0000","last_became_active":"2024-03-04T12:23:16.941298+0000","last_became_peered":"2024-03-04T12:23:16.941298+0000","last_unstale":"2024-03-04T12:25:06.940260+0000","last_undegraded":"2024-03-04T12:25:06.940260+0000","last_fullsized":"2024-03-04T12:25:06.940260+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:09:02.019531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933616+0000","last_change":"2024-03-04T12:23:11.896413+0000","last_active":"2024-03-04T12:25:06.933616+0000","last_peered":"2024-03-04T12:25:06.933616+0000","last_clean":"2024-03-04T12:25:06.933616+0000","last_became_active":"2024-03-04T12:23:11.895679+0000","last_became_peered":"2024-03-04T12:23:11.895679+0000","last_unstale":"2024-03-04T12:25:06.933616+0000","last_undegraded":"2024-03-04T12:25:06.933616+0000","last_fullsized":"2024-03-04T12:25:06.933616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:25:41.967530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036828+0000","last_change":"2024-03-04T12:23:12.920468+0000","last_active":"2024-03-04T12:25:07.036828+0000","last_peered":"2024-03-04T12:25:07.036828+0000","last_clean":"2024-03-04T12:25:07.036828+0000","last_became_active":"2024-03-04T12:23:12.920113+0000","last_became_peered":"2024-03-04T12:23:12.920113+0000","last_unstale":"2024-03-04T12:25:07.036828+0000","last_undegraded":"2024-03-04T12:25:07.036828+0000","last_fullsized":"2024-03-04T12:25:07.036828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:00:02.283518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.937863+0000","last_change":"2024-03-04T12:24:06.030444+0000","last_active":"2024-03-04T12:25:06.937863+0000","last_peered":"2024-03-04T12:25:06.937863+0000","last_clean":"2024-03-04T12:25:06.937863+0000","last_became_active":"2024-03-04T12:24:06.029934+0000","last_became_peered":"2024-03-04T12:24:06.029934+0000","last_unstale":"2024-03-04T12:25:06.937863+0000","last_undegraded":"2024-03-04T12:25:06.937863+0000","last_fullsized":"2024-03-04T12:25:06.937863+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:08:16.820509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932650+0000","last_change":"2024-03-04T12:23:11.903323+0000","last_active":"2024-03-04T12:25:06.932650+0000","last_peered":"2024-03-04T12:25:06.932650+0000","last_clean":"2024-03-04T12:25:06.932650+0000","last_became_active":"2024-03-04T12:23:11.902987+0000","last_became_peered":"2024-03-04T12:23:11.902987+0000","last_unstale":"2024-03-04T12:25:06.932650+0000","last_undegraded":"2024-03-04T12:25:06.932650+0000","last_fullsized":"2024-03-04T12:25:06.932650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:14.044391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037158+0000","last_change":"2024-03-04T12:23:12.913491+0000","last_active":"2024-03-04T12:25:07.037158+0000","last_peered":"2024-03-04T12:25:07.037158+0000","last_clean":"2024-03-04T12:25:07.037158+0000","last_became_active":"2024-03-04T12:23:12.912103+0000","last_became_peered":"2024-03-04T12:23:12.912103+0000","last_unstale":"2024-03-04T12:25:07.037158+0000","last_undegraded":"2024-03-04T12:25:07.037158+0000","last_fullsized":"2024-03-04T12:25:07.037158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:05:38.033596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276868+0000","last_change":"2024-03-04T12:23:12.908437+0000","last_active":"2024-03-04T12:25:07.276868+0000","last_peered":"2024-03-04T12:25:07.276868+0000","last_clean":"2024-03-04T12:25:07.276868+0000","last_became_active":"2024-03-04T12:23:12.908071+0000","last_became_peered":"2024-03-04T12:23:12.908071+0000","last_unstale":"2024-03-04T12:25:07.276868+0000","last_undegraded":"2024-03-04T12:25:07.276868+0000","last_fullsized":"2024-03-04T12:25:07.276868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:20.112645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939915+0000","last_change":"2024-03-04T12:23:12.921645+0000","last_active":"2024-03-04T12:25:06.939915+0000","last_peered":"2024-03-04T12:25:06.939915+0000","last_clean":"2024-03-04T12:25:06.939915+0000","last_became_active":"2024-03-04T12:23:12.921275+0000","last_became_peered":"2024-03-04T12:23:12.921275+0000","last_unstale":"2024-03-04T12:25:06.939915+0000","last_undegraded":"2024-03-04T12:25:06.939915+0000","last_fullsized":"2024-03-04T12:25:06.939915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:56:17.740782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939311+0000","last_change":"2024-03-04T12:23:12.919813+0000","last_active":"2024-03-04T12:25:06.939311+0000","last_peered":"2024-03-04T12:25:06.939311+0000","last_clean":"2024-03-04T12:25:06.939311+0000","last_became_active":"2024-03-04T12:23:12.919469+0000","last_became_peered":"2024-03-04T12:23:12.919469+0000","last_unstale":"2024-03-04T12:25:06.939311+0000","last_undegraded":"2024-03-04T12:25:06.939311+0000","last_fullsized":"2024-03-04T12:25:06.939311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:46.720268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940192+0000","last_change":"2024-03-04T12:23:12.918476+0000","last_active":"2024-03-04T12:25:06.940192+0000","last_peered":"2024-03-04T12:25:06.940192+0000","last_clean":"2024-03-04T12:25:06.940192+0000","last_became_active":"2024-03-04T12:23:12.917743+0000","last_became_peered":"2024-03-04T12:23:12.917743+0000","last_unstale":"2024-03-04T12:25:06.940192+0000","last_undegraded":"2024-03-04T12:25:06.940192+0000","last_fullsized":"2024-03-04T12:25:06.940192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:20.246769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932605+0000","last_change":"2024-03-04T12:23:12.918254+0000","last_active":"2024-03-04T12:25:06.932605+0000","last_peered":"2024-03-04T12:25:06.932605+0000","last_clean":"2024-03-04T12:25:06.932605+0000","last_became_active":"2024-03-04T12:23:12.917922+0000","last_became_peered":"2024-03-04T12:23:12.917922+0000","last_unstale":"2024-03-04T12:25:06.932605+0000","last_undegraded":"2024-03-04T12:25:06.932605+0000","last_fullsized":"2024-03-04T12:25:06.932605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:48:20.847984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276507+0000","last_change":"2024-03-04T12:23:12.910368+0000","last_active":"2024-03-04T12:25:07.276507+0000","last_peered":"2024-03-04T12:25:07.276507+0000","last_clean":"2024-03-04T12:25:07.276507+0000","last_became_active":"2024-03-04T12:23:12.910017+0000","last_became_peered":"2024-03-04T12:23:12.910017+0000","last_unstale":"2024-03-04T12:25:07.276507+0000","last_undegraded":"2024-03-04T12:25:07.276507+0000","last_fullsized":"2024-03-04T12:25:07.276507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:15:22.093644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939872+0000","last_change":"2024-03-04T12:23:12.921021+0000","last_active":"2024-03-04T12:25:06.939872+0000","last_peered":"2024-03-04T12:25:06.939872+0000","last_clean":"2024-03-04T12:25:06.939872+0000","last_became_active":"2024-03-04T12:23:12.920600+0000","last_became_peered":"2024-03-04T12:23:12.920600+0000","last_unstale":"2024-03-04T12:25:06.939872+0000","last_undegraded":"2024-03-04T12:25:06.939872+0000","last_fullsized":"2024-03-04T12:25:06.939872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:00:31.706900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939264+0000","last_change":"2024-03-04T12:24:05.957607+0000","last_active":"2024-03-04T12:25:06.939264+0000","last_peered":"2024-03-04T12:25:06.939264+0000","last_clean":"2024-03-04T12:25:06.939264+0000","last_became_active":"2024-03-04T12:24:05.957335+0000","last_became_peered":"2024-03-04T12:24:05.957335+0000","last_unstale":"2024-03-04T12:25:06.939264+0000","last_undegraded":"2024-03-04T12:25:06.939264+0000","last_fullsized":"2024-03-04T12:25:06.939264+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:12.926219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933250+0000","last_change":"2024-03-04T12:23:12.911394+0000","last_active":"2024-03-04T12:25:06.933250+0000","last_peered":"2024-03-04T12:25:06.933250+0000","last_clean":"2024-03-04T12:25:06.933250+0000","last_became_active":"2024-03-04T12:23:12.911061+0000","last_became_peered":"2024-03-04T12:23:12.911061+0000","last_unstale":"2024-03-04T12:25:06.933250+0000","last_undegraded":"2024-03-04T12:25:06.933250+0000","last_fullsized":"2024-03-04T12:25:06.933250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:39.880383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938372+0000","last_change":"2024-03-04T12:23:12.911356+0000","last_active":"2024-03-04T12:25:06.938372+0000","last_peered":"2024-03-04T12:25:06.938372+0000","last_clean":"2024-03-04T12:25:06.938372+0000","last_became_active":"2024-03-04T12:23:12.911029+0000","last_became_peered":"2024-03-04T12:23:12.911029+0000","last_unstale":"2024-03-04T12:25:06.938372+0000","last_undegraded":"2024-03-04T12:25:06.938372+0000","last_fullsized":"2024-03-04T12:25:06.938372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:52.611625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936612+0000","last_change":"2024-03-04T12:23:12.924921+0000","last_active":"2024-03-04T12:25:06.936612+0000","last_peered":"2024-03-04T12:25:06.936612+0000","last_clean":"2024-03-04T12:25:06.936612+0000","last_became_active":"2024-03-04T12:23:12.924558+0000","last_became_peered":"2024-03-04T12:23:12.924558+0000","last_unstale":"2024-03-04T12:25:06.936612+0000","last_undegraded":"2024-03-04T12:25:06.936612+0000","last_fullsized":"2024-03-04T12:25:06.936612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:10:53.238440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932437+0000","last_change":"2024-03-04T12:23:12.913106+0000","last_active":"2024-03-04T12:25:06.932437+0000","last_peered":"2024-03-04T12:25:06.932437+0000","last_clean":"2024-03-04T12:25:06.932437+0000","last_became_active":"2024-03-04T12:23:12.912741+0000","last_became_peered":"2024-03-04T12:23:12.912741+0000","last_unstale":"2024-03-04T12:25:06.932437+0000","last_undegraded":"2024-03-04T12:25:06.932437+0000","last_fullsized":"2024-03-04T12:25:06.932437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:46:21.406383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276375+0000","last_change":"2024-03-04T12:23:12.918082+0000","last_active":"2024-03-04T12:25:07.276375+0000","last_peered":"2024-03-04T12:25:07.276375+0000","last_clean":"2024-03-04T12:25:07.276375+0000","last_became_active":"2024-03-04T12:23:12.917754+0000","last_became_peered":"2024-03-04T12:23:12.917754+0000","last_unstale":"2024-03-04T12:25:07.276375+0000","last_undegraded":"2024-03-04T12:25:07.276375+0000","last_fullsized":"2024-03-04T12:25:07.276375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:56.601779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286257,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68620,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1221,"num_read_kb":689,"num_write":261,"num_write_kb":152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":210956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383724,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":520215,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28164,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716988,"statfs":{"total":95995035648,"available":95966195712,"internally_reserved":0,"allocated":1175552,"data_stored":1168389,"data_compressed":63026,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808887,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":524241,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":520523,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1118001,"data_compressed":63026,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515649,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":577505,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1052672,"data_stored":517105,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":578565,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":515357,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":520255,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1110016,"data_stored":1113051,"data_compressed":63026,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T12:25:42.546 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-04T12:25:42.546 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-04T12:25:42.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.677+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/2199166821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff8100260 msgr2=0x7f6ff8100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:42.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.677+0000 7f6ffdb7e640 1 --2- 172.21.15.60:0/2199166821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff8100260 0x7f6ff8100680 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009900 tx=0x7f6fe802bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:42.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.677+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/2199166821 shutdown_connections 2024-03-04T12:25:42.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.677+0000 7f6ffdb7e640 1 --2- 172.21.15.60:0/2199166821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff8101e10 0x7f6ff81086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.677+0000 7f6ffdb7e640 1 --2- 172.21.15.60:0/2199166821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ff8101450 0x7f6ff81018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.677+0000 7f6ffdb7e640 1 --2- 172.21.15.60:0/2199166821 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff8100260 0x7f6ff8100680 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.677+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/2199166821 >> 172.21.15.60:0/2199166821 conn(0x7f6ff80fb7d0 msgr2=0x7f6ff80fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.677+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/2199166821 shutdown_connections 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.677+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/2199166821 wait complete. 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.677+0000 7f6ffdb7e640 1 Processor -- start 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.677+0000 7f6ffdb7e640 1 -- start start 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ffdb7e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff8100260 0x7f6ff8069170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ffdb7e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ff8101450 0x7f6ff819e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ffdb7e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff8101e10 0x7f6ff81a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ffdb7e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff810bef0 con 0x7f6ff8100260 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ffdb7e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff810bd70 con 0x7f6ff8101450 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ffdb7e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff810c070 con 0x7f6ff8101e10 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff8100260 0x7f6ff8069170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff8100260 0x7f6ff8069170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55902/0 (socket says 172.21.15.60:55902) 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff77fe640 1 -- 172.21.15.60:0/3231963070 learned_addr learned my addr 172.21.15.60:0/3231963070 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff6ffd640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ff8101450 0x7f6ff819e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:42.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff7fff640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff8101e10 0x7f6ff81a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:42.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff77fe640 1 -- 172.21.15.60:0/3231963070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff8101e10 msgr2=0x7f6ff81a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:42.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff77fe640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff8101e10 0x7f6ff81a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff77fe640 1 -- 172.21.15.60:0/3231963070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ff8101450 msgr2=0x7f6ff819e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:42.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff77fe640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ff8101450 0x7f6ff819e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff77fe640 1 -- 172.21.15.60:0/3231963070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff81a1470 con 0x7f6ff8100260 2024-03-04T12:25:42.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff77fe640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff8100260 0x7f6ff8069170 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8005ab0 tx=0x7f6fe8005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:42.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff4ff9640 1 -- 172.21.15.60:0/3231963070 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8041070 con 0x7f6ff8100260 2024-03-04T12:25:42.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff4ff9640 1 -- 172.21.15.60:0/3231963070 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fe80054e0 con 0x7f6ff8100260 2024-03-04T12:25:42.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/3231963070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff81a16a0 con 0x7f6ff8100260 2024-03-04T12:25:42.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ff4ff9640 1 -- 172.21.15.60:0/3231963070 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8034440 con 0x7f6ff8100260 2024-03-04T12:25:42.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.678+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/3231963070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff81a1b90 con 0x7f6ff8100260 2024-03-04T12:25:42.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.679+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/3231963070 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff8107520 con 0x7f6ff8100260 2024-03-04T12:25:42.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.680+0000 7f6ff4ff9640 1 -- 172.21.15.60:0/3231963070 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f6fe8044070 con 0x7f6ff8100260 2024-03-04T12:25:42.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.681+0000 7f6ff4ff9640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6fe00a1e40 0x7f6fe00a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:42.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.681+0000 7f6ff4ff9640 1 -- 172.21.15.60:0/3231963070 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f6fe80efd70 con 0x7f6ff8100260 2024-03-04T12:25:42.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.681+0000 7f6ff6ffd640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6fe00a1e40 0x7f6fe00a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:42.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.681+0000 7f6ff6ffd640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6fe00a1e40 0x7f6fe00a4300 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8068ee0 tx=0x7f6fec00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:42.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.685+0000 7f6ff4ff9640 1 -- 172.21.15.60:0/3231963070 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6fe80bcdc0 con 0x7f6ff8100260 2024-03-04T12:25:42.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.863+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/3231963070 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6ff8061770 con 0x7f6fe00a1e40 2024-03-04T12:25:42.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.873+0000 7f6ff4ff9640 1 -- 172.21.15.60:0/3231963070 <== mgr.14152 v2:172.21.15.60:6800/1914863179 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340807 (secure 0 0 0) 0x7f6ff8061770 con 0x7f6fe00a1e40 2024-03-04T12:25:42.883 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:42.885 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-03-04T12:25:42.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.876+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/3231963070 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6fe00a1e40 msgr2=0x7f6fe00a4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:42.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.876+0000 7f6ffdb7e640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6fe00a1e40 0x7f6fe00a4300 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8068ee0 tx=0x7f6fec00a2d0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.876+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/3231963070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff8100260 msgr2=0x7f6ff8069170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:42.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.876+0000 7f6ffdb7e640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff8100260 0x7f6ff8069170 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8005ab0 tx=0x7f6fe8005dc0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.877+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/3231963070 shutdown_connections 2024-03-04T12:25:42.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.877+0000 7f6ffdb7e640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6fe00a1e40 0x7f6fe00a4300 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.877+0000 7f6ffdb7e640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff8101e10 0x7f6ff81a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.877+0000 7f6ffdb7e640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ff8101450 0x7f6ff819e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.877+0000 7f6ffdb7e640 1 --2- 172.21.15.60:0/3231963070 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6ff8100260 0x7f6ff8069170 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:42.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.877+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/3231963070 >> 172.21.15.60:0/3231963070 conn(0x7f6ff80fb7d0 msgr2=0x7f6ff81046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:42.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.877+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/3231963070 shutdown_connections 2024-03-04T12:25:42.887 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:42.877+0000 7f6ffdb7e640 1 -- 172.21.15.60:0/3231963070 wait complete. 2024-03-04T12:25:42.895 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-03-04T12:25:42.192927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659222,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1357,"num_read_kb":808,"num_write":444,"num_write_kb":3092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386748,"kb_used_data":12584,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555076,"statfs":{"total":1151940427776,"available":1151544397824,"internally_reserved":0,"allocated":12886016,"data_stored":8188856,"data_compressed":428748,"data_compressed_allocated":577536,"data_compressed_original":2654208,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":57,"num_write":13,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003123"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276572+0000","last_change":"2024-03-04T12:23:12.926673+0000","last_active":"2024-03-04T12:25:07.276572+0000","last_peered":"2024-03-04T12:25:07.276572+0000","last_clean":"2024-03-04T12:25:07.276572+0000","last_became_active":"2024-03-04T12:23:12.926299+0000","last_became_peered":"2024-03-04T12:23:12.926299+0000","last_unstale":"2024-03-04T12:25:07.276572+0000","last_undegraded":"2024-03-04T12:25:07.276572+0000","last_fullsized":"2024-03-04T12:25:07.276572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:19.608980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940339+0000","last_change":"2024-03-04T12:23:12.921758+0000","last_active":"2024-03-04T12:25:06.940339+0000","last_peered":"2024-03-04T12:25:06.940339+0000","last_clean":"2024-03-04T12:25:06.940339+0000","last_became_active":"2024-03-04T12:23:12.921317+0000","last_became_peered":"2024-03-04T12:23:12.921317+0000","last_unstale":"2024-03-04T12:25:06.940339+0000","last_undegraded":"2024-03-04T12:25:06.940339+0000","last_fullsized":"2024-03-04T12:25:06.940339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:20:25.179004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939582+0000","last_change":"2024-03-04T12:23:12.925771+0000","last_active":"2024-03-04T12:25:06.939582+0000","last_peered":"2024-03-04T12:25:06.939582+0000","last_clean":"2024-03-04T12:25:06.939582+0000","last_became_active":"2024-03-04T12:23:12.925447+0000","last_became_peered":"2024-03-04T12:23:12.925447+0000","last_unstale":"2024-03-04T12:25:06.939582+0000","last_undegraded":"2024-03-04T12:25:06.939582+0000","last_fullsized":"2024-03-04T12:25:06.939582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:19:12.380831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935876+0000","last_change":"2024-03-04T12:24:05.959806+0000","last_active":"2024-03-04T12:25:06.935876+0000","last_peered":"2024-03-04T12:25:06.935876+0000","last_clean":"2024-03-04T12:25:06.935876+0000","last_became_active":"2024-03-04T12:24:05.959557+0000","last_became_peered":"2024-03-04T12:24:05.959557+0000","last_unstale":"2024-03-04T12:25:06.935876+0000","last_undegraded":"2024-03-04T12:25:06.935876+0000","last_fullsized":"2024-03-04T12:25:06.935876+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:51.945517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938061+0000","last_change":"2024-03-04T12:23:12.921118+0000","last_active":"2024-03-04T12:25:06.938061+0000","last_peered":"2024-03-04T12:25:06.938061+0000","last_clean":"2024-03-04T12:25:06.938061+0000","last_became_active":"2024-03-04T12:23:12.920127+0000","last_became_peered":"2024-03-04T12:23:12.920127+0000","last_unstale":"2024-03-04T12:25:06.938061+0000","last_undegraded":"2024-03-04T12:25:06.938061+0000","last_fullsized":"2024-03-04T12:25:06.938061+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:22:10.893431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.025786+0000","last_change":"2024-03-04T12:23:12.916751+0000","last_active":"2024-03-04T12:25:07.025786+0000","last_peered":"2024-03-04T12:25:07.025786+0000","last_clean":"2024-03-04T12:25:07.025786+0000","last_became_active":"2024-03-04T12:23:12.916404+0000","last_became_peered":"2024-03-04T12:23:12.916404+0000","last_unstale":"2024-03-04T12:25:07.025786+0000","last_undegraded":"2024-03-04T12:25:07.025786+0000","last_fullsized":"2024-03-04T12:25:07.025786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:39.998992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036504+0000","last_change":"2024-03-04T12:24:06.029608+0000","last_active":"2024-03-04T12:25:07.036504+0000","last_peered":"2024-03-04T12:25:07.036504+0000","last_clean":"2024-03-04T12:25:07.036504+0000","last_became_active":"2024-03-04T12:24:06.029021+0000","last_became_peered":"2024-03-04T12:24:06.029021+0000","last_unstale":"2024-03-04T12:25:07.036504+0000","last_undegraded":"2024-03-04T12:25:07.036504+0000","last_fullsized":"2024-03-04T12:25:07.036504+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:13.140511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938055+0000","last_change":"2024-03-04T12:23:12.917627+0000","last_active":"2024-03-04T12:25:06.938055+0000","last_peered":"2024-03-04T12:25:06.938055+0000","last_clean":"2024-03-04T12:25:06.938055+0000","last_became_active":"2024-03-04T12:23:12.917279+0000","last_became_peered":"2024-03-04T12:23:12.917279+0000","last_unstale":"2024-03-04T12:25:06.938055+0000","last_undegraded":"2024-03-04T12:25:06.938055+0000","last_fullsized":"2024-03-04T12:25:06.938055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:33:03.487273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.937093+0000","last_change":"2024-03-04T12:23:12.922601+0000","last_active":"2024-03-04T12:25:06.937093+0000","last_peered":"2024-03-04T12:25:06.937093+0000","last_clean":"2024-03-04T12:25:06.937093+0000","last_became_active":"2024-03-04T12:23:12.922037+0000","last_became_peered":"2024-03-04T12:23:12.922037+0000","last_unstale":"2024-03-04T12:25:06.937093+0000","last_undegraded":"2024-03-04T12:25:06.937093+0000","last_fullsized":"2024-03-04T12:25:06.937093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:37:32.076439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.026254+0000","last_change":"2024-03-04T12:23:12.916308+0000","last_active":"2024-03-04T12:25:07.026254+0000","last_peered":"2024-03-04T12:25:07.026254+0000","last_clean":"2024-03-04T12:25:07.026254+0000","last_became_active":"2024-03-04T12:23:12.915897+0000","last_became_peered":"2024-03-04T12:23:12.915897+0000","last_unstale":"2024-03-04T12:25:07.026254+0000","last_undegraded":"2024-03-04T12:25:07.026254+0000","last_fullsized":"2024-03-04T12:25:07.026254+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:15:53.133053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037015+0000","last_change":"2024-03-04T12:23:12.930700+0000","last_active":"2024-03-04T12:25:07.037015+0000","last_peered":"2024-03-04T12:25:07.037015+0000","last_clean":"2024-03-04T12:25:07.037015+0000","last_became_active":"2024-03-04T12:23:12.930328+0000","last_became_peered":"2024-03-04T12:23:12.930328+0000","last_unstale":"2024-03-04T12:25:07.037015+0000","last_undegraded":"2024-03-04T12:25:07.037015+0000","last_fullsized":"2024-03-04T12:25:07.037015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:54:29.990527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933883+0000","last_change":"2024-03-04T12:23:12.915471+0000","last_active":"2024-03-04T12:25:06.933883+0000","last_peered":"2024-03-04T12:25:06.933883+0000","last_clean":"2024-03-04T12:25:06.933883+0000","last_became_active":"2024-03-04T12:23:12.913627+0000","last_became_peered":"2024-03-04T12:23:12.913627+0000","last_unstale":"2024-03-04T12:25:06.933883+0000","last_undegraded":"2024-03-04T12:25:06.933883+0000","last_fullsized":"2024-03-04T12:25:06.933883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:42:56.762782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932392+0000","last_change":"2024-03-04T12:23:12.899143+0000","last_active":"2024-03-04T12:25:06.932392+0000","last_peered":"2024-03-04T12:25:06.932392+0000","last_clean":"2024-03-04T12:25:06.932392+0000","last_became_active":"2024-03-04T12:23:12.898498+0000","last_became_peered":"2024-03-04T12:23:12.898498+0000","last_unstale":"2024-03-04T12:25:06.932392+0000","last_undegraded":"2024-03-04T12:25:06.932392+0000","last_fullsized":"2024-03-04T12:25:06.932392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:15:33.840683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939467+0000","last_change":"2024-03-04T12:23:12.920873+0000","last_active":"2024-03-04T12:25:06.939467+0000","last_peered":"2024-03-04T12:25:06.939467+0000","last_clean":"2024-03-04T12:25:06.939467+0000","last_became_active":"2024-03-04T12:23:12.919080+0000","last_became_peered":"2024-03-04T12:23:12.919080+0000","last_unstale":"2024-03-04T12:25:06.939467+0000","last_undegraded":"2024-03-04T12:25:06.939467+0000","last_fullsized":"2024-03-04T12:25:06.939467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:47:26.710624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939195+0000","last_change":"2024-03-04T12:24:05.959024+0000","last_active":"2024-03-04T12:25:06.939195+0000","last_peered":"2024-03-04T12:25:06.939195+0000","last_clean":"2024-03-04T12:25:06.939195+0000","last_became_active":"2024-03-04T12:24:05.958750+0000","last_became_peered":"2024-03-04T12:24:05.958750+0000","last_unstale":"2024-03-04T12:25:06.939195+0000","last_undegraded":"2024-03-04T12:25:06.939195+0000","last_fullsized":"2024-03-04T12:25:06.939195+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:59:33.263357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936333+0000","last_change":"2024-03-04T12:23:12.923860+0000","last_active":"2024-03-04T12:25:06.936333+0000","last_peered":"2024-03-04T12:25:06.936333+0000","last_clean":"2024-03-04T12:25:06.936333+0000","last_became_active":"2024-03-04T12:23:12.923530+0000","last_became_peered":"2024-03-04T12:23:12.923530+0000","last_unstale":"2024-03-04T12:25:06.936333+0000","last_undegraded":"2024-03-04T12:25:06.936333+0000","last_fullsized":"2024-03-04T12:25:06.936333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:30:01.466458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276427+0000","last_change":"2024-03-04T12:23:12.927159+0000","last_active":"2024-03-04T12:25:07.276427+0000","last_peered":"2024-03-04T12:25:07.276427+0000","last_clean":"2024-03-04T12:25:07.276427+0000","last_became_active":"2024-03-04T12:23:12.926800+0000","last_became_peered":"2024-03-04T12:23:12.926800+0000","last_unstale":"2024-03-04T12:25:07.276427+0000","last_undegraded":"2024-03-04T12:25:07.276427+0000","last_fullsized":"2024-03-04T12:25:07.276427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:52.633648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936123+0000","last_change":"2024-03-04T12:23:12.920607+0000","last_active":"2024-03-04T12:25:06.936123+0000","last_peered":"2024-03-04T12:25:06.936123+0000","last_clean":"2024-03-04T12:25:06.936123+0000","last_became_active":"2024-03-04T12:23:12.920257+0000","last_became_peered":"2024-03-04T12:23:12.920257+0000","last_unstale":"2024-03-04T12:25:06.936123+0000","last_undegraded":"2024-03-04T12:25:06.936123+0000","last_fullsized":"2024-03-04T12:25:06.936123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:39.353673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939723+0000","last_change":"2024-03-04T12:23:12.906635+0000","last_active":"2024-03-04T12:25:06.939723+0000","last_peered":"2024-03-04T12:25:06.939723+0000","last_clean":"2024-03-04T12:25:06.939723+0000","last_became_active":"2024-03-04T12:23:12.906243+0000","last_became_peered":"2024-03-04T12:23:12.906243+0000","last_unstale":"2024-03-04T12:25:06.939723+0000","last_undegraded":"2024-03-04T12:25:06.939723+0000","last_fullsized":"2024-03-04T12:25:06.939723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:44.375934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939135+0000","last_change":"2024-03-04T12:23:16.951685+0000","last_active":"2024-03-04T12:25:06.939135+0000","last_peered":"2024-03-04T12:25:06.939135+0000","last_clean":"2024-03-04T12:25:06.939135+0000","last_became_active":"2024-03-04T12:23:16.951329+0000","last_became_peered":"2024-03-04T12:23:16.951329+0000","last_unstale":"2024-03-04T12:25:06.939135+0000","last_undegraded":"2024-03-04T12:25:06.939135+0000","last_fullsized":"2024-03-04T12:25:06.939135+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:03:27.233466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":165,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:39.077764+0000","last_change":"2024-03-04T12:23:11.900681+0000","last_active":"2024-03-04T12:25:39.077764+0000","last_peered":"2024-03-04T12:25:39.077764+0000","last_clean":"2024-03-04T12:25:39.077764+0000","last_became_active":"2024-03-04T12:23:11.900365+0000","last_became_peered":"2024-03-04T12:23:11.900365+0000","last_unstale":"2024-03-04T12:25:39.077764+0000","last_undegraded":"2024-03-04T12:25:39.077764+0000","last_fullsized":"2024-03-04T12:25:39.077764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:53.052513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936030+0000","last_change":"2024-03-04T12:24:06.030707+0000","last_active":"2024-03-04T12:25:06.936030+0000","last_peered":"2024-03-04T12:25:06.936030+0000","last_clean":"2024-03-04T12:25:06.936030+0000","last_became_active":"2024-03-04T12:24:06.030306+0000","last_became_peered":"2024-03-04T12:24:06.030306+0000","last_unstale":"2024-03-04T12:25:06.936030+0000","last_undegraded":"2024-03-04T12:25:06.936030+0000","last_fullsized":"2024-03-04T12:25:06.936030+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:59:18.939734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,5],"acting":[9,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933263+0000","last_change":"2024-03-04T12:23:16.953704+0000","last_active":"2024-03-04T12:25:06.933263+0000","last_peered":"2024-03-04T12:25:06.933263+0000","last_clean":"2024-03-04T12:25:06.933263+0000","last_became_active":"2024-03-04T12:23:16.953177+0000","last_became_peered":"2024-03-04T12:23:16.953177+0000","last_unstale":"2024-03-04T12:25:06.933263+0000","last_undegraded":"2024-03-04T12:25:06.933263+0000","last_fullsized":"2024-03-04T12:25:06.933263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:37:00.469767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939293+0000","last_change":"2024-03-04T12:23:11.904271+0000","last_active":"2024-03-04T12:25:06.939293+0000","last_peered":"2024-03-04T12:25:06.939293+0000","last_clean":"2024-03-04T12:25:06.939293+0000","last_became_active":"2024-03-04T12:23:11.903963+0000","last_became_peered":"2024-03-04T12:23:11.903963+0000","last_unstale":"2024-03-04T12:25:06.939293+0000","last_undegraded":"2024-03-04T12:25:06.939293+0000","last_fullsized":"2024-03-04T12:25:06.939293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:09:44.774621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939180+0000","last_change":"2024-03-04T12:23:12.925750+0000","last_active":"2024-03-04T12:25:06.939180+0000","last_peered":"2024-03-04T12:25:06.939180+0000","last_clean":"2024-03-04T12:25:06.939180+0000","last_became_active":"2024-03-04T12:23:12.925380+0000","last_became_peered":"2024-03-04T12:23:12.925380+0000","last_unstale":"2024-03-04T12:25:06.939180+0000","last_undegraded":"2024-03-04T12:25:06.939180+0000","last_fullsized":"2024-03-04T12:25:06.939180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:14.665127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933357+0000","last_change":"2024-03-04T12:23:16.952303+0000","last_active":"2024-03-04T12:25:06.933357+0000","last_peered":"2024-03-04T12:25:06.933357+0000","last_clean":"2024-03-04T12:25:06.933357+0000","last_became_active":"2024-03-04T12:23:16.951666+0000","last_became_peered":"2024-03-04T12:23:16.951666+0000","last_unstale":"2024-03-04T12:25:06.933357+0000","last_undegraded":"2024-03-04T12:25:06.933357+0000","last_fullsized":"2024-03-04T12:25:06.933357+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:20:41.326808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938665+0000","last_change":"2024-03-04T12:23:11.903871+0000","last_active":"2024-03-04T12:25:06.938665+0000","last_peered":"2024-03-04T12:25:06.938665+0000","last_clean":"2024-03-04T12:25:06.938665+0000","last_became_active":"2024-03-04T12:23:11.902735+0000","last_became_peered":"2024-03-04T12:23:11.902735+0000","last_unstale":"2024-03-04T12:25:06.938665+0000","last_undegraded":"2024-03-04T12:25:06.938665+0000","last_fullsized":"2024-03-04T12:25:06.938665+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:45:09.897740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.937164+0000","last_change":"2024-03-04T12:23:12.924504+0000","last_active":"2024-03-04T12:25:06.937164+0000","last_peered":"2024-03-04T12:25:06.937164+0000","last_clean":"2024-03-04T12:25:06.937164+0000","last_became_active":"2024-03-04T12:23:12.922934+0000","last_became_peered":"2024-03-04T12:23:12.922934+0000","last_unstale":"2024-03-04T12:25:06.937164+0000","last_undegraded":"2024-03-04T12:25:06.937164+0000","last_fullsized":"2024-03-04T12:25:06.937164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:41:28.974601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933262+0000","last_change":"2024-03-04T12:23:16.952324+0000","last_active":"2024-03-04T12:25:06.933262+0000","last_peered":"2024-03-04T12:25:06.933262+0000","last_clean":"2024-03-04T12:25:06.933262+0000","last_became_active":"2024-03-04T12:23:16.951145+0000","last_became_peered":"2024-03-04T12:23:16.951145+0000","last_unstale":"2024-03-04T12:25:06.933262+0000","last_undegraded":"2024-03-04T12:25:06.933262+0000","last_fullsized":"2024-03-04T12:25:06.933262+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:20:47.029523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938577+0000","last_change":"2024-03-04T12:23:11.904391+0000","last_active":"2024-03-04T12:25:06.938577+0000","last_peered":"2024-03-04T12:25:06.938577+0000","last_clean":"2024-03-04T12:25:06.938577+0000","last_became_active":"2024-03-04T12:23:11.903244+0000","last_became_peered":"2024-03-04T12:23:11.903244+0000","last_unstale":"2024-03-04T12:25:06.938577+0000","last_undegraded":"2024-03-04T12:25:06.938577+0000","last_fullsized":"2024-03-04T12:25:06.938577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:06:27.590418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933368+0000","last_change":"2024-03-04T12:23:12.925623+0000","last_active":"2024-03-04T12:25:06.933368+0000","last_peered":"2024-03-04T12:25:06.933368+0000","last_clean":"2024-03-04T12:25:06.933368+0000","last_became_active":"2024-03-04T12:23:12.925243+0000","last_became_peered":"2024-03-04T12:23:12.925243+0000","last_unstale":"2024-03-04T12:25:06.933368+0000","last_undegraded":"2024-03-04T12:25:06.933368+0000","last_fullsized":"2024-03-04T12:25:06.933368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:11:04.142655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939469+0000","last_change":"2024-03-04T12:23:16.946064+0000","last_active":"2024-03-04T12:25:06.939469+0000","last_peered":"2024-03-04T12:25:06.939469+0000","last_clean":"2024-03-04T12:25:06.939469+0000","last_became_active":"2024-03-04T12:23:16.945342+0000","last_became_peered":"2024-03-04T12:23:16.945342+0000","last_unstale":"2024-03-04T12:25:06.939469+0000","last_undegraded":"2024-03-04T12:25:06.939469+0000","last_fullsized":"2024-03-04T12:25:06.939469+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:43:37.500216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":162,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:37.071102+0000","last_change":"2024-03-04T12:23:11.899478+0000","last_active":"2024-03-04T12:25:37.071102+0000","last_peered":"2024-03-04T12:25:37.071102+0000","last_clean":"2024-03-04T12:25:37.071102+0000","last_became_active":"2024-03-04T12:23:11.899153+0000","last_became_peered":"2024-03-04T12:23:11.899153+0000","last_unstale":"2024-03-04T12:25:37.071102+0000","last_undegraded":"2024-03-04T12:25:37.071102+0000","last_fullsized":"2024-03-04T12:25:37.071102+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:54:10.709054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938359+0000","last_change":"2024-03-04T12:23:12.908560+0000","last_active":"2024-03-04T12:25:06.938359+0000","last_peered":"2024-03-04T12:25:06.938359+0000","last_clean":"2024-03-04T12:25:06.938359+0000","last_became_active":"2024-03-04T12:23:12.908221+0000","last_became_peered":"2024-03-04T12:23:12.908221+0000","last_unstale":"2024-03-04T12:25:06.938359+0000","last_undegraded":"2024-03-04T12:25:06.938359+0000","last_fullsized":"2024-03-04T12:25:06.938359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:28:06.545945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939489+0000","last_change":"2024-03-04T12:24:06.031257+0000","last_active":"2024-03-04T12:25:06.939489+0000","last_peered":"2024-03-04T12:25:06.939489+0000","last_clean":"2024-03-04T12:25:06.939489+0000","last_became_active":"2024-03-04T12:24:06.030409+0000","last_became_peered":"2024-03-04T12:24:06.030409+0000","last_unstale":"2024-03-04T12:25:06.939489+0000","last_undegraded":"2024-03-04T12:25:06.939489+0000","last_fullsized":"2024-03-04T12:25:06.939489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:47:31.403224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.025691+0000","last_change":"2024-03-04T12:23:11.898360+0000","last_active":"2024-03-04T12:25:07.025691+0000","last_peered":"2024-03-04T12:25:07.025691+0000","last_clean":"2024-03-04T12:25:07.025691+0000","last_became_active":"2024-03-04T12:23:11.897988+0000","last_became_peered":"2024-03-04T12:23:11.897988+0000","last_unstale":"2024-03-04T12:25:07.025691+0000","last_undegraded":"2024-03-04T12:25:07.025691+0000","last_fullsized":"2024-03-04T12:25:07.025691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:46:17.448032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933598+0000","last_change":"2024-03-04T12:23:12.918392+0000","last_active":"2024-03-04T12:25:06.933598+0000","last_peered":"2024-03-04T12:25:06.933598+0000","last_clean":"2024-03-04T12:25:06.933598+0000","last_became_active":"2024-03-04T12:23:12.917656+0000","last_became_peered":"2024-03-04T12:23:12.917656+0000","last_unstale":"2024-03-04T12:25:06.933598+0000","last_undegraded":"2024-03-04T12:25:06.933598+0000","last_fullsized":"2024-03-04T12:25:06.933598+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:44:14.545925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276449+0000","last_change":"2024-03-04T12:23:16.941841+0000","last_active":"2024-03-04T12:25:07.276449+0000","last_peered":"2024-03-04T12:25:07.276449+0000","last_clean":"2024-03-04T12:25:07.276449+0000","last_became_active":"2024-03-04T12:23:16.941449+0000","last_became_peered":"2024-03-04T12:23:16.941449+0000","last_unstale":"2024-03-04T12:25:07.276449+0000","last_undegraded":"2024-03-04T12:25:07.276449+0000","last_fullsized":"2024-03-04T12:25:07.276449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:50:35.451834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932169+0000","last_change":"2024-03-04T12:23:11.906723+0000","last_active":"2024-03-04T12:25:06.932169+0000","last_peered":"2024-03-04T12:25:06.932169+0000","last_clean":"2024-03-04T12:25:06.932169+0000","last_became_active":"2024-03-04T12:23:11.906357+0000","last_became_peered":"2024-03-04T12:23:11.906357+0000","last_unstale":"2024-03-04T12:25:06.932169+0000","last_undegraded":"2024-03-04T12:25:06.932169+0000","last_fullsized":"2024-03-04T12:25:06.932169+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:09.592236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938664+0000","last_change":"2024-03-04T12:23:12.916435+0000","last_active":"2024-03-04T12:25:06.938664+0000","last_peered":"2024-03-04T12:25:06.938664+0000","last_clean":"2024-03-04T12:25:06.938664+0000","last_became_active":"2024-03-04T12:23:12.915607+0000","last_became_peered":"2024-03-04T12:23:12.915607+0000","last_unstale":"2024-03-04T12:25:06.938664+0000","last_undegraded":"2024-03-04T12:25:06.938664+0000","last_fullsized":"2024-03-04T12:25:06.938664+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:41:32.900344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940234+0000","last_change":"2024-03-04T12:23:16.944342+0000","last_active":"2024-03-04T12:25:06.940234+0000","last_peered":"2024-03-04T12:25:06.940234+0000","last_clean":"2024-03-04T12:25:06.940234+0000","last_became_active":"2024-03-04T12:23:16.943683+0000","last_became_peered":"2024-03-04T12:23:16.943683+0000","last_unstale":"2024-03-04T12:25:06.940234+0000","last_undegraded":"2024-03-04T12:25:06.940234+0000","last_fullsized":"2024-03-04T12:25:06.940234+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:04:19.658860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933018+0000","last_change":"2024-03-04T12:23:11.899194+0000","last_active":"2024-03-04T12:25:06.933018+0000","last_peered":"2024-03-04T12:25:06.933018+0000","last_clean":"2024-03-04T12:25:06.933018+0000","last_became_active":"2024-03-04T12:23:11.898109+0000","last_became_peered":"2024-03-04T12:23:11.898109+0000","last_unstale":"2024-03-04T12:25:06.933018+0000","last_undegraded":"2024-03-04T12:25:06.933018+0000","last_fullsized":"2024-03-04T12:25:06.933018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:23:08.808202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935619+0000","last_change":"2024-03-04T12:23:12.916549+0000","last_active":"2024-03-04T12:25:06.935619+0000","last_peered":"2024-03-04T12:25:06.935619+0000","last_clean":"2024-03-04T12:25:06.935619+0000","last_became_active":"2024-03-04T12:23:12.916204+0000","last_became_peered":"2024-03-04T12:23:12.916204+0000","last_unstale":"2024-03-04T12:25:06.935619+0000","last_undegraded":"2024-03-04T12:25:06.935619+0000","last_fullsized":"2024-03-04T12:25:06.935619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:56.412600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938437+0000","last_change":"2024-03-04T12:23:16.945903+0000","last_active":"2024-03-04T12:25:06.938437+0000","last_peered":"2024-03-04T12:25:06.938437+0000","last_clean":"2024-03-04T12:25:06.938437+0000","last_became_active":"2024-03-04T12:23:16.945535+0000","last_became_peered":"2024-03-04T12:23:16.945535+0000","last_unstale":"2024-03-04T12:25:06.938437+0000","last_undegraded":"2024-03-04T12:25:06.938437+0000","last_fullsized":"2024-03-04T12:25:06.938437+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:45:01.178370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'6","reported_seq":272,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:37.082131+0000","last_change":"2024-03-04T12:23:11.901570+0000","last_active":"2024-03-04T12:25:37.082131+0000","last_peered":"2024-03-04T12:25:37.082131+0000","last_clean":"2024-03-04T12:25:37.082131+0000","last_became_active":"2024-03-04T12:23:11.900702+0000","last_became_peered":"2024-03-04T12:23:11.900702+0000","last_unstale":"2024-03-04T12:25:37.082131+0000","last_undegraded":"2024-03-04T12:25:37.082131+0000","last_fullsized":"2024-03-04T12:25:37.082131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:18.758453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":234,"num_read_kb":234,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936880+0000","last_change":"2024-03-04T12:23:12.919571+0000","last_active":"2024-03-04T12:25:06.936880+0000","last_peered":"2024-03-04T12:25:06.936880+0000","last_clean":"2024-03-04T12:25:06.936880+0000","last_became_active":"2024-03-04T12:23:12.919143+0000","last_became_peered":"2024-03-04T12:23:12.919143+0000","last_unstale":"2024-03-04T12:25:06.936880+0000","last_undegraded":"2024-03-04T12:25:06.936880+0000","last_fullsized":"2024-03-04T12:25:06.936880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:25.459417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935561+0000","last_change":"2024-03-04T12:23:16.946380+0000","last_active":"2024-03-04T12:25:06.935561+0000","last_peered":"2024-03-04T12:25:06.935561+0000","last_clean":"2024-03-04T12:25:06.935561+0000","last_became_active":"2024-03-04T12:23:16.945983+0000","last_became_peered":"2024-03-04T12:23:16.945983+0000","last_unstale":"2024-03-04T12:25:06.935561+0000","last_undegraded":"2024-03-04T12:25:06.935561+0000","last_fullsized":"2024-03-04T12:25:06.935561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:39:43.297619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036989+0000","last_change":"2024-03-04T12:23:11.897176+0000","last_active":"2024-03-04T12:25:07.036989+0000","last_peered":"2024-03-04T12:25:07.036989+0000","last_clean":"2024-03-04T12:25:07.036989+0000","last_became_active":"2024-03-04T12:23:11.896747+0000","last_became_peered":"2024-03-04T12:23:11.896747+0000","last_unstale":"2024-03-04T12:25:07.036989+0000","last_undegraded":"2024-03-04T12:25:07.036989+0000","last_fullsized":"2024-03-04T12:25:07.036989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:33:02.206704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939064+0000","last_change":"2024-03-04T12:23:12.922021+0000","last_active":"2024-03-04T12:25:06.939064+0000","last_peered":"2024-03-04T12:25:06.939064+0000","last_clean":"2024-03-04T12:25:06.939064+0000","last_became_active":"2024-03-04T12:23:12.918724+0000","last_became_peered":"2024-03-04T12:23:12.918724+0000","last_unstale":"2024-03-04T12:25:06.939064+0000","last_undegraded":"2024-03-04T12:25:06.939064+0000","last_fullsized":"2024-03-04T12:25:06.939064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:49:09.017979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.934143+0000","last_change":"2024-03-04T12:23:16.943271+0000","last_active":"2024-03-04T12:25:06.934143+0000","last_peered":"2024-03-04T12:25:06.934143+0000","last_clean":"2024-03-04T12:25:06.934143+0000","last_became_active":"2024-03-04T12:23:16.942400+0000","last_became_peered":"2024-03-04T12:23:16.942400+0000","last_unstale":"2024-03-04T12:25:06.934143+0000","last_undegraded":"2024-03-04T12:25:06.934143+0000","last_fullsized":"2024-03-04T12:25:06.934143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:41:06.010960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933398+0000","last_change":"2024-03-04T12:23:11.896706+0000","last_active":"2024-03-04T12:25:06.933398+0000","last_peered":"2024-03-04T12:25:06.933398+0000","last_clean":"2024-03-04T12:25:06.933398+0000","last_became_active":"2024-03-04T12:23:11.896100+0000","last_became_peered":"2024-03-04T12:23:11.896100+0000","last_unstale":"2024-03-04T12:25:06.933398+0000","last_undegraded":"2024-03-04T12:25:06.933398+0000","last_fullsized":"2024-03-04T12:25:06.933398+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:42:19.754322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.937310+0000","last_change":"2024-03-04T12:23:12.923631+0000","last_active":"2024-03-04T12:25:06.937310+0000","last_peered":"2024-03-04T12:25:06.937310+0000","last_clean":"2024-03-04T12:25:06.937310+0000","last_became_active":"2024-03-04T12:23:12.922487+0000","last_became_peered":"2024-03-04T12:23:12.922487+0000","last_unstale":"2024-03-04T12:25:06.937310+0000","last_undegraded":"2024-03-04T12:25:06.937310+0000","last_fullsized":"2024-03-04T12:25:06.937310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:30:33.923064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.025491+0000","last_change":"2024-03-04T12:24:06.032075+0000","last_active":"2024-03-04T12:25:07.025491+0000","last_peered":"2024-03-04T12:25:07.025491+0000","last_clean":"2024-03-04T12:25:07.025491+0000","last_became_active":"2024-03-04T12:24:06.031639+0000","last_became_peered":"2024-03-04T12:24:06.031639+0000","last_unstale":"2024-03-04T12:25:07.025491+0000","last_undegraded":"2024-03-04T12:25:07.025491+0000","last_fullsized":"2024-03-04T12:25:07.025491+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:44:48.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'39","reported_seq":316,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:37.070833+0000","last_change":"2024-03-04T12:23:11.898903+0000","last_active":"2024-03-04T12:25:37.070833+0000","last_peered":"2024-03-04T12:25:37.070833+0000","last_clean":"2024-03-04T12:25:37.070833+0000","last_became_active":"2024-03-04T12:23:11.898477+0000","last_became_peered":"2024-03-04T12:23:11.898477+0000","last_unstale":"2024-03-04T12:25:37.070833+0000","last_undegraded":"2024-03-04T12:25:37.070833+0000","last_fullsized":"2024-03-04T12:25:37.070833+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:36:58.638192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60826,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":245,"num_read_kb":182,"num_write":41,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938545+0000","last_change":"2024-03-04T12:23:12.908497+0000","last_active":"2024-03-04T12:25:06.938545+0000","last_peered":"2024-03-04T12:25:06.938545+0000","last_clean":"2024-03-04T12:25:06.938545+0000","last_became_active":"2024-03-04T12:23:12.907847+0000","last_became_peered":"2024-03-04T12:23:12.907847+0000","last_unstale":"2024-03-04T12:25:06.938545+0000","last_undegraded":"2024-03-04T12:25:06.938545+0000","last_fullsized":"2024-03-04T12:25:06.938545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:51:44.592851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933682+0000","last_change":"2024-03-04T12:23:16.951844+0000","last_active":"2024-03-04T12:25:06.933682+0000","last_peered":"2024-03-04T12:25:06.933682+0000","last_clean":"2024-03-04T12:25:06.933682+0000","last_became_active":"2024-03-04T12:23:16.951491+0000","last_became_peered":"2024-03-04T12:23:16.951491+0000","last_unstale":"2024-03-04T12:25:06.933682+0000","last_undegraded":"2024-03-04T12:25:06.933682+0000","last_fullsized":"2024-03-04T12:25:06.933682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:35:54.098239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939315+0000","last_change":"2024-03-04T12:23:11.899592+0000","last_active":"2024-03-04T12:25:06.939315+0000","last_peered":"2024-03-04T12:25:06.939315+0000","last_clean":"2024-03-04T12:25:06.939315+0000","last_became_active":"2024-03-04T12:23:11.899275+0000","last_became_peered":"2024-03-04T12:23:11.899275+0000","last_unstale":"2024-03-04T12:25:06.939315+0000","last_undegraded":"2024-03-04T12:25:06.939315+0000","last_fullsized":"2024-03-04T12:25:06.939315+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:03:02.184268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932222+0000","last_change":"2024-03-04T12:23:12.910084+0000","last_active":"2024-03-04T12:25:06.932222+0000","last_peered":"2024-03-04T12:25:06.932222+0000","last_clean":"2024-03-04T12:25:06.932222+0000","last_became_active":"2024-03-04T12:23:12.909719+0000","last_became_peered":"2024-03-04T12:23:12.909719+0000","last_unstale":"2024-03-04T12:25:06.932222+0000","last_undegraded":"2024-03-04T12:25:06.932222+0000","last_fullsized":"2024-03-04T12:25:06.932222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:16:41.756405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276279+0000","last_change":"2024-03-04T12:24:06.029579+0000","last_active":"2024-03-04T12:25:07.276279+0000","last_peered":"2024-03-04T12:25:07.276279+0000","last_clean":"2024-03-04T12:25:07.276279+0000","last_became_active":"2024-03-04T12:24:06.029042+0000","last_became_peered":"2024-03-04T12:24:06.029042+0000","last_unstale":"2024-03-04T12:25:07.276279+0000","last_undegraded":"2024-03-04T12:25:07.276279+0000","last_fullsized":"2024-03-04T12:25:07.276279+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:59:35.867212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933484+0000","last_change":"2024-03-04T12:23:11.897506+0000","last_active":"2024-03-04T12:25:06.933484+0000","last_peered":"2024-03-04T12:25:06.933484+0000","last_clean":"2024-03-04T12:25:06.933484+0000","last_became_active":"2024-03-04T12:23:11.897090+0000","last_became_peered":"2024-03-04T12:23:11.897090+0000","last_unstale":"2024-03-04T12:25:06.933484+0000","last_undegraded":"2024-03-04T12:25:06.933484+0000","last_fullsized":"2024-03-04T12:25:06.933484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:10:27.868713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037179+0000","last_change":"2024-03-04T12:23:12.921825+0000","last_active":"2024-03-04T12:25:07.037179+0000","last_peered":"2024-03-04T12:25:07.037179+0000","last_clean":"2024-03-04T12:25:07.037179+0000","last_became_active":"2024-03-04T12:23:12.921491+0000","last_became_peered":"2024-03-04T12:23:12.921491+0000","last_unstale":"2024-03-04T12:25:07.037179+0000","last_undegraded":"2024-03-04T12:25:07.037179+0000","last_fullsized":"2024-03-04T12:25:07.037179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:17:08.203502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935989+0000","last_change":"2024-03-04T12:23:16.951920+0000","last_active":"2024-03-04T12:25:06.935989+0000","last_peered":"2024-03-04T12:25:06.935989+0000","last_clean":"2024-03-04T12:25:06.935989+0000","last_became_active":"2024-03-04T12:23:16.951507+0000","last_became_peered":"2024-03-04T12:23:16.951507+0000","last_unstale":"2024-03-04T12:25:06.935989+0000","last_undegraded":"2024-03-04T12:25:06.935989+0000","last_fullsized":"2024-03-04T12:25:06.935989+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:01:27.673692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":161,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:36.068598+0000","last_change":"2024-03-04T12:23:11.897722+0000","last_active":"2024-03-04T12:25:36.068598+0000","last_peered":"2024-03-04T12:25:36.068598+0000","last_clean":"2024-03-04T12:25:36.068598+0000","last_became_active":"2024-03-04T12:23:11.897276+0000","last_became_peered":"2024-03-04T12:23:11.897276+0000","last_unstale":"2024-03-04T12:25:36.068598+0000","last_undegraded":"2024-03-04T12:25:36.068598+0000","last_fullsized":"2024-03-04T12:25:36.068598+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:12:37.968804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036780+0000","last_change":"2024-03-04T12:23:12.914773+0000","last_active":"2024-03-04T12:25:07.036780+0000","last_peered":"2024-03-04T12:25:07.036780+0000","last_clean":"2024-03-04T12:25:07.036780+0000","last_became_active":"2024-03-04T12:23:12.914431+0000","last_became_peered":"2024-03-04T12:23:12.914431+0000","last_unstale":"2024-03-04T12:25:07.036780+0000","last_undegraded":"2024-03-04T12:25:07.036780+0000","last_fullsized":"2024-03-04T12:25:07.036780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:30:50.234491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932697+0000","last_change":"2024-03-04T12:23:16.952405+0000","last_active":"2024-03-04T12:25:06.932697+0000","last_peered":"2024-03-04T12:25:06.932697+0000","last_clean":"2024-03-04T12:25:06.932697+0000","last_became_active":"2024-03-04T12:23:16.951915+0000","last_became_peered":"2024-03-04T12:23:16.951915+0000","last_unstale":"2024-03-04T12:25:06.932697+0000","last_undegraded":"2024-03-04T12:25:06.932697+0000","last_fullsized":"2024-03-04T12:25:06.932697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:38:03.765883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938109+0000","last_change":"2024-03-04T12:23:11.903267+0000","last_active":"2024-03-04T12:25:06.938109+0000","last_peered":"2024-03-04T12:25:06.938109+0000","last_clean":"2024-03-04T12:25:06.938109+0000","last_became_active":"2024-03-04T12:23:11.902870+0000","last_became_peered":"2024-03-04T12:23:11.902870+0000","last_unstale":"2024-03-04T12:25:06.938109+0000","last_undegraded":"2024-03-04T12:25:06.938109+0000","last_fullsized":"2024-03-04T12:25:06.938109+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:33:43.679869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933267+0000","last_change":"2024-03-04T12:23:12.925303+0000","last_active":"2024-03-04T12:25:06.933267+0000","last_peered":"2024-03-04T12:25:06.933267+0000","last_clean":"2024-03-04T12:25:06.933267+0000","last_became_active":"2024-03-04T12:23:12.924945+0000","last_became_peered":"2024-03-04T12:23:12.924945+0000","last_unstale":"2024-03-04T12:25:06.933267+0000","last_undegraded":"2024-03-04T12:25:06.933267+0000","last_fullsized":"2024-03-04T12:25:06.933267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:29:49.680925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933148+0000","last_change":"2024-03-04T12:23:16.951457+0000","last_active":"2024-03-04T12:25:06.933148+0000","last_peered":"2024-03-04T12:25:06.933148+0000","last_clean":"2024-03-04T12:25:06.933148+0000","last_became_active":"2024-03-04T12:23:16.950835+0000","last_became_peered":"2024-03-04T12:23:16.950835+0000","last_unstale":"2024-03-04T12:25:06.933148+0000","last_undegraded":"2024-03-04T12:25:06.933148+0000","last_fullsized":"2024-03-04T12:25:06.933148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:39.337275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:29.397242+0000","last_change":"2024-03-04T12:23:11.903718+0000","last_active":"2024-03-04T12:25:29.397242+0000","last_peered":"2024-03-04T12:25:29.397242+0000","last_clean":"2024-03-04T12:25:29.397242+0000","last_became_active":"2024-03-04T12:23:11.902814+0000","last_became_peered":"2024-03-04T12:23:11.902814+0000","last_unstale":"2024-03-04T12:25:29.397242+0000","last_undegraded":"2024-03-04T12:25:29.397242+0000","last_fullsized":"2024-03-04T12:25:29.397242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:40:30.978576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276785+0000","last_change":"2024-03-04T12:24:05.958392+0000","last_active":"2024-03-04T12:25:07.276785+0000","last_peered":"2024-03-04T12:25:07.276785+0000","last_clean":"2024-03-04T12:25:07.276785+0000","last_became_active":"2024-03-04T12:24:05.958012+0000","last_became_peered":"2024-03-04T12:24:05.958012+0000","last_unstale":"2024-03-04T12:25:07.276785+0000","last_undegraded":"2024-03-04T12:25:07.276785+0000","last_fullsized":"2024-03-04T12:25:07.276785+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:43:50.400090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938459+0000","last_change":"2024-03-04T12:23:16.950046+0000","last_active":"2024-03-04T12:25:06.938459+0000","last_peered":"2024-03-04T12:25:06.938459+0000","last_clean":"2024-03-04T12:25:06.938459+0000","last_became_active":"2024-03-04T12:23:16.949632+0000","last_became_peered":"2024-03-04T12:23:16.949632+0000","last_unstale":"2024-03-04T12:25:06.938459+0000","last_undegraded":"2024-03-04T12:25:06.938459+0000","last_fullsized":"2024-03-04T12:25:06.938459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:13:25.817212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.935925+0000","last_change":"2024-03-04T12:23:11.897016+0000","last_active":"2024-03-04T12:25:06.935925+0000","last_peered":"2024-03-04T12:25:06.935925+0000","last_clean":"2024-03-04T12:25:06.935925+0000","last_became_active":"2024-03-04T12:23:11.896668+0000","last_became_peered":"2024-03-04T12:23:11.896668+0000","last_unstale":"2024-03-04T12:25:06.935925+0000","last_undegraded":"2024-03-04T12:25:06.935925+0000","last_fullsized":"2024-03-04T12:25:06.935925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:50:01.089658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.026051+0000","last_change":"2024-03-04T12:23:12.908169+0000","last_active":"2024-03-04T12:25:07.026051+0000","last_peered":"2024-03-04T12:25:07.026051+0000","last_clean":"2024-03-04T12:25:07.026051+0000","last_became_active":"2024-03-04T12:23:12.907786+0000","last_became_peered":"2024-03-04T12:23:12.907786+0000","last_unstale":"2024-03-04T12:25:07.026051+0000","last_undegraded":"2024-03-04T12:25:07.026051+0000","last_fullsized":"2024-03-04T12:25:07.026051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:53:10.602216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932425+0000","last_change":"2024-03-04T12:23:16.944232+0000","last_active":"2024-03-04T12:25:06.932425+0000","last_peered":"2024-03-04T12:25:06.932425+0000","last_clean":"2024-03-04T12:25:06.932425+0000","last_became_active":"2024-03-04T12:23:16.943855+0000","last_became_peered":"2024-03-04T12:23:16.943855+0000","last_unstale":"2024-03-04T12:25:06.932425+0000","last_undegraded":"2024-03-04T12:25:06.932425+0000","last_fullsized":"2024-03-04T12:25:06.932425+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:37:12.083622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":155,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:37.075695+0000","last_change":"2024-03-04T12:23:11.899742+0000","last_active":"2024-03-04T12:25:37.075695+0000","last_peered":"2024-03-04T12:25:37.075695+0000","last_clean":"2024-03-04T12:25:37.075695+0000","last_became_active":"2024-03-04T12:23:11.898445+0000","last_became_peered":"2024-03-04T12:23:11.898445+0000","last_unstale":"2024-03-04T12:25:37.075695+0000","last_undegraded":"2024-03-04T12:25:37.075695+0000","last_fullsized":"2024-03-04T12:25:37.075695+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:13.128348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.025878+0000","last_change":"2024-03-04T12:23:12.921321+0000","last_active":"2024-03-04T12:25:07.025878+0000","last_peered":"2024-03-04T12:25:07.025878+0000","last_clean":"2024-03-04T12:25:07.025878+0000","last_became_active":"2024-03-04T12:23:12.918119+0000","last_became_peered":"2024-03-04T12:23:12.918119+0000","last_unstale":"2024-03-04T12:25:07.025878+0000","last_undegraded":"2024-03-04T12:25:07.025878+0000","last_fullsized":"2024-03-04T12:25:07.025878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:22:31.693583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932634+0000","last_change":"2024-03-04T12:23:16.943887+0000","last_active":"2024-03-04T12:25:06.932634+0000","last_peered":"2024-03-04T12:25:06.932634+0000","last_clean":"2024-03-04T12:25:06.932634+0000","last_became_active":"2024-03-04T12:23:16.943487+0000","last_became_peered":"2024-03-04T12:23:16.943487+0000","last_unstale":"2024-03-04T12:25:06.932634+0000","last_undegraded":"2024-03-04T12:25:06.932634+0000","last_fullsized":"2024-03-04T12:25:06.932634+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:41:12.451176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933359+0000","last_change":"2024-03-04T12:23:12.918363+0000","last_active":"2024-03-04T12:25:06.933359+0000","last_peered":"2024-03-04T12:25:06.933359+0000","last_clean":"2024-03-04T12:25:06.933359+0000","last_became_active":"2024-03-04T12:23:12.917563+0000","last_became_peered":"2024-03-04T12:23:12.917563+0000","last_unstale":"2024-03-04T12:25:06.933359+0000","last_undegraded":"2024-03-04T12:25:06.933359+0000","last_fullsized":"2024-03-04T12:25:06.933359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:26:35.297513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'7","reported_seq":159,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:38.071274+0000","last_change":"2024-03-04T12:23:11.904072+0000","last_active":"2024-03-04T12:25:38.071274+0000","last_peered":"2024-03-04T12:25:38.071274+0000","last_clean":"2024-03-04T12:25:38.071274+0000","last_became_active":"2024-03-04T12:23:11.903097+0000","last_became_peered":"2024-03-04T12:23:11.903097+0000","last_unstale":"2024-03-04T12:25:38.071274+0000","last_undegraded":"2024-03-04T12:25:38.071274+0000","last_fullsized":"2024-03-04T12:25:38.071274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:36.606667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036799+0000","last_change":"2024-03-04T12:23:16.951019+0000","last_active":"2024-03-04T12:25:07.036799+0000","last_peered":"2024-03-04T12:25:07.036799+0000","last_clean":"2024-03-04T12:25:07.036799+0000","last_became_active":"2024-03-04T12:23:16.950524+0000","last_became_peered":"2024-03-04T12:23:16.950524+0000","last_unstale":"2024-03-04T12:25:07.036799+0000","last_undegraded":"2024-03-04T12:25:07.036799+0000","last_fullsized":"2024-03-04T12:25:07.036799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:56:48.376539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938226+0000","last_change":"2024-03-04T12:23:12.927615+0000","last_active":"2024-03-04T12:25:06.938226+0000","last_peered":"2024-03-04T12:25:06.938226+0000","last_clean":"2024-03-04T12:25:06.938226+0000","last_became_active":"2024-03-04T12:23:12.927190+0000","last_became_peered":"2024-03-04T12:23:12.927190+0000","last_unstale":"2024-03-04T12:25:06.938226+0000","last_undegraded":"2024-03-04T12:25:06.938226+0000","last_fullsized":"2024-03-04T12:25:06.938226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:54:19.019030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933773+0000","last_change":"2024-03-04T12:23:11.893542+0000","last_active":"2024-03-04T12:25:06.933773+0000","last_peered":"2024-03-04T12:25:06.933773+0000","last_clean":"2024-03-04T12:25:06.933773+0000","last_became_active":"2024-03-04T12:23:11.893166+0000","last_became_peered":"2024-03-04T12:23:11.893166+0000","last_unstale":"2024-03-04T12:25:06.933773+0000","last_undegraded":"2024-03-04T12:25:06.933773+0000","last_fullsized":"2024-03-04T12:25:06.933773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:04:32.117863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":365,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036450+0000","last_change":"2024-03-04T12:21:55.865103+0000","last_active":"2024-03-04T12:25:07.036450+0000","last_peered":"2024-03-04T12:25:07.036450+0000","last_clean":"2024-03-04T12:25:07.036450+0000","last_became_active":"2024-03-04T12:21:55.864553+0000","last_became_peered":"2024-03-04T12:21:55.864553+0000","last_unstale":"2024-03-04T12:25:07.036450+0000","last_undegraded":"2024-03-04T12:25:07.036450+0000","last_fullsized":"2024-03-04T12:25:07.036450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:18:28.442423+0000","last_clean_scrub_stamp":"2024-03-04T12:18:28.442423+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:30:31.052782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933731+0000","last_change":"2024-03-04T12:23:16.945852+0000","last_active":"2024-03-04T12:25:06.933731+0000","last_peered":"2024-03-04T12:25:06.933731+0000","last_clean":"2024-03-04T12:25:06.933731+0000","last_became_active":"2024-03-04T12:23:16.945496+0000","last_became_peered":"2024-03-04T12:23:16.945496+0000","last_unstale":"2024-03-04T12:25:06.933731+0000","last_undegraded":"2024-03-04T12:25:06.933731+0000","last_fullsized":"2024-03-04T12:25:06.933731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:35:10.880757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932050+0000","last_change":"2024-03-04T12:23:12.919219+0000","last_active":"2024-03-04T12:25:06.932050+0000","last_peered":"2024-03-04T12:25:06.932050+0000","last_clean":"2024-03-04T12:25:06.932050+0000","last_became_active":"2024-03-04T12:23:12.918897+0000","last_became_peered":"2024-03-04T12:23:12.918897+0000","last_unstale":"2024-03-04T12:25:06.932050+0000","last_undegraded":"2024-03-04T12:25:06.932050+0000","last_fullsized":"2024-03-04T12:25:06.932050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:24:32.594172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036668+0000","last_change":"2024-03-04T12:23:11.894544+0000","last_active":"2024-03-04T12:25:07.036668+0000","last_peered":"2024-03-04T12:25:07.036668+0000","last_clean":"2024-03-04T12:25:07.036668+0000","last_became_active":"2024-03-04T12:23:11.894188+0000","last_became_peered":"2024-03-04T12:23:11.894188+0000","last_unstale":"2024-03-04T12:25:07.036668+0000","last_undegraded":"2024-03-04T12:25:07.036668+0000","last_fullsized":"2024-03-04T12:25:07.036668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:28:14.687530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276193+0000","last_change":"2024-03-04T12:23:16.945510+0000","last_active":"2024-03-04T12:25:07.276193+0000","last_peered":"2024-03-04T12:25:07.276193+0000","last_clean":"2024-03-04T12:25:07.276193+0000","last_became_active":"2024-03-04T12:23:16.945140+0000","last_became_peered":"2024-03-04T12:23:16.945140+0000","last_unstale":"2024-03-04T12:25:07.276193+0000","last_undegraded":"2024-03-04T12:25:07.276193+0000","last_fullsized":"2024-03-04T12:25:07.276193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:44.102503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938493+0000","last_change":"2024-03-04T12:23:11.904515+0000","last_active":"2024-03-04T12:25:06.938493+0000","last_peered":"2024-03-04T12:25:06.938493+0000","last_clean":"2024-03-04T12:25:06.938493+0000","last_became_active":"2024-03-04T12:23:11.904149+0000","last_became_peered":"2024-03-04T12:23:11.904149+0000","last_unstale":"2024-03-04T12:25:06.938493+0000","last_undegraded":"2024-03-04T12:25:06.938493+0000","last_fullsized":"2024-03-04T12:25:06.938493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:04:51.382626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932543+0000","last_change":"2024-03-04T12:23:12.908749+0000","last_active":"2024-03-04T12:25:06.932543+0000","last_peered":"2024-03-04T12:25:06.932543+0000","last_clean":"2024-03-04T12:25:06.932543+0000","last_became_active":"2024-03-04T12:23:12.908405+0000","last_became_peered":"2024-03-04T12:23:12.908405+0000","last_unstale":"2024-03-04T12:25:06.932543+0000","last_undegraded":"2024-03-04T12:25:06.932543+0000","last_fullsized":"2024-03-04T12:25:06.932543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:10.472953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938204+0000","last_change":"2024-03-04T12:23:16.951145+0000","last_active":"2024-03-04T12:25:06.938204+0000","last_peered":"2024-03-04T12:25:06.938204+0000","last_clean":"2024-03-04T12:25:06.938204+0000","last_became_active":"2024-03-04T12:23:16.950702+0000","last_became_peered":"2024-03-04T12:23:16.950702+0000","last_unstale":"2024-03-04T12:25:06.938204+0000","last_undegraded":"2024-03-04T12:25:06.938204+0000","last_fullsized":"2024-03-04T12:25:06.938204+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:55:42.720616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'14","reported_seq":62,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:36.932982+0000","last_change":"2024-03-04T12:25:07.284006+0000","last_active":"2024-03-04T12:25:36.932982+0000","last_peered":"2024-03-04T12:25:36.932982+0000","last_clean":"2024-03-04T12:25:36.932982+0000","last_became_active":"2024-03-04T12:25:07.277446+0000","last_became_peered":"2024-03-04T12:25:07.277446+0000","last_unstale":"2024-03-04T12:25:36.932982+0000","last_undegraded":"2024-03-04T12:25:36.932982+0000","last_fullsized":"2024-03-04T12:25:36.932982+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:08:40.719222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938663+0000","last_change":"2024-03-04T12:23:12.925763+0000","last_active":"2024-03-04T12:25:06.938663+0000","last_peered":"2024-03-04T12:25:06.938663+0000","last_clean":"2024-03-04T12:25:06.938663+0000","last_became_active":"2024-03-04T12:23:12.925429+0000","last_became_peered":"2024-03-04T12:23:12.925429+0000","last_unstale":"2024-03-04T12:25:06.938663+0000","last_undegraded":"2024-03-04T12:25:06.938663+0000","last_fullsized":"2024-03-04T12:25:06.938663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:19:46.814459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939067+0000","last_change":"2024-03-04T12:23:16.942624+0000","last_active":"2024-03-04T12:25:06.939067+0000","last_peered":"2024-03-04T12:25:06.939067+0000","last_clean":"2024-03-04T12:25:06.939067+0000","last_became_active":"2024-03-04T12:23:16.942287+0000","last_became_peered":"2024-03-04T12:23:16.942287+0000","last_unstale":"2024-03-04T12:25:06.939067+0000","last_undegraded":"2024-03-04T12:25:06.939067+0000","last_fullsized":"2024-03-04T12:25:06.939067+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:59.479611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933762+0000","last_change":"2024-03-04T12:23:11.891504+0000","last_active":"2024-03-04T12:25:06.933762+0000","last_peered":"2024-03-04T12:25:06.933762+0000","last_clean":"2024-03-04T12:25:06.933762+0000","last_became_active":"2024-03-04T12:23:11.891134+0000","last_became_peered":"2024-03-04T12:23:11.891134+0000","last_unstale":"2024-03-04T12:25:06.933762+0000","last_undegraded":"2024-03-04T12:25:06.933762+0000","last_fullsized":"2024-03-04T12:25:06.933762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:34:41.481003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940405+0000","last_change":"2024-03-04T12:23:12.928989+0000","last_active":"2024-03-04T12:25:06.940405+0000","last_peered":"2024-03-04T12:25:06.940405+0000","last_clean":"2024-03-04T12:25:06.940405+0000","last_became_active":"2024-03-04T12:23:12.928609+0000","last_became_peered":"2024-03-04T12:23:12.928609+0000","last_unstale":"2024-03-04T12:25:06.940405+0000","last_undegraded":"2024-03-04T12:25:06.940405+0000","last_fullsized":"2024-03-04T12:25:06.940405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:10:39.979546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933694+0000","last_change":"2024-03-04T12:23:16.949590+0000","last_active":"2024-03-04T12:25:06.933694+0000","last_peered":"2024-03-04T12:25:06.933694+0000","last_clean":"2024-03-04T12:25:06.933694+0000","last_became_active":"2024-03-04T12:23:16.949086+0000","last_became_peered":"2024-03-04T12:23:16.949086+0000","last_unstale":"2024-03-04T12:25:06.933694+0000","last_undegraded":"2024-03-04T12:25:06.933694+0000","last_fullsized":"2024-03-04T12:25:06.933694+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:07:09.819097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.286558+0000","last_change":"2024-03-04T12:25:07.286087+0000","last_active":"2024-03-04T12:25:07.286558+0000","last_peered":"2024-03-04T12:25:07.286558+0000","last_clean":"2024-03-04T12:25:07.286558+0000","last_became_active":"2024-03-04T12:25:07.277551+0000","last_became_peered":"2024-03-04T12:25:07.277551+0000","last_unstale":"2024-03-04T12:25:07.286558+0000","last_undegraded":"2024-03-04T12:25:07.286558+0000","last_fullsized":"2024-03-04T12:25:07.286558+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T12:27:23.167325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938356+0000","last_change":"2024-03-04T12:23:12.907403+0000","last_active":"2024-03-04T12:25:06.938356+0000","last_peered":"2024-03-04T12:25:06.938356+0000","last_clean":"2024-03-04T12:25:06.938356+0000","last_became_active":"2024-03-04T12:23:12.906674+0000","last_became_peered":"2024-03-04T12:23:12.906674+0000","last_unstale":"2024-03-04T12:25:06.938356+0000","last_undegraded":"2024-03-04T12:25:06.938356+0000","last_fullsized":"2024-03-04T12:25:06.938356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:49:01.133676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936150+0000","last_change":"2024-03-04T12:23:16.955113+0000","last_active":"2024-03-04T12:25:06.936150+0000","last_peered":"2024-03-04T12:25:06.936150+0000","last_clean":"2024-03-04T12:25:06.936150+0000","last_became_active":"2024-03-04T12:23:16.951624+0000","last_became_peered":"2024-03-04T12:23:16.951624+0000","last_unstale":"2024-03-04T12:25:06.936150+0000","last_undegraded":"2024-03-04T12:25:06.936150+0000","last_fullsized":"2024-03-04T12:25:06.936150+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:22:23.935460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938252+0000","last_change":"2024-03-04T12:23:11.900213+0000","last_active":"2024-03-04T12:25:06.938252+0000","last_peered":"2024-03-04T12:25:06.938252+0000","last_clean":"2024-03-04T12:25:06.938252+0000","last_became_active":"2024-03-04T12:23:11.899832+0000","last_became_peered":"2024-03-04T12:23:11.899832+0000","last_unstale":"2024-03-04T12:25:06.938252+0000","last_undegraded":"2024-03-04T12:25:06.938252+0000","last_fullsized":"2024-03-04T12:25:06.938252+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:55:24.553309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276791+0000","last_change":"2024-03-04T12:23:12.913315+0000","last_active":"2024-03-04T12:25:07.276791+0000","last_peered":"2024-03-04T12:25:07.276791+0000","last_clean":"2024-03-04T12:25:07.276791+0000","last_became_active":"2024-03-04T12:23:12.912850+0000","last_became_peered":"2024-03-04T12:23:12.912850+0000","last_unstale":"2024-03-04T12:25:07.276791+0000","last_undegraded":"2024-03-04T12:25:07.276791+0000","last_fullsized":"2024-03-04T12:25:07.276791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:57:55.140866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933591+0000","last_change":"2024-03-04T12:23:16.943295+0000","last_active":"2024-03-04T12:25:06.933591+0000","last_peered":"2024-03-04T12:25:06.933591+0000","last_clean":"2024-03-04T12:25:06.933591+0000","last_became_active":"2024-03-04T12:23:16.942853+0000","last_became_peered":"2024-03-04T12:23:16.942853+0000","last_unstale":"2024-03-04T12:25:06.933591+0000","last_undegraded":"2024-03-04T12:25:06.933591+0000","last_fullsized":"2024-03-04T12:25:06.933591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:05:51.162179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:22.480402+0000","last_change":"2024-03-04T12:23:11.897870+0000","last_active":"2024-03-04T12:25:22.480402+0000","last_peered":"2024-03-04T12:25:22.480402+0000","last_clean":"2024-03-04T12:25:22.480402+0000","last_became_active":"2024-03-04T12:23:11.897443+0000","last_became_peered":"2024-03-04T12:23:11.897443+0000","last_unstale":"2024-03-04T12:25:22.480402+0000","last_undegraded":"2024-03-04T12:25:22.480402+0000","last_fullsized":"2024-03-04T12:25:22.480402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:49:37.875541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938638+0000","last_change":"2024-03-04T12:23:12.916441+0000","last_active":"2024-03-04T12:25:06.938638+0000","last_peered":"2024-03-04T12:25:06.938638+0000","last_clean":"2024-03-04T12:25:06.938638+0000","last_became_active":"2024-03-04T12:23:12.916078+0000","last_became_peered":"2024-03-04T12:23:12.916078+0000","last_unstale":"2024-03-04T12:25:06.938638+0000","last_undegraded":"2024-03-04T12:25:06.938638+0000","last_fullsized":"2024-03-04T12:25:06.938638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:45:23.663627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933671+0000","last_change":"2024-03-04T12:23:16.953341+0000","last_active":"2024-03-04T12:25:06.933671+0000","last_peered":"2024-03-04T12:25:06.933671+0000","last_clean":"2024-03-04T12:25:06.933671+0000","last_became_active":"2024-03-04T12:23:16.952878+0000","last_became_peered":"2024-03-04T12:23:16.952878+0000","last_unstale":"2024-03-04T12:25:06.933671+0000","last_undegraded":"2024-03-04T12:25:06.933671+0000","last_fullsized":"2024-03-04T12:25:06.933671+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:08:55.575263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938122+0000","last_change":"2024-03-04T12:23:11.903182+0000","last_active":"2024-03-04T12:25:06.938122+0000","last_peered":"2024-03-04T12:25:06.938122+0000","last_clean":"2024-03-04T12:25:06.938122+0000","last_became_active":"2024-03-04T12:23:11.902709+0000","last_became_peered":"2024-03-04T12:23:11.902709+0000","last_unstale":"2024-03-04T12:25:06.938122+0000","last_undegraded":"2024-03-04T12:25:06.938122+0000","last_fullsized":"2024-03-04T12:25:06.938122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:13:33.438929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037059+0000","last_change":"2024-03-04T12:23:12.931339+0000","last_active":"2024-03-04T12:25:07.037059+0000","last_peered":"2024-03-04T12:25:07.037059+0000","last_clean":"2024-03-04T12:25:07.037059+0000","last_became_active":"2024-03-04T12:23:12.930213+0000","last_became_peered":"2024-03-04T12:23:12.930213+0000","last_unstale":"2024-03-04T12:25:07.037059+0000","last_undegraded":"2024-03-04T12:25:07.037059+0000","last_fullsized":"2024-03-04T12:25:07.037059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:55:23.084139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936423+0000","last_change":"2024-03-04T12:23:16.948723+0000","last_active":"2024-03-04T12:25:06.936423+0000","last_peered":"2024-03-04T12:25:06.936423+0000","last_clean":"2024-03-04T12:25:06.936423+0000","last_became_active":"2024-03-04T12:23:16.948256+0000","last_became_peered":"2024-03-04T12:23:16.948256+0000","last_unstale":"2024-03-04T12:25:06.936423+0000","last_undegraded":"2024-03-04T12:25:06.936423+0000","last_fullsized":"2024-03-04T12:25:06.936423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:22.060517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":155,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:36.081533+0000","last_change":"2024-03-04T12:23:11.898475+0000","last_active":"2024-03-04T12:25:36.081533+0000","last_peered":"2024-03-04T12:25:36.081533+0000","last_clean":"2024-03-04T12:25:36.081533+0000","last_became_active":"2024-03-04T12:23:11.898117+0000","last_became_peered":"2024-03-04T12:23:11.898117+0000","last_unstale":"2024-03-04T12:25:36.081533+0000","last_undegraded":"2024-03-04T12:25:36.081533+0000","last_fullsized":"2024-03-04T12:25:36.081533+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:21:25.119591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037220+0000","last_change":"2024-03-04T12:23:12.929772+0000","last_active":"2024-03-04T12:25:07.037220+0000","last_peered":"2024-03-04T12:25:07.037220+0000","last_clean":"2024-03-04T12:25:07.037220+0000","last_became_active":"2024-03-04T12:23:12.929405+0000","last_became_peered":"2024-03-04T12:23:12.929405+0000","last_unstale":"2024-03-04T12:25:07.037220+0000","last_undegraded":"2024-03-04T12:25:07.037220+0000","last_fullsized":"2024-03-04T12:25:07.037220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:22:42.874989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940260+0000","last_change":"2024-03-04T12:23:16.941715+0000","last_active":"2024-03-04T12:25:06.940260+0000","last_peered":"2024-03-04T12:25:06.940260+0000","last_clean":"2024-03-04T12:25:06.940260+0000","last_became_active":"2024-03-04T12:23:16.941298+0000","last_became_peered":"2024-03-04T12:23:16.941298+0000","last_unstale":"2024-03-04T12:25:06.940260+0000","last_undegraded":"2024-03-04T12:25:06.940260+0000","last_fullsized":"2024-03-04T12:25:06.940260+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:09:02.019531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933616+0000","last_change":"2024-03-04T12:23:11.896413+0000","last_active":"2024-03-04T12:25:06.933616+0000","last_peered":"2024-03-04T12:25:06.933616+0000","last_clean":"2024-03-04T12:25:06.933616+0000","last_became_active":"2024-03-04T12:23:11.895679+0000","last_became_peered":"2024-03-04T12:23:11.895679+0000","last_unstale":"2024-03-04T12:25:06.933616+0000","last_undegraded":"2024-03-04T12:25:06.933616+0000","last_fullsized":"2024-03-04T12:25:06.933616+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:25:41.967530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.036828+0000","last_change":"2024-03-04T12:23:12.920468+0000","last_active":"2024-03-04T12:25:07.036828+0000","last_peered":"2024-03-04T12:25:07.036828+0000","last_clean":"2024-03-04T12:25:07.036828+0000","last_became_active":"2024-03-04T12:23:12.920113+0000","last_became_peered":"2024-03-04T12:23:12.920113+0000","last_unstale":"2024-03-04T12:25:07.036828+0000","last_undegraded":"2024-03-04T12:25:07.036828+0000","last_fullsized":"2024-03-04T12:25:07.036828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:00:02.283518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.937863+0000","last_change":"2024-03-04T12:24:06.030444+0000","last_active":"2024-03-04T12:25:06.937863+0000","last_peered":"2024-03-04T12:25:06.937863+0000","last_clean":"2024-03-04T12:25:06.937863+0000","last_became_active":"2024-03-04T12:24:06.029934+0000","last_became_peered":"2024-03-04T12:24:06.029934+0000","last_unstale":"2024-03-04T12:25:06.937863+0000","last_undegraded":"2024-03-04T12:25:06.937863+0000","last_fullsized":"2024-03-04T12:25:06.937863+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:15.912888+0000","last_clean_scrub_stamp":"2024-03-04T12:23:15.912888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:08:16.820509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932650+0000","last_change":"2024-03-04T12:23:11.903323+0000","last_active":"2024-03-04T12:25:06.932650+0000","last_peered":"2024-03-04T12:25:06.932650+0000","last_clean":"2024-03-04T12:25:06.932650+0000","last_became_active":"2024-03-04T12:23:11.902987+0000","last_became_peered":"2024-03-04T12:23:11.902987+0000","last_unstale":"2024-03-04T12:25:06.932650+0000","last_undegraded":"2024-03-04T12:25:06.932650+0000","last_fullsized":"2024-03-04T12:25:06.932650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:10.872133+0000","last_clean_scrub_stamp":"2024-03-04T12:23:10.872133+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:53:14.044391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.037158+0000","last_change":"2024-03-04T12:23:12.913491+0000","last_active":"2024-03-04T12:25:07.037158+0000","last_peered":"2024-03-04T12:25:07.037158+0000","last_clean":"2024-03-04T12:25:07.037158+0000","last_became_active":"2024-03-04T12:23:12.912103+0000","last_became_peered":"2024-03-04T12:23:12.912103+0000","last_unstale":"2024-03-04T12:25:07.037158+0000","last_undegraded":"2024-03-04T12:25:07.037158+0000","last_fullsized":"2024-03-04T12:25:07.037158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:05:38.033596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276868+0000","last_change":"2024-03-04T12:23:12.908437+0000","last_active":"2024-03-04T12:25:07.276868+0000","last_peered":"2024-03-04T12:25:07.276868+0000","last_clean":"2024-03-04T12:25:07.276868+0000","last_became_active":"2024-03-04T12:23:12.908071+0000","last_became_peered":"2024-03-04T12:23:12.908071+0000","last_unstale":"2024-03-04T12:25:07.276868+0000","last_undegraded":"2024-03-04T12:25:07.276868+0000","last_fullsized":"2024-03-04T12:25:07.276868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:20.112645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939915+0000","last_change":"2024-03-04T12:23:12.921645+0000","last_active":"2024-03-04T12:25:06.939915+0000","last_peered":"2024-03-04T12:25:06.939915+0000","last_clean":"2024-03-04T12:25:06.939915+0000","last_became_active":"2024-03-04T12:23:12.921275+0000","last_became_peered":"2024-03-04T12:23:12.921275+0000","last_unstale":"2024-03-04T12:25:06.939915+0000","last_undegraded":"2024-03-04T12:25:06.939915+0000","last_fullsized":"2024-03-04T12:25:06.939915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:56:17.740782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939311+0000","last_change":"2024-03-04T12:23:12.919813+0000","last_active":"2024-03-04T12:25:06.939311+0000","last_peered":"2024-03-04T12:25:06.939311+0000","last_clean":"2024-03-04T12:25:06.939311+0000","last_became_active":"2024-03-04T12:23:12.919469+0000","last_became_peered":"2024-03-04T12:23:12.919469+0000","last_unstale":"2024-03-04T12:25:06.939311+0000","last_undegraded":"2024-03-04T12:25:06.939311+0000","last_fullsized":"2024-03-04T12:25:06.939311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:44:46.720268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.940192+0000","last_change":"2024-03-04T12:23:12.918476+0000","last_active":"2024-03-04T12:25:06.940192+0000","last_peered":"2024-03-04T12:25:06.940192+0000","last_clean":"2024-03-04T12:25:06.940192+0000","last_became_active":"2024-03-04T12:23:12.917743+0000","last_became_peered":"2024-03-04T12:23:12.917743+0000","last_unstale":"2024-03-04T12:25:06.940192+0000","last_undegraded":"2024-03-04T12:25:06.940192+0000","last_fullsized":"2024-03-04T12:25:06.940192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T14:03:20.246769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932605+0000","last_change":"2024-03-04T12:23:12.918254+0000","last_active":"2024-03-04T12:25:06.932605+0000","last_peered":"2024-03-04T12:25:06.932605+0000","last_clean":"2024-03-04T12:25:06.932605+0000","last_became_active":"2024-03-04T12:23:12.917922+0000","last_became_peered":"2024-03-04T12:23:12.917922+0000","last_unstale":"2024-03-04T12:25:06.932605+0000","last_undegraded":"2024-03-04T12:25:06.932605+0000","last_fullsized":"2024-03-04T12:25:06.932605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:48:20.847984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276507+0000","last_change":"2024-03-04T12:23:12.910368+0000","last_active":"2024-03-04T12:25:07.276507+0000","last_peered":"2024-03-04T12:25:07.276507+0000","last_clean":"2024-03-04T12:25:07.276507+0000","last_became_active":"2024-03-04T12:23:12.910017+0000","last_became_peered":"2024-03-04T12:23:12.910017+0000","last_unstale":"2024-03-04T12:25:07.276507+0000","last_undegraded":"2024-03-04T12:25:07.276507+0000","last_fullsized":"2024-03-04T12:25:07.276507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:15:22.093644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939872+0000","last_change":"2024-03-04T12:23:12.921021+0000","last_active":"2024-03-04T12:25:06.939872+0000","last_peered":"2024-03-04T12:25:06.939872+0000","last_clean":"2024-03-04T12:25:06.939872+0000","last_became_active":"2024-03-04T12:23:12.920600+0000","last_became_peered":"2024-03-04T12:23:12.920600+0000","last_unstale":"2024-03-04T12:25:06.939872+0000","last_undegraded":"2024-03-04T12:25:06.939872+0000","last_fullsized":"2024-03-04T12:25:06.939872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:00:31.706900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.939264+0000","last_change":"2024-03-04T12:24:05.957607+0000","last_active":"2024-03-04T12:25:06.939264+0000","last_peered":"2024-03-04T12:25:06.939264+0000","last_clean":"2024-03-04T12:25:06.939264+0000","last_became_active":"2024-03-04T12:24:05.957335+0000","last_became_peered":"2024-03-04T12:24:05.957335+0000","last_unstale":"2024-03-04T12:25:06.939264+0000","last_undegraded":"2024-03-04T12:25:06.939264+0000","last_fullsized":"2024-03-04T12:25:06.939264+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:47:12.926219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.933250+0000","last_change":"2024-03-04T12:23:12.911394+0000","last_active":"2024-03-04T12:25:06.933250+0000","last_peered":"2024-03-04T12:25:06.933250+0000","last_clean":"2024-03-04T12:25:06.933250+0000","last_became_active":"2024-03-04T12:23:12.911061+0000","last_became_peered":"2024-03-04T12:23:12.911061+0000","last_unstale":"2024-03-04T12:25:06.933250+0000","last_undegraded":"2024-03-04T12:25:06.933250+0000","last_fullsized":"2024-03-04T12:25:06.933250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T13:39:39.880383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.938372+0000","last_change":"2024-03-04T12:23:12.911356+0000","last_active":"2024-03-04T12:25:06.938372+0000","last_peered":"2024-03-04T12:25:06.938372+0000","last_clean":"2024-03-04T12:25:06.938372+0000","last_became_active":"2024-03-04T12:23:12.911029+0000","last_became_peered":"2024-03-04T12:23:12.911029+0000","last_unstale":"2024-03-04T12:25:06.938372+0000","last_undegraded":"2024-03-04T12:25:06.938372+0000","last_fullsized":"2024-03-04T12:25:06.938372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:21:52.611625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.936612+0000","last_change":"2024-03-04T12:23:12.924921+0000","last_active":"2024-03-04T12:25:06.936612+0000","last_peered":"2024-03-04T12:25:06.936612+0000","last_clean":"2024-03-04T12:25:06.936612+0000","last_became_active":"2024-03-04T12:23:12.924558+0000","last_became_peered":"2024-03-04T12:23:12.924558+0000","last_unstale":"2024-03-04T12:25:06.936612+0000","last_undegraded":"2024-03-04T12:25:06.936612+0000","last_fullsized":"2024-03-04T12:25:06.936612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:10:53.238440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:06.932437+0000","last_change":"2024-03-04T12:23:12.913106+0000","last_active":"2024-03-04T12:25:06.932437+0000","last_peered":"2024-03-04T12:25:06.932437+0000","last_clean":"2024-03-04T12:25:06.932437+0000","last_became_active":"2024-03-04T12:23:12.912741+0000","last_became_peered":"2024-03-04T12:23:12.912741+0000","last_unstale":"2024-03-04T12:25:06.932437+0000","last_undegraded":"2024-03-04T12:25:06.932437+0000","last_fullsized":"2024-03-04T12:25:06.932437+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:46:21.406383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-04T12:25:07.276375+0000","last_change":"2024-03-04T12:23:12.918082+0000","last_active":"2024-03-04T12:25:07.276375+0000","last_peered":"2024-03-04T12:25:07.276375+0000","last_clean":"2024-03-04T12:25:07.276375+0000","last_became_active":"2024-03-04T12:23:12.917754+0000","last_became_peered":"2024-03-04T12:23:12.917754+0000","last_unstale":"2024-03-04T12:25:07.276375+0000","last_undegraded":"2024-03-04T12:25:07.276375+0000","last_fullsized":"2024-03-04T12:25:07.276375+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T12:23:11.876379+0000","last_clean_scrub_stamp":"2024-03-04T12:23:11.876379+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:26:56.601779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":286257,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68620,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1221,"num_read_kb":689,"num_write":261,"num_write_kb":152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":210956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383724,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1028096,"data_stored":520215,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28164,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716988,"statfs":{"total":95995035648,"available":95966195712,"internally_reserved":0,"allocated":1175552,"data_stored":1168389,"data_compressed":63026,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808887,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":524241,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070524,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":520523,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332160,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1101824,"data_stored":1118001,"data_compressed":63026,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515649,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":577505,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1052672,"data_stored":517105,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36424,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708728,"statfs":{"total":95995035648,"available":95957737472,"internally_reserved":0,"allocated":1114112,"data_stored":578565,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1040384,"data_stored":515357,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":520255,"data_compressed":26630,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32260,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712892,"statfs":{"total":95995035648,"available":95962001408,"internally_reserved":0,"allocated":1110016,"data_stored":1113051,"data_compressed":63026,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34759,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36396,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T12:25:42.900 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:25:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.031+0000 7fe95c759640 1 -- 172.21.15.60:0/1697526618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe954101e10 msgr2=0x7fe9541086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.031+0000 7fe95c759640 1 --2- 172.21.15.60:0/1697526618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe954101e10 0x7fe9541086a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe94c009900 tx=0x7fe94c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.031+0000 7fe95c759640 1 -- 172.21.15.60:0/1697526618 shutdown_connections 2024-03-04T12:25:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.031+0000 7fe95c759640 1 --2- 172.21.15.60:0/1697526618 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe954101e10 0x7fe9541086a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.031+0000 7fe95c759640 1 --2- 172.21.15.60:0/1697526618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe954101450 0x7fe9541018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.031+0000 7fe95c759640 1 --2- 172.21.15.60:0/1697526618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe954100260 0x7fe954100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.031+0000 7fe95c759640 1 -- 172.21.15.60:0/1697526618 >> 172.21.15.60:0/1697526618 conn(0x7fe9540fb7d0 msgr2=0x7fe9540fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.031+0000 7fe95c759640 1 -- 172.21.15.60:0/1697526618 shutdown_connections 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.031+0000 7fe95c759640 1 -- 172.21.15.60:0/1697526618 wait complete. 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.032+0000 7fe95c759640 1 Processor -- start 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.032+0000 7fe95c759640 1 -- start start 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.032+0000 7fe95c759640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe954100260 0x7fe954197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.032+0000 7fe95c759640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe954101450 0x7fe954197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.032+0000 7fe95c759640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe954101e10 0x7fe95419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.032+0000 7fe95c759640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95410bef0 con 0x7fe954100260 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.032+0000 7fe95c759640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95410bd70 con 0x7fe954101e10 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.032+0000 7fe95c759640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95410c070 con 0x7fe954101450 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.032+0000 7fe95a4f5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe954100260 0x7fe954197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.032+0000 7fe95a4f5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe954100260 0x7fe954197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55912/0 (socket says 172.21.15.60:55912) 2024-03-04T12:25:43.042 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.032+0000 7fe95a4f5640 1 -- 172.21.15.60:0/3425028461 learned_addr learned my addr 172.21.15.60:0/3425028461 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.033+0000 7fe95a4f5640 1 -- 172.21.15.60:0/3425028461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe954101450 msgr2=0x7fe954197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.033+0000 7fe95a4f5640 1 --2- 172.21.15.60:0/3425028461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe954101450 0x7fe954197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.033+0000 7fe95a4f5640 1 -- 172.21.15.60:0/3425028461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe954101e10 msgr2=0x7fe95419ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.033+0000 7fe95a4f5640 1 --2- 172.21.15.60:0/3425028461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe954101e10 0x7fe95419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.033+0000 7fe95a4f5640 1 -- 172.21.15.60:0/3425028461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95419f240 con 0x7fe954100260 2024-03-04T12:25:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.033+0000 7fe95a4f5640 1 --2- 172.21.15.60:0/3425028461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe954100260 0x7fe954197620 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe94400ea80 tx=0x7fe94400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.033+0000 7fe9537fe640 1 -- 172.21.15.60:0/3425028461 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94400ce40 con 0x7fe954100260 2024-03-04T12:25:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.033+0000 7fe9537fe640 1 -- 172.21.15.60:0/3425028461 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe944011950 con 0x7fe954100260 2024-03-04T12:25:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.033+0000 7fe95c759640 1 -- 172.21.15.60:0/3425028461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95419f530 con 0x7fe954100260 2024-03-04T12:25:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.033+0000 7fe9537fe640 1 -- 172.21.15.60:0/3425028461 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94401ab80 con 0x7fe954100260 2024-03-04T12:25:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.033+0000 7fe95c759640 1 -- 172.21.15.60:0/3425028461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95419fad0 con 0x7fe954100260 2024-03-04T12:25:43.043 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.034+0000 7fe95c759640 1 -- 172.21.15.60:0/3425028461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95419fd90 con 0x7fe954100260 2024-03-04T12:25:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.035+0000 7fe9537fe640 1 -- 172.21.15.60:0/3425028461 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fe944011b00 con 0x7fe954100260 2024-03-04T12:25:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.035+0000 7fe9537fe640 1 --2- 172.21.15.60:0/3425028461 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe93c0a1e90 0x7fe93c0a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.044 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.036+0000 7fe959cf4640 1 --2- 172.21.15.60:0/3425028461 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe93c0a1e90 0x7fe93c0a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.036+0000 7fe9537fe640 1 -- 172.21.15.60:0/3425028461 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fe9440ce9b0 con 0x7fe954100260 2024-03-04T12:25:43.045 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.036+0000 7fe959cf4640 1 --2- 172.21.15.60:0/3425028461 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe93c0a1e90 0x7fe93c0a4350 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe948005fd0 tx=0x7fe948005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:43.047 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.040+0000 7fe9537fe640 1 -- 172.21.15.60:0/3425028461 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe944016030 con 0x7fe954100260 2024-03-04T12:25:43.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:43 smithi060 ceph-mon[32745]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:25:43.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.223+0000 7fe95c759640 1 -- 172.21.15.60:0/3425028461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9541a00d0 con 0x7fe954100260 2024-03-04T12:25:43.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.224+0000 7fe9537fe640 1 -- 172.21.15.60:0/3425028461 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24562 (secure 0 0 0) 0x7fe94409b890 con 0x7fe954100260 2024-03-04T12:25:43.232 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:43.232 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":93,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:25:06.929248+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:25:43.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.226+0000 7fe95c759640 1 -- 172.21.15.60:0/3425028461 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe93c0a1e90 msgr2=0x7fe93c0a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.226+0000 7fe95c759640 1 --2- 172.21.15.60:0/3425028461 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe93c0a1e90 0x7fe93c0a4350 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe948005fd0 tx=0x7fe948005850 comp rx=0 tx=0).stop 2024-03-04T12:25:43.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.226+0000 7fe95c759640 1 -- 172.21.15.60:0/3425028461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe954100260 msgr2=0x7fe954197620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.226+0000 7fe95c759640 1 --2- 172.21.15.60:0/3425028461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe954100260 0x7fe954197620 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe94400ea80 tx=0x7fe94400ef40 comp rx=0 tx=0).stop 2024-03-04T12:25:43.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.226+0000 7fe95c759640 1 -- 172.21.15.60:0/3425028461 shutdown_connections 2024-03-04T12:25:43.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.226+0000 7fe95c759640 1 --2- 172.21.15.60:0/3425028461 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fe93c0a1e90 0x7fe93c0a4350 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.226+0000 7fe95c759640 1 --2- 172.21.15.60:0/3425028461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe954101e10 0x7fe95419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.226+0000 7fe95c759640 1 --2- 172.21.15.60:0/3425028461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe954101450 0x7fe954197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.226+0000 7fe95c759640 1 --2- 172.21.15.60:0/3425028461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe954100260 0x7fe954197620 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.226+0000 7fe95c759640 1 -- 172.21.15.60:0/3425028461 >> 172.21.15.60:0/3425028461 conn(0x7fe9540fb7d0 msgr2=0x7fe954068600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:43.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.226+0000 7fe95c759640 1 -- 172.21.15.60:0/3425028461 shutdown_connections 2024-03-04T12:25:43.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.226+0000 7fe95c759640 1 -- 172.21.15.60:0/3425028461 wait complete. 2024-03-04T12:25:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T12:23:10.438928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T12:23:11.472998+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T12:23:15.162500+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T12:25:43.245 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:25:43.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:43 smithi086 ceph-mon[36851]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:25:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.344+0000 7feb2b153640 1 -- 172.21.15.60:0/1498870576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24100250 msgr2=0x7feb24100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.344+0000 7feb2b153640 1 --2- 172.21.15.60:0/1498870576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24100250 0x7feb24100670 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7feb18009900 tx=0x7feb1802bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.344+0000 7feb2b153640 1 -- 172.21.15.60:0/1498870576 shutdown_connections 2024-03-04T12:25:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.344+0000 7feb2b153640 1 --2- 172.21.15.60:0/1498870576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb24101e00 0x7feb24108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.344+0000 7feb2b153640 1 --2- 172.21.15.60:0/1498870576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24101440 0x7feb241018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.344+0000 7feb2b153640 1 --2- 172.21.15.60:0/1498870576 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24100250 0x7feb24100670 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.344+0000 7feb2b153640 1 -- 172.21.15.60:0/1498870576 >> 172.21.15.60:0/1498870576 conn(0x7feb240fb7e0 msgr2=0x7feb240fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.344+0000 7feb2b153640 1 -- 172.21.15.60:0/1498870576 shutdown_connections 2024-03-04T12:25:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.344+0000 7feb2b153640 1 -- 172.21.15.60:0/1498870576 wait complete. 2024-03-04T12:25:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb2b153640 1 Processor -- start 2024-03-04T12:25:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb2b153640 1 -- start start 2024-03-04T12:25:43.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb2b153640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100250 0x7feb241041e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb2b153640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb24101440 0x7feb24102820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb2b153640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24101e00 0x7feb24102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb2b153640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2410bee0 con 0x7feb24101e00 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb2b153640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2410bd60 con 0x7feb24101440 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb2b153640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2410c060 con 0x7feb24100250 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb296f0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24101e00 0x7feb24102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb296f0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24101e00 0x7feb24102d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55936/0 (socket says 172.21.15.60:55936) 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb296f0640 1 -- 172.21.15.60:0/2042432660 learned_addr learned my addr 172.21.15.60:0/2042432660 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb23fff640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb24101440 0x7feb24102820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb28eef640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100250 0x7feb241041e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb296f0640 1 -- 172.21.15.60:0/2042432660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100250 msgr2=0x7feb241041e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb296f0640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100250 0x7feb241041e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb296f0640 1 -- 172.21.15.60:0/2042432660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb24101440 msgr2=0x7feb24102820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb296f0640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb24101440 0x7feb24102820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb296f0640 1 -- 172.21.15.60:0/2042432660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb241035f0 con 0x7feb24101e00 2024-03-04T12:25:43.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb28eef640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100250 0x7feb241041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:43.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.345+0000 7feb296f0640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24101e00 0x7feb24102d60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00eaa0 tx=0x7feb1c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:43.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.346+0000 7feb21ffb640 1 -- 172.21.15.60:0/2042432660 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c00cdb0 con 0x7feb24101e00 2024-03-04T12:25:43.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.346+0000 7feb2b153640 1 -- 172.21.15.60:0/2042432660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb241a1790 con 0x7feb24101e00 2024-03-04T12:25:43.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.346+0000 7feb21ffb640 1 -- 172.21.15.60:0/2042432660 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb1c005020 con 0x7feb24101e00 2024-03-04T12:25:43.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.346+0000 7feb21ffb640 1 -- 172.21.15.60:0/2042432660 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1c0116b0 con 0x7feb24101e00 2024-03-04T12:25:43.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.346+0000 7feb2b153640 1 -- 172.21.15.60:0/2042432660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb241a1c90 con 0x7feb24101e00 2024-03-04T12:25:43.356 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.347+0000 7feb2b153640 1 -- 172.21.15.60:0/2042432660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb241a1fb0 con 0x7feb24101e00 2024-03-04T12:25:43.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:43 smithi063 ceph-mon[36717]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:25:43.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.348+0000 7feb21ffb640 1 -- 172.21.15.60:0/2042432660 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7feb1c005560 con 0x7feb24101e00 2024-03-04T12:25:43.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.348+0000 7feb21ffb640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7feb140a1e10 0x7feb140a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.348+0000 7feb28eef640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7feb140a1e10 0x7feb140a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.348+0000 7feb21ffb640 1 -- 172.21.15.60:0/2042432660 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7feb1c0ce590 con 0x7feb24101e00 2024-03-04T12:25:43.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.349+0000 7feb28eef640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7feb140a1e10 0x7feb140a42d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7feb1802c7b0 tx=0x7feb18005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:43.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.353+0000 7feb21ffb640 1 -- 172.21.15.60:0/2042432660 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feb1c016030 con 0x7feb24101e00 2024-03-04T12:25:43.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.534+0000 7feb2b153640 1 -- 172.21.15.60:0/2042432660 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb241a2590 con 0x7feb24101e00 2024-03-04T12:25:43.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.534+0000 7feb21ffb640 1 -- 172.21.15.60:0/2042432660 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb1c09b4d0 con 0x7feb24101e00 2024-03-04T12:25:43.542 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:25:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.536+0000 7feb2b153640 1 -- 172.21.15.60:0/2042432660 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7feb140a1e10 msgr2=0x7feb140a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.536+0000 7feb2b153640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7feb140a1e10 0x7feb140a42d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7feb1802c7b0 tx=0x7feb18005fd0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.536+0000 7feb2b153640 1 -- 172.21.15.60:0/2042432660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24101e00 msgr2=0x7feb24102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.536+0000 7feb2b153640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24101e00 0x7feb24102d60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7feb1c00eaa0 tx=0x7feb1c00ef60 comp rx=0 tx=0).stop 2024-03-04T12:25:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.536+0000 7feb2b153640 1 -- 172.21.15.60:0/2042432660 shutdown_connections 2024-03-04T12:25:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.536+0000 7feb2b153640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7feb140a1e10 0x7feb140a42d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.536+0000 7feb2b153640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7feb24101e00 0x7feb24102d60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.536+0000 7feb2b153640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb24101440 0x7feb24102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.536+0000 7feb2b153640 1 --2- 172.21.15.60:0/2042432660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb24100250 0x7feb241041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.536+0000 7feb2b153640 1 -- 172.21.15.60:0/2042432660 >> 172.21.15.60:0/2042432660 conn(0x7feb240fb7e0 msgr2=0x7feb240fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:43.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.536+0000 7feb2b153640 1 -- 172.21.15.60:0/2042432660 shutdown_connections 2024-03-04T12:25:43.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.536+0000 7feb2b153640 1 -- 172.21.15.60:0/2042432660 wait complete. 2024-03-04T12:25:43.554 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T12:25:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.652+0000 7f2489685640 1 -- 172.21.15.60:0/864781660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2484101e00 msgr2=0x7f2484108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.652+0000 7f2489685640 1 --2- 172.21.15.60:0/864781660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2484101e00 0x7f2484108690 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2474009ca0 tx=0x7f247402c100 comp rx=0 tx=0).stop 2024-03-04T12:25:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.652+0000 7f2489685640 1 -- 172.21.15.60:0/864781660 shutdown_connections 2024-03-04T12:25:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.652+0000 7f2489685640 1 --2- 172.21.15.60:0/864781660 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2484101e00 0x7f2484108690 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 --2- 172.21.15.60:0/864781660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2484101440 0x7f24841018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 --2- 172.21.15.60:0/864781660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2484100250 0x7f2484100670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 -- 172.21.15.60:0/864781660 >> 172.21.15.60:0/864781660 conn(0x7f24840fb7c0 msgr2=0x7f24840fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 -- 172.21.15.60:0/864781660 shutdown_connections 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 -- 172.21.15.60:0/864781660 wait complete. 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 Processor -- start 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 -- start start 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2484100250 0x7f248406a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2484101440 0x7f248419c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2484101e00 0x7f248419e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248410bee0 con 0x7f2484100250 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248410bd60 con 0x7f2484101e00 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.653+0000 7f2489685640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248410c060 con 0x7f2484101440 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f2482ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2484100250 0x7f248406a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f2482ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2484100250 0x7f248406a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55958/0 (socket says 172.21.15.60:55958) 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f2482ffd640 1 -- 172.21.15.60:0/117315029 learned_addr learned my addr 172.21.15.60:0/117315029 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f24827fc640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2484101440 0x7f248419c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f24837fe640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2484101e00 0x7f248419e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f2482ffd640 1 -- 172.21.15.60:0/117315029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2484101440 msgr2=0x7f248419c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f2482ffd640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2484101440 0x7f248419c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f2482ffd640 1 -- 172.21.15.60:0/117315029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2484101e00 msgr2=0x7f248419e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f2482ffd640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2484101e00 0x7f248419e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f2482ffd640 1 -- 172.21.15.60:0/117315029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248419f290 con 0x7f2484100250 2024-03-04T12:25:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f24837fe640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2484101e00 0x7f248419e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f2482ffd640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2484100250 0x7f248406a540 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f247c00e9c0 tx=0x7f247c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f246bfff640 1 -- 172.21.15.60:0/117315029 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c00cd80 con 0x7f2484100250 2024-03-04T12:25:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f2489685640 1 -- 172.21.15.60:0/117315029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248419f520 con 0x7f2484100250 2024-03-04T12:25:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f246bfff640 1 -- 172.21.15.60:0/117315029 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f247c0035f0 con 0x7f2484100250 2024-03-04T12:25:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f246bfff640 1 -- 172.21.15.60:0/117315029 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c010690 con 0x7f2484100250 2024-03-04T12:25:43.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.654+0000 7f2489685640 1 -- 172.21.15.60:0/117315029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248419fa70 con 0x7f2484100250 2024-03-04T12:25:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.655+0000 7f2489685640 1 -- 172.21.15.60:0/117315029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24841074a0 con 0x7f2484100250 2024-03-04T12:25:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.656+0000 7f246bfff640 1 -- 172.21.15.60:0/117315029 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f247c0037a0 con 0x7f2484100250 2024-03-04T12:25:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.657+0000 7f246bfff640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f246c0a1e10 0x7f246c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.657+0000 7f24827fc640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f246c0a1e10 0x7f246c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.657+0000 7f246bfff640 1 -- 172.21.15.60:0/117315029 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f247c0ce820 con 0x7f2484100250 2024-03-04T12:25:43.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.657+0000 7f24827fc640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f246c0a1e10 0x7f246c0a42d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f248406a2b0 tx=0x7f247800c4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:43.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.661+0000 7f246bfff640 1 -- 172.21.15.60:0/117315029 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f247c09b700 con 0x7f2484100250 2024-03-04T12:25:43.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.842+0000 7f2489685640 1 -- 172.21.15.60:0/117315029 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f248402d160 con 0x7f2484100250 2024-03-04T12:25:43.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.843+0000 7f246bfff640 1 -- 172.21.15.60:0/117315029 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f247c09b000 con 0x7f2484100250 2024-03-04T12:25:43.850 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:43.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.845+0000 7f2489685640 1 -- 172.21.15.60:0/117315029 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f246c0a1e10 msgr2=0x7f246c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.852 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.845+0000 7f2489685640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f246c0a1e10 0x7f246c0a42d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f248406a2b0 tx=0x7f247800c4a0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.845+0000 7f2489685640 1 -- 172.21.15.60:0/117315029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2484100250 msgr2=0x7f248406a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.845+0000 7f2489685640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2484100250 0x7f248406a540 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f247c00e9c0 tx=0x7f247c00ee80 comp rx=0 tx=0).stop 2024-03-04T12:25:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.845+0000 7f2489685640 1 -- 172.21.15.60:0/117315029 shutdown_connections 2024-03-04T12:25:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.845+0000 7f2489685640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f246c0a1e10 0x7f246c0a42d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.845+0000 7f2489685640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2484101e00 0x7f248419e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.845+0000 7f2489685640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2484101440 0x7f248419c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.845+0000 7f2489685640 1 --2- 172.21.15.60:0/117315029 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2484100250 0x7f248406a540 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.845+0000 7f2489685640 1 -- 172.21.15.60:0/117315029 >> 172.21.15.60:0/117315029 conn(0x7f24840fb7c0 msgr2=0x7f2484104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.845+0000 7f2489685640 1 -- 172.21.15.60:0/117315029 shutdown_connections 2024-03-04T12:25:43.853 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.845+0000 7f2489685640 1 -- 172.21.15.60:0/117315029 wait complete. 2024-03-04T12:25:43.863 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T12:25:43.972 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.963+0000 7fd64dd7e640 1 -- 172.21.15.60:0/2211659748 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6481046f0 msgr2=0x7fd648106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.963+0000 7fd64dd7e640 1 --2- 172.21.15.60:0/2211659748 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6481046f0 0x7fd648106ae0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009900 tx=0x7fd63c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:43.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.963+0000 7fd64dd7e640 1 -- 172.21.15.60:0/2211659748 shutdown_connections 2024-03-04T12:25:43.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.963+0000 7fd64dd7e640 1 --2- 172.21.15.60:0/2211659748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd648107020 0x7fd648109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.963+0000 7fd64dd7e640 1 --2- 172.21.15.60:0/2211659748 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd6481046f0 0x7fd648106ae0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.963+0000 7fd64dd7e640 1 --2- 172.21.15.60:0/2211659748 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd648101dc0 0x7fd6481041b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.963+0000 7fd64dd7e640 1 -- 172.21.15.60:0/2211659748 >> 172.21.15.60:0/2211659748 conn(0x7fd6480fb790 msgr2=0x7fd6480fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:43.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.963+0000 7fd64dd7e640 1 -- 172.21.15.60:0/2211659748 shutdown_connections 2024-03-04T12:25:43.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.963+0000 7fd64dd7e640 1 -- 172.21.15.60:0/2211659748 wait complete. 2024-03-04T12:25:43.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.963+0000 7fd64dd7e640 1 Processor -- start 2024-03-04T12:25:43.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.963+0000 7fd64dd7e640 1 -- start start 2024-03-04T12:25:43.973 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd64dd7e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd648101dc0 0x7fd648101730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd64dd7e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6481046f0 0x7fd6480ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd64dd7e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd648107020 0x7fd6481002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd64dd7e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64810cfd0 con 0x7fd648107020 2024-03-04T12:25:43.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd64dd7e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64810ce50 con 0x7fd648101dc0 2024-03-04T12:25:43.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd64dd7e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64810d150 con 0x7fd6481046f0 2024-03-04T12:25:43.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd647fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd648107020 0x7fd6481002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd647fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd648107020 0x7fd6481002b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55976/0 (socket says 172.21.15.60:55976) 2024-03-04T12:25:43.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd647fff640 1 -- 172.21.15.60:0/1243206269 learned_addr learned my addr 172.21.15.60:0/1243206269 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:43.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd6477fe640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd648101dc0 0x7fd648101730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd646ffd640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6481046f0 0x7fd6480ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.974 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd647fff640 1 -- 172.21.15.60:0/1243206269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6481046f0 msgr2=0x7fd6480ffd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd647fff640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6481046f0 0x7fd6480ffd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd647fff640 1 -- 172.21.15.60:0/1243206269 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd648101dc0 msgr2=0x7fd648101730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:43.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd647fff640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd648101dc0 0x7fd648101730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:43.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd647fff640 1 -- 172.21.15.60:0/1243206269 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd648100b40 con 0x7fd648107020 2024-03-04T12:25:43.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd647fff640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd648107020 0x7fd6481002b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd64000cce0 tx=0x7fd64000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:43.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd644ff9640 1 -- 172.21.15.60:0/1243206269 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64000f450 con 0x7fd648107020 2024-03-04T12:25:43.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd644ff9640 1 -- 172.21.15.60:0/1243206269 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd64000fad0 con 0x7fd648107020 2024-03-04T12:25:43.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.964+0000 7fd64dd7e640 1 -- 172.21.15.60:0/1243206269 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6481a1760 con 0x7fd648107020 2024-03-04T12:25:43.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.965+0000 7fd644ff9640 1 -- 172.21.15.60:0/1243206269 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd640018c10 con 0x7fd648107020 2024-03-04T12:25:43.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.965+0000 7fd64dd7e640 1 -- 172.21.15.60:0/1243206269 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6481a1c80 con 0x7fd648107020 2024-03-04T12:25:43.975 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.965+0000 7fd64dd7e640 1 -- 172.21.15.60:0/1243206269 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd648068600 con 0x7fd648107020 2024-03-04T12:25:43.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.967+0000 7fd644ff9640 1 -- 172.21.15.60:0/1243206269 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fd64000f5f0 con 0x7fd648107020 2024-03-04T12:25:43.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.967+0000 7fd644ff9640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd6300a1e90 0x7fd6300a4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:43.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.967+0000 7fd6477fe640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd6300a1e90 0x7fd6300a4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:43.976 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.967+0000 7fd644ff9640 1 -- 172.21.15.60:0/1243206269 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fd6400cea10 con 0x7fd648107020 2024-03-04T12:25:43.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.968+0000 7fd6477fe640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd6300a1e90 0x7fd6300a4350 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd63800bbe0 tx=0x7fd63800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:43.979 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:43.972+0000 7fd644ff9640 1 -- 172.21.15.60:0/1243206269 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd640014030 con 0x7fd648107020 2024-03-04T12:25:44.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.157+0000 7fd64dd7e640 1 -- 172.21.15.60:0/1243206269 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd64802ceb0 con 0x7fd648107020 2024-03-04T12:25:44.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.158+0000 7fd644ff9640 1 -- 172.21.15.60:0/1243206269 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd64009b8f0 con 0x7fd648107020 2024-03-04T12:25:44.165 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-03-04T12:25:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.160+0000 7fd64dd7e640 1 -- 172.21.15.60:0/1243206269 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd6300a1e90 msgr2=0x7fd6300a4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.160+0000 7fd64dd7e640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd6300a1e90 0x7fd6300a4350 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd63800bbe0 tx=0x7fd63800b390 comp rx=0 tx=0).stop 2024-03-04T12:25:44.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.160+0000 7fd64dd7e640 1 -- 172.21.15.60:0/1243206269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd648107020 msgr2=0x7fd6481002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.160+0000 7fd64dd7e640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd648107020 0x7fd6481002b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd64000cce0 tx=0x7fd64000a620 comp rx=0 tx=0).stop 2024-03-04T12:25:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.160+0000 7fd64dd7e640 1 -- 172.21.15.60:0/1243206269 shutdown_connections 2024-03-04T12:25:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.160+0000 7fd64dd7e640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fd6300a1e90 0x7fd6300a4350 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.160+0000 7fd64dd7e640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd648107020 0x7fd6481002b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.160+0000 7fd64dd7e640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6481046f0 0x7fd6480ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.160+0000 7fd64dd7e640 1 --2- 172.21.15.60:0/1243206269 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd648101dc0 0x7fd648101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.160+0000 7fd64dd7e640 1 -- 172.21.15.60:0/1243206269 >> 172.21.15.60:0/1243206269 conn(0x7fd6480fb790 msgr2=0x7fd648102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.160+0000 7fd64dd7e640 1 -- 172.21.15.60:0/1243206269 shutdown_connections 2024-03-04T12:25:44.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.160+0000 7fd64dd7e640 1 -- 172.21.15.60:0/1243206269 wait complete. 2024-03-04T12:25:44.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:44 smithi060 ceph-mon[32745]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-04T12:25:44.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:44 smithi060 ceph-mon[32745]: from='client.15627 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:25:44.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:44 smithi060 ceph-mon[32745]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:25:44.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:44 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3425028461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:44.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:44 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2042432660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:44.177 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:44 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/117315029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:44.178 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T12:25:44.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.276+0000 7f2430d0a640 1 -- 172.21.15.60:0/3924098716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f242c100250 msgr2=0x7f242c100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.276+0000 7f2430d0a640 1 --2- 172.21.15.60:0/3924098716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f242c100250 0x7f242c100670 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2424009ca0 tx=0x7f242402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.276+0000 7f2430d0a640 1 -- 172.21.15.60:0/3924098716 shutdown_connections 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.276+0000 7f2430d0a640 1 --2- 172.21.15.60:0/3924098716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f242c101e00 0x7f242c108690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.276+0000 7f2430d0a640 1 --2- 172.21.15.60:0/3924098716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f242c101440 0x7f242c1018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.276+0000 7f2430d0a640 1 --2- 172.21.15.60:0/3924098716 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f242c100250 0x7f242c100670 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.276+0000 7f2430d0a640 1 -- 172.21.15.60:0/3924098716 >> 172.21.15.60:0/3924098716 conn(0x7f242c0fb7e0 msgr2=0x7f242c0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.276+0000 7f2430d0a640 1 -- 172.21.15.60:0/3924098716 shutdown_connections 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.276+0000 7f2430d0a640 1 -- 172.21.15.60:0/3924098716 wait complete. 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2430d0a640 1 Processor -- start 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2430d0a640 1 -- start start 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2430d0a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f242c100250 0x7f242c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2430d0a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f242c101440 0x7f242c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:44.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2430d0a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f242c101e00 0x7f242c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2430d0a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c10cfe0 con 0x7f242c101440 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2430d0a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c10ce60 con 0x7f242c101e00 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2430d0a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c10d160 con 0x7f242c100250 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2429d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f242c101440 0x7f242c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2429d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f242c101440 0x7f242c197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55992/0 (socket says 172.21.15.60:55992) 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2429d9b640 1 -- 172.21.15.60:0/2827457817 learned_addr learned my addr 172.21.15.60:0/2827457817 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f242ad9d640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f242c101e00 0x7f242c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f242a59c640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f242c100250 0x7f242c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2429d9b640 1 -- 172.21.15.60:0/2827457817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f242c100250 msgr2=0x7f242c1975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2429d9b640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f242c100250 0x7f242c1975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2429d9b640 1 -- 172.21.15.60:0/2827457817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f242c101e00 msgr2=0x7f242c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2429d9b640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f242c101e00 0x7f242c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.277+0000 7f2429d9b640 1 -- 172.21.15.60:0/2827457817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c19f200 con 0x7f242c101440 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.278+0000 7f2429d9b640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f242c101440 0x7f242c197b20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f242000d950 tx=0x7f242000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:44.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.278+0000 7f241b7fe640 1 -- 172.21.15.60:0/2827457817 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420017030 con 0x7f242c101440 2024-03-04T12:25:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.278+0000 7f2430d0a640 1 -- 172.21.15.60:0/2827457817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242c19f4f0 con 0x7f242c101440 2024-03-04T12:25:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.278+0000 7f241b7fe640 1 -- 172.21.15.60:0/2827457817 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2420004020 con 0x7f242c101440 2024-03-04T12:25:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.278+0000 7f241b7fe640 1 -- 172.21.15.60:0/2827457817 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24200047a0 con 0x7f242c101440 2024-03-04T12:25:44.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.278+0000 7f2430d0a640 1 -- 172.21.15.60:0/2827457817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c19fa40 con 0x7f242c101440 2024-03-04T12:25:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.279+0000 7f2430d0a640 1 -- 172.21.15.60:0/2827457817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242c107510 con 0x7f242c101440 2024-03-04T12:25:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.280+0000 7f241b7fe640 1 -- 172.21.15.60:0/2827457817 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f24200041d0 con 0x7f242c101440 2024-03-04T12:25:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.280+0000 7f241b7fe640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f24140a1e10 0x7f24140a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.280+0000 7f241b7fe640 1 -- 172.21.15.60:0/2827457817 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f24200cdd50 con 0x7f242c101440 2024-03-04T12:25:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.280+0000 7f242a59c640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f24140a1e10 0x7f24140a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:44.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.281+0000 7f242a59c640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f24140a1e10 0x7f24140a42d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f242402c700 tx=0x7f2424009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:44.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.285+0000 7f241b7fe640 1 -- 172.21.15.60:0/2827457817 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f242009ac90 con 0x7f242c101440 2024-03-04T12:25:44.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:44 smithi086 ceph-mon[36851]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-04T12:25:44.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:44 smithi086 ceph-mon[36851]: from='client.15627 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:25:44.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:44 smithi086 ceph-mon[36851]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:25:44.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:44 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3425028461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:44.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:44 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2042432660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:44.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:44 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/117315029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:44.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:44 smithi063 ceph-mon[36717]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-04T12:25:44.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:44 smithi063 ceph-mon[36717]: from='client.15627 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:25:44.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:44 smithi063 ceph-mon[36717]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T12:25:44.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:44 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3425028461' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:44.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:44 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2042432660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:25:44.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:44 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/117315029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:25:44.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.466+0000 7f2430d0a640 1 -- 172.21.15.60:0/2827457817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f242c02d100 con 0x7f242c101440 2024-03-04T12:25:44.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.466+0000 7f241b7fe640 1 -- 172.21.15.60:0/2827457817 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f242009a590 con 0x7f242c101440 2024-03-04T12:25:44.473 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:25:44.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.468+0000 7f2430d0a640 1 -- 172.21.15.60:0/2827457817 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f24140a1e10 msgr2=0x7f24140a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.468+0000 7f2430d0a640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f24140a1e10 0x7f24140a42d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f242402c700 tx=0x7f2424009360 comp rx=0 tx=0).stop 2024-03-04T12:25:44.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.468+0000 7f2430d0a640 1 -- 172.21.15.60:0/2827457817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f242c101440 msgr2=0x7f242c197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.468+0000 7f2430d0a640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f242c101440 0x7f242c197b20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f242000d950 tx=0x7f242000de10 comp rx=0 tx=0).stop 2024-03-04T12:25:44.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.468+0000 7f2430d0a640 1 -- 172.21.15.60:0/2827457817 shutdown_connections 2024-03-04T12:25:44.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.468+0000 7f2430d0a640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f24140a1e10 0x7f24140a42d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.468+0000 7f2430d0a640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f242c101e00 0x7f242c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.468+0000 7f2430d0a640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f242c101440 0x7f242c197b20 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.468+0000 7f2430d0a640 1 --2- 172.21.15.60:0/2827457817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f242c100250 0x7f242c1975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.468+0000 7f2430d0a640 1 -- 172.21.15.60:0/2827457817 >> 172.21.15.60:0/2827457817 conn(0x7f242c0fb7e0 msgr2=0x7f242c1046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:44.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.468+0000 7f2430d0a640 1 -- 172.21.15.60:0/2827457817 shutdown_connections 2024-03-04T12:25:44.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.468+0000 7f2430d0a640 1 -- 172.21.15.60:0/2827457817 wait complete. 2024-03-04T12:25:44.486 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:25:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.582+0000 7f66c1bc4640 1 -- 172.21.15.60:0/1320394948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66bc102550 msgr2=0x7f66bc1029d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.582+0000 7f66c1bc4640 1 --2- 172.21.15.60:0/1320394948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66bc102550 0x7f66bc1029d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009900 tx=0x7f66b002bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.582+0000 7f66c1bc4640 1 -- 172.21.15.60:0/1320394948 shutdown_connections 2024-03-04T12:25:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.582+0000 7f66c1bc4640 1 --2- 172.21.15.60:0/1320394948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66bc102f10 0x7f66bc1097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.582+0000 7f66c1bc4640 1 --2- 172.21.15.60:0/1320394948 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66bc102550 0x7f66bc1029d0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.582+0000 7f66c1bc4640 1 --2- 172.21.15.60:0/1320394948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66bc101360 0x7f66bc101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.582+0000 7f66c1bc4640 1 -- 172.21.15.60:0/1320394948 >> 172.21.15.60:0/1320394948 conn(0x7f66bc0fc910 msgr2=0x7f66bc0fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.582+0000 7f66c1bc4640 1 -- 172.21.15.60:0/1320394948 shutdown_connections 2024-03-04T12:25:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.582+0000 7f66c1bc4640 1 -- 172.21.15.60:0/1320394948 wait complete. 2024-03-04T12:25:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66c1bc4640 1 Processor -- start 2024-03-04T12:25:44.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66c1bc4640 1 -- start start 2024-03-04T12:25:44.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66c1bc4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66bc101360 0x7f66bc197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66c1bc4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66bc102550 0x7f66bc197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66c1bc4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66bc102f10 0x7f66bc19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66c1bc4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc10cff0 con 0x7f66bc102550 2024-03-04T12:25:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66c1bc4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc10ce70 con 0x7f66bc101360 2024-03-04T12:25:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66c1bc4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc10d170 con 0x7f66bc102f10 2024-03-04T12:25:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66baffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66bc102550 0x7f66bc197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66baffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66bc102550 0x7f66bc197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56022/0 (socket says 172.21.15.60:56022) 2024-03-04T12:25:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66baffd640 1 -- 172.21.15.60:0/3030684530 learned_addr learned my addr 172.21.15.60:0/3030684530 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66bb7fe640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66bc101360 0x7f66bc197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66bbfff640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66bc102f10 0x7f66bc19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66baffd640 1 -- 172.21.15.60:0/3030684530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66bc102f10 msgr2=0x7f66bc19ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66baffd640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66bc102f10 0x7f66bc19ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66baffd640 1 -- 172.21.15.60:0/3030684530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66bc101360 msgr2=0x7f66bc197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66baffd640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66bc101360 0x7f66bc197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66baffd640 1 -- 172.21.15.60:0/3030684530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc19f270 con 0x7f66bc102550 2024-03-04T12:25:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.583+0000 7f66bbfff640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66bc102f10 0x7f66bc19ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.584+0000 7f66baffd640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66bc102550 0x7f66bc197b90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009230 tx=0x7f66b0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.584+0000 7f66b8ff9640 1 -- 172.21.15.60:0/3030684530 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0041070 con 0x7f66bc102550 2024-03-04T12:25:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.584+0000 7f66b8ff9640 1 -- 172.21.15.60:0/3030684530 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66b00054e0 con 0x7f66bc102550 2024-03-04T12:25:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.584+0000 7f66c1bc4640 1 -- 172.21.15.60:0/3030684530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66bc19f500 con 0x7f66bc102550 2024-03-04T12:25:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.584+0000 7f66b8ff9640 1 -- 172.21.15.60:0/3030684530 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0034440 con 0x7f66bc102550 2024-03-04T12:25:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.584+0000 7f66c1bc4640 1 -- 172.21.15.60:0/3030684530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66bc19f9f0 con 0x7f66bc102550 2024-03-04T12:25:44.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.585+0000 7f66c1bc4640 1 -- 172.21.15.60:0/3030684530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66bc108680 con 0x7f66bc102550 2024-03-04T12:25:44.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.586+0000 7f66b8ff9640 1 -- 172.21.15.60:0/3030684530 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f66b0044070 con 0x7f66bc102550 2024-03-04T12:25:44.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.586+0000 7f66b8ff9640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f66a40a1e10 0x7f66a40a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:44.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.587+0000 7f66bb7fe640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f66a40a1e10 0x7f66a40a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:44.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.587+0000 7f66b8ff9640 1 -- 172.21.15.60:0/3030684530 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f66b00f0300 con 0x7f66bc102550 2024-03-04T12:25:44.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.587+0000 7f66bb7fe640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f66a40a1e10 0x7f66a40a42d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f66bc1023b0 tx=0x7f66ac00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:44.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.590+0000 7f66b8ff9640 1 -- 172.21.15.60:0/3030684530 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f66b00bd2f0 con 0x7f66bc102550 2024-03-04T12:25:44.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.810+0000 7f66c1bc4640 1 -- 172.21.15.60:0/3030684530 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66bc02d070 con 0x7f66bc102550 2024-03-04T12:25:44.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.811+0000 7f66b8ff9640 1 -- 172.21.15.60:0/3030684530 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11373 (secure 0 0 0) 0x7f66b00bcbf0 con 0x7f66bc102550 2024-03-04T12:25:44.819 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:44.819 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:25:37.524434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:25:44.819 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 29 2024-03-04T12:25:44.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.813+0000 7f66c1bc4640 1 -- 172.21.15.60:0/3030684530 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f66a40a1e10 msgr2=0x7f66a40a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.813+0000 7f66c1bc4640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f66a40a1e10 0x7f66a40a42d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f66bc1023b0 tx=0x7f66ac00b390 comp rx=0 tx=0).stop 2024-03-04T12:25:44.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.813+0000 7f66c1bc4640 1 -- 172.21.15.60:0/3030684530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66bc102550 msgr2=0x7f66bc197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.813+0000 7f66c1bc4640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66bc102550 0x7f66bc197b90 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009230 tx=0x7f66b0005dd0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.813+0000 7f66c1bc4640 1 -- 172.21.15.60:0/3030684530 shutdown_connections 2024-03-04T12:25:44.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.813+0000 7f66c1bc4640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f66a40a1e10 0x7f66a40a42d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.813+0000 7f66c1bc4640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66bc102f10 0x7f66bc19ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.813+0000 7f66c1bc4640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f66bc102550 0x7f66bc197b90 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.813+0000 7f66c1bc4640 1 --2- 172.21.15.60:0/3030684530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66bc101360 0x7f66bc197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.813+0000 7f66c1bc4640 1 -- 172.21.15.60:0/3030684530 >> 172.21.15.60:0/3030684530 conn(0x7f66bc0fc910 msgr2=0x7f66bc105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:44.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.813+0000 7f66c1bc4640 1 -- 172.21.15.60:0/3030684530 shutdown_connections 2024-03-04T12:25:44.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.814+0000 7f66c1bc4640 1 -- 172.21.15.60:0/3030684530 wait complete. 2024-03-04T12:25:44.832 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:25:44.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.931+0000 7fa5128c6640 1 -- 172.21.15.60:0/693044882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa50c100260 msgr2=0x7fa50c100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.931+0000 7fa5128c6640 1 --2- 172.21.15.60:0/693044882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa50c100260 0x7fa50c100680 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009900 tx=0x7fa4fc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:44.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.931+0000 7fa5128c6640 1 -- 172.21.15.60:0/693044882 shutdown_connections 2024-03-04T12:25:44.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.931+0000 7fa5128c6640 1 --2- 172.21.15.60:0/693044882 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa50c101e10 0x7fa50c1086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.931+0000 7fa5128c6640 1 --2- 172.21.15.60:0/693044882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c101450 0x7fa50c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.931+0000 7fa5128c6640 1 --2- 172.21.15.60:0/693044882 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa50c100260 0x7fa50c100680 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.931+0000 7fa5128c6640 1 -- 172.21.15.60:0/693044882 >> 172.21.15.60:0/693044882 conn(0x7fa50c0fb7d0 msgr2=0x7fa50c0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:44.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.931+0000 7fa5128c6640 1 -- 172.21.15.60:0/693044882 shutdown_connections 2024-03-04T12:25:44.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.931+0000 7fa5128c6640 1 -- 172.21.15.60:0/693044882 wait complete. 2024-03-04T12:25:44.940 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.931+0000 7fa5128c6640 1 Processor -- start 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.931+0000 7fa5128c6640 1 -- start start 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa5128c6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c100260 0x7fa50c1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa5128c6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa50c101450 0x7fa50c199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa5128c6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa50c101e10 0x7fa50c1a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa5128c6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c10bef0 con 0x7fa50c101e10 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa5128c6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c10bd70 con 0x7fa50c101450 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa5128c6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c10c070 con 0x7fa50c100260 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa510e63640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa50c101e10 0x7fa50c1a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa50bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c100260 0x7fa50c1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa50b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa50c101450 0x7fa50c199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa510e63640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa50c101e10 0x7fa50c1a0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56044/0 (socket says 172.21.15.60:56044) 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa510e63640 1 -- 172.21.15.60:0/54006971 learned_addr learned my addr 172.21.15.60:0/54006971 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa510e63640 1 -- 172.21.15.60:0/54006971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c100260 msgr2=0x7fa50c1998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.941 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa510e63640 1 --2- 172.21.15.60:0/54006971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c100260 0x7fa50c1998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa510e63640 1 -- 172.21.15.60:0/54006971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa50c101450 msgr2=0x7fa50c199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:44.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa510e63640 1 --2- 172.21.15.60:0/54006971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa50c101450 0x7fa50c199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:44.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa510e63640 1 -- 172.21.15.60:0/54006971 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c1a14c0 con 0x7fa50c101e10 2024-03-04T12:25:44.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.932+0000 7fa510e63640 1 --2- 172.21.15.60:0/54006971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa50c101e10 0x7fa50c1a0e50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa50400eb10 tx=0x7fa50400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:44.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.933+0000 7fa5097fa640 1 -- 172.21.15.60:0/54006971 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504004b20 con 0x7fa50c101e10 2024-03-04T12:25:44.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.933+0000 7fa5097fa640 1 -- 172.21.15.60:0/54006971 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa504004cc0 con 0x7fa50c101e10 2024-03-04T12:25:44.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.933+0000 7fa5128c6640 1 -- 172.21.15.60:0/54006971 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50c1a17b0 con 0x7fa50c101e10 2024-03-04T12:25:44.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.933+0000 7fa5097fa640 1 -- 172.21.15.60:0/54006971 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504010760 con 0x7fa50c101e10 2024-03-04T12:25:44.942 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.933+0000 7fa5128c6640 1 -- 172.21.15.60:0/54006971 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50c1a1cb0 con 0x7fa50c101e10 2024-03-04T12:25:44.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.934+0000 7fa5128c6640 1 -- 172.21.15.60:0/54006971 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa50c1a1f70 con 0x7fa50c101e10 2024-03-04T12:25:44.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.935+0000 7fa5097fa640 1 -- 172.21.15.60:0/54006971 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fa504005020 con 0x7fa50c101e10 2024-03-04T12:25:44.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.935+0000 7fa5097fa640 1 --2- 172.21.15.60:0/54006971 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa4f40a1e10 0x7fa4f40a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:44.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.935+0000 7fa50bfff640 1 --2- 172.21.15.60:0/54006971 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa4f40a1e10 0x7fa4f40a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:44.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.935+0000 7fa5097fa640 1 -- 172.21.15.60:0/54006971 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fa5040ce880 con 0x7fa50c101e10 2024-03-04T12:25:44.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.936+0000 7fa50bfff640 1 --2- 172.21.15.60:0/54006971 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa4f40a1e10 0x7fa4f40a42d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc000c00 tx=0x7fa4fc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:44.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:44.939+0000 7fa5097fa640 1 -- 172.21.15.60:0/54006971 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa50409b7c0 con 0x7fa50c101e10 2024-03-04T12:25:45.135 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.128+0000 7fa5128c6640 1 -- 172.21.15.60:0/54006971 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa50c1a2140 con 0x7fa50c101e10 2024-03-04T12:25:45.136 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.129+0000 7fa5097fa640 1 -- 172.21.15.60:0/54006971 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24562 (secure 0 0 0) 0x7fa50409b0c0 con 0x7fa50c101e10 2024-03-04T12:25:45.137 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:25:45.137 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":93,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:25:06.929248+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:25:45.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.132+0000 7fa5128c6640 1 -- 172.21.15.60:0/54006971 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa4f40a1e10 msgr2=0x7fa4f40a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.132+0000 7fa5128c6640 1 --2- 172.21.15.60:0/54006971 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa4f40a1e10 0x7fa4f40a42d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc000c00 tx=0x7fa4fc005fd0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.132+0000 7fa5128c6640 1 -- 172.21.15.60:0/54006971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa50c101e10 msgr2=0x7fa50c1a0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.132+0000 7fa5128c6640 1 --2- 172.21.15.60:0/54006971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa50c101e10 0x7fa50c1a0e50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa50400eb10 tx=0x7fa50400c2d0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.132+0000 7fa5128c6640 1 -- 172.21.15.60:0/54006971 shutdown_connections 2024-03-04T12:25:45.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.132+0000 7fa5128c6640 1 --2- 172.21.15.60:0/54006971 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa4f40a1e10 0x7fa4f40a42d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.132+0000 7fa5128c6640 1 --2- 172.21.15.60:0/54006971 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa50c101e10 0x7fa50c1a0e50 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.132+0000 7fa5128c6640 1 --2- 172.21.15.60:0/54006971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa50c101450 0x7fa50c199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.132+0000 7fa5128c6640 1 --2- 172.21.15.60:0/54006971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa50c100260 0x7fa50c1998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.132+0000 7fa5128c6640 1 -- 172.21.15.60:0/54006971 >> 172.21.15.60:0/54006971 conn(0x7fa50c0fb7d0 msgr2=0x7fa50c0fd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.132+0000 7fa5128c6640 1 -- 172.21.15.60:0/54006971 shutdown_connections 2024-03-04T12:25:45.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.132+0000 7fa5128c6640 1 -- 172.21.15.60:0/54006971 wait complete. 2024-03-04T12:25:45.149 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:45 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1243206269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:45.149 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:45 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2827457817' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:45.149 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:45 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3030684530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:45.150 DEBUG:tasks.fwd_scrub:Yielding 2024-03-04T12:25:45.152 INFO:teuthology.run_tasks:Running task exec... 2024-03-04T12:25:45.162 INFO:teuthology.task.exec:Executing custom commands... 2024-03-04T12:25:45.162 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi060.front.sepia.ceph.com 2024-03-04T12:25:45.162 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-04T12:25:45.162 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-04T12:25:45.162 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T12:25:45.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:25:45.163 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T12:25:45.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.259+0000 7fa722ed1640 1 -- 172.21.15.60:0/2934527807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa71c101480 msgr2=0x7fa71c101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.259+0000 7fa722ed1640 1 --2- 172.21.15.60:0/2934527807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa71c101480 0x7fa71c101900 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa708009900 tx=0x7fa70802bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:45.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.259+0000 7fa722ed1640 1 -- 172.21.15.60:0/2934527807 shutdown_connections 2024-03-04T12:25:45.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.259+0000 7fa722ed1640 1 --2- 172.21.15.60:0/2934527807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa71c101e40 0x7fa71c1086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.259+0000 7fa722ed1640 1 --2- 172.21.15.60:0/2934527807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa71c101480 0x7fa71c101900 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.259+0000 7fa722ed1640 1 --2- 172.21.15.60:0/2934527807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa71c100290 0x7fa71c1006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.259+0000 7fa722ed1640 1 -- 172.21.15.60:0/2934527807 >> 172.21.15.60:0/2934527807 conn(0x7fa71c0fb840 msgr2=0x7fa71c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.259+0000 7fa722ed1640 1 -- 172.21.15.60:0/2934527807 shutdown_connections 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.259+0000 7fa722ed1640 1 -- 172.21.15.60:0/2934527807 wait complete. 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa722ed1640 1 Processor -- start 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa722ed1640 1 -- start start 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa722ed1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa71c100290 0x7fa71c197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa722ed1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa71c101480 0x7fa71c197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa722ed1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa71c101e40 0x7fa71c19eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa722ed1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71c10bf60 con 0x7fa71c101480 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa722ed1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71c10bde0 con 0x7fa71c100290 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa722ed1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71c10c0e0 con 0x7fa71c101e40 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa71bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa71c101480 0x7fa71c197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa71bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa71c101480 0x7fa71c197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56076/0 (socket says 172.21.15.60:56076) 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa71bfff640 1 -- 172.21.15.60:0/1786069285 learned_addr learned my addr 172.21.15.60:0/1786069285 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa720c6d640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa71c100290 0x7fa71c197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa72146e640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa71c101e40 0x7fa71c19eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa71bfff640 1 -- 172.21.15.60:0/1786069285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa71c101e40 msgr2=0x7fa71c19eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa71bfff640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa71c101e40 0x7fa71c19eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa71bfff640 1 -- 172.21.15.60:0/1786069285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa71c100290 msgr2=0x7fa71c197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa71bfff640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa71c100290 0x7fa71c197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.260+0000 7fa71bfff640 1 -- 172.21.15.60:0/1786069285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71c19f310 con 0x7fa71c101480 2024-03-04T12:25:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.261+0000 7fa71bfff640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa71c101480 0x7fa71c197c80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa708009230 tx=0x7fa708005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.261+0000 7fa719ffb640 1 -- 172.21.15.60:0/1786069285 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708041070 con 0x7fa71c101480 2024-03-04T12:25:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.261+0000 7fa722ed1640 1 -- 172.21.15.60:0/1786069285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa71c19f5a0 con 0x7fa71c101480 2024-03-04T12:25:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.261+0000 7fa719ffb640 1 -- 172.21.15.60:0/1786069285 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7080054e0 con 0x7fa71c101480 2024-03-04T12:25:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.261+0000 7fa719ffb640 1 -- 172.21.15.60:0/1786069285 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708034440 con 0x7fa71c101480 2024-03-04T12:25:45.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.261+0000 7fa722ed1640 1 -- 172.21.15.60:0/1786069285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa71c19fae0 con 0x7fa71c101480 2024-03-04T12:25:45.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.262+0000 7fa722ed1640 1 -- 172.21.15.60:0/1786069285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa71c19fd40 con 0x7fa71c101480 2024-03-04T12:25:45.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.263+0000 7f8f4483d640 1 -- 172.21.15.60:0/747474589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40101460 msgr2=0x7f8f401018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.263+0000 7f8f4483d640 1 --2- 172.21.15.60:0/747474589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40101460 0x7f8f401018e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009900 tx=0x7f8f2802bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:45.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.263+0000 7f8f4483d640 1 -- 172.21.15.60:0/747474589 shutdown_connections 2024-03-04T12:25:45.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.263+0000 7f8f4483d640 1 --2- 172.21.15.60:0/747474589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40101e20 0x7f8f401086b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.263+0000 7f8f4483d640 1 --2- 172.21.15.60:0/747474589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40101460 0x7f8f401018e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.263+0000 7f8f4483d640 1 --2- 172.21.15.60:0/747474589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f40100270 0x7f8f40100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.263+0000 7f8f4483d640 1 -- 172.21.15.60:0/747474589 >> 172.21.15.60:0/747474589 conn(0x7f8f400fb800 msgr2=0x7f8f400fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.263+0000 7f8f4483d640 1 -- 172.21.15.60:0/747474589 shutdown_connections 2024-03-04T12:25:45.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.263+0000 7f8f4483d640 1 -- 172.21.15.60:0/747474589 wait complete. 2024-03-04T12:25:45.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.263+0000 7fa719ffb640 1 -- 172.21.15.60:0/1786069285 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7fa708044070 con 0x7fa71c101480 2024-03-04T12:25:45.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f4483d640 1 Processor -- start 2024-03-04T12:25:45.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f4483d640 1 -- start start 2024-03-04T12:25:45.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7fa719ffb640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa70c0a1e00 0x7fa70c0a42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f4483d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40100270 0x7f8f40197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f4483d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40101460 0x7f8f40197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f4483d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f40101e20 0x7f8f4019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f4483d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4010bf50 con 0x7f8f40101460 2024-03-04T12:25:45.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f4483d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4010bdd0 con 0x7f8f40100270 2024-03-04T12:25:45.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f4483d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4010c0d0 con 0x7f8f40101e20 2024-03-04T12:25:45.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f37fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40101460 0x7f8f40197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f3e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40100270 0x7f8f40197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f3e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40100270 0x7f8f40197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:45784/0 (socket says 172.21.15.60:45784) 2024-03-04T12:25:45.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f3e59c640 1 -- 172.21.15.60:0/2309555376 learned_addr learned my addr 172.21.15.60:0/2309555376 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:45.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7fa719ffb640 1 -- 172.21.15.60:0/1786069285 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7fa7080f0280 con 0x7fa71c101480 2024-03-04T12:25:45.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7fa720c6d640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa70c0a1e00 0x7fa70c0a42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f40101e20 0x7f8f4019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f37fff640 1 -- 172.21.15.60:0/2309555376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f40101e20 msgr2=0x7f8f4019ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f37fff640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f40101e20 0x7f8f4019ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f37fff640 1 -- 172.21.15.60:0/2309555376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40100270 msgr2=0x7f8f40197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f37fff640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40100270 0x7f8f40197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f37fff640 1 -- 172.21.15.60:0/2309555376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4019f2a0 con 0x7f8f40101460 2024-03-04T12:25:45.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.264+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40100270 0x7f8f40197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:25:45.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.265+0000 7fa720c6d640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa70c0a1e00 0x7fa70c0a42c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa71c1012e0 tx=0x7fa71000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.265+0000 7f8f37fff640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40101460 0x7f8f40197bc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8f28036040 tx=0x7f8f2802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.268+0000 7f8f377fe640 1 -- 172.21.15.60:0/2309555376 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28043020 con 0x7f8f40101460 2024-03-04T12:25:45.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.268+0000 7f8f377fe640 1 -- 172.21.15.60:0/2309555376 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f280054e0 con 0x7f8f40101460 2024-03-04T12:25:45.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.268+0000 7f8f4483d640 1 -- 172.21.15.60:0/2309555376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4019f530 con 0x7f8f40101460 2024-03-04T12:25:45.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.268+0000 7f8f377fe640 1 -- 172.21.15.60:0/2309555376 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28003c30 con 0x7f8f40101460 2024-03-04T12:25:45.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.268+0000 7f8f4483d640 1 -- 172.21.15.60:0/2309555376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4019fa50 con 0x7f8f40101460 2024-03-04T12:25:45.277 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.268+0000 7fa719ffb640 1 -- 172.21.15.60:0/1786069285 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa7080bcc80 con 0x7fa71c101480 2024-03-04T12:25:45.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.270+0000 7f8f377fe640 1 -- 172.21.15.60:0/2309555376 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7f8f28046020 con 0x7f8f40101460 2024-03-04T12:25:45.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.270+0000 7f8f377fe640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8f2c0a1e10 0x7f8f2c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.270+0000 7f8f377fe640 1 -- 172.21.15.60:0/2309555376 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f8f280f0210 con 0x7f8f40101460 2024-03-04T12:25:45.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.270+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8f2c0a1e10 0x7f8f2c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.271+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8f2c0a1e10 0x7f8f2c0a42d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8f300058c0 tx=0x7f8f30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.278 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.271+0000 7f8f4741e740 1 Processor -- start 2024-03-04T12:25:45.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.271+0000 7f8f4741e740 1 -- start start 2024-03-04T12:25:45.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.271+0000 7f8f4741e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bbf40 0x55f62c5bc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.271+0000 7f8f4741e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 0x55f62c5c1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.271+0000 7f8f4741e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5c1aa0 0x55f62c5c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f4741e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f62c520b40 con 0x55f62c5bc930 2024-03-04T12:25:45.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f4741e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f62c4c9d80 con 0x55f62c5bbf40 2024-03-04T12:25:45.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f4741e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f62c51cbe0 con 0x55f62c5c1aa0 2024-03-04T12:25:45.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f37fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 0x55f62c5c1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f37fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 0x55f62c5c1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56110/0 (socket says 172.21.15.60:56110) 2024-03-04T12:25:45.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f37fff640 1 -- 172.21.15.60:0/3693471613 learned_addr learned my addr 172.21.15.60:0/3693471613 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:45.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/3693471613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bbf40 0x55f62c5bc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/3693471613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5c1aa0 0x55f62c5c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f37fff640 1 -- 172.21.15.60:0/3693471613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5c1aa0 msgr2=0x55f62c5c3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f37fff640 1 --2- 172.21.15.60:0/3693471613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5c1aa0 0x55f62c5c3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f37fff640 1 -- 172.21.15.60:0/3693471613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bbf40 msgr2=0x55f62c5bc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f37fff640 1 --2- 172.21.15.60:0/3693471613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bbf40 0x55f62c5bc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f37fff640 1 -- 172.21.15.60:0/3693471613 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f62c2fd7e0 con 0x55f62c5bc930 2024-03-04T12:25:45.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f37fff640 1 --2- 172.21.15.60:0/3693471613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 0x55f62c5c1430 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8f28005ab0 tx=0x7f8f28003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a32deadfac4aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f34ff9640 1 -- 172.21.15.60:0/3693471613 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280bd200 con 0x55f62c5bc930 2024-03-04T12:25:45.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f34ff9640 1 -- 172.21.15.60:0/3693471613 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f280bcb00 con 0x55f62c5bc930 2024-03-04T12:25:45.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.272+0000 7f8f34ff9640 1 -- 172.21.15.60:0/3693471613 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280eda20 con 0x55f62c5bc930 2024-03-04T12:25:45.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.273+0000 7f8f4741e740 1 -- 172.21.15.60:0/3693471613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 msgr2=0x55f62c5c1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.273+0000 7f8f4741e740 1 --2- 172.21.15.60:0/3693471613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 0x55f62c5c1430 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f8f28005ab0 tx=0x7f8f28003760 comp rx=0 tx=0).stop 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.273+0000 7f8f4741e740 1 -- 172.21.15.60:0/3693471613 shutdown_connections 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.273+0000 7f8f4741e740 1 --2- 172.21.15.60:0/3693471613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5c1aa0 0x55f62c5c3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.273+0000 7f8f4741e740 1 --2- 172.21.15.60:0/3693471613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 0x55f62c5c1430 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.273+0000 7f8f4741e740 1 --2- 172.21.15.60:0/3693471613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bbf40 0x55f62c5bc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.273+0000 7f8f4741e740 1 -- 172.21.15.60:0/3693471613 >> 172.21.15.60:0/3693471613 conn(0x55f62c5b74b0 msgr2=0x55f62c5b78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.273+0000 7f8f4741e740 1 -- 172.21.15.60:0/3693471613 shutdown_connections 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.273+0000 7f8f4741e740 1 -- 172.21.15.60:0/3693471613 wait complete. 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f4741e740 1 -- start start 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f4741e740 1 Processor -- start 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f4741e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5bbf40 0x55f62c5c5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f4741e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bc930 0x55f62c5c6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f4741e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f4741e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f62c520b40 con 0x55f62c5c1aa0 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f4741e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f62c4dd1c0 con 0x55f62c5bc930 2024-03-04T12:25:45.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f4741e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f62c5c4810 con 0x55f62c5bbf40 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f4741e740 10 client.0 _ll_register_callbacks cb 0x55f62c4c2f80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f3ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f3ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56124/0 (socket says 172.21.15.60:56124) 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f3ed9d640 1 -- 172.21.15.60:0/1867960350 learned_addr learned my addr 172.21.15.60:0/1867960350 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.274+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5bbf40 0x55f62c5c5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f37fff640 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bc930 0x55f62c5c6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f3ed9d640 1 -- 172.21.15.60:0/1867960350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5bbf40 msgr2=0x55f62c5c5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5bbf40 0x55f62c5c5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f3ed9d640 1 -- 172.21.15.60:0/1867960350 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bc930 msgr2=0x55f62c5c6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bc930 0x55f62c5c6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f3ed9d640 1 -- 172.21.15.60:0/1867960350 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f62c35df70 con 0x55f62c5c1aa0 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cd370 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8f38012d20 tx=0x7f8f3800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c56fb419bb7676e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f25ffb640 1 -- 172.21.15.60:0/1867960350 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3800fe00 con 0x55f62c5c1aa0 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f25ffb640 1 -- 172.21.15.60:0/1867960350 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f8f38012110 con 0x55f62c5c1aa0 2024-03-04T12:25:45.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f4741e740 1 -- 172.21.15.60:0/1867960350 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f280bcb00 con 0x55f62c5c1aa0 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f25ffb640 1 -- 172.21.15.60:0/1867960350 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f38024050 con 0x55f62c5c1aa0 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f4741e740 10 client.15693 fetch_fsmap learned FSMap version 29 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f4741e740 1 -- 172.21.15.60:0/1867960350 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55f62c1f5170 con 0x55f62c5c1aa0 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.275+0000 7f8f25ffb640 1 -- 172.21.15.60:0/1867960350 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f280bcb00 con 0x55f62c5c1aa0 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.276+0000 7f8f25ffb640 1 -- 172.21.15.60:0/1867960350 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8f38010430 con 0x55f62c5c1aa0 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.276+0000 7f8f4741e740 10 client.15693 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.276+0000 7f8f4741e740 10 client.15693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.276+0000 7f8f4741e740 20 client.15693 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.276+0000 7f8f4741e740 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55f62c5d1cc0 0x55f62c5d4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.276+0000 7f8f4741e740 4 client.15693 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.276+0000 7f8f4741e740 1 -- 172.21.15.60:0/1867960350 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f62c4f6960 con 0x55f62c5d1cc0 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.276+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55f62c5d1cc0 0x55f62c5d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.276+0000 7f8f25ffb640 10 client.15693 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.276+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55f62c5d1cc0 0x55f62c5d4100 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0a1dd7e63248470 server_cookie=eed1bdd25cc75d8 in_seq=0 out_seq=0 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f25ffb640 1 -- 172.21.15.60:0/1867960350 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f62c4f6960 con 0x55f62c5d1cc0 2024-03-04T12:25:45.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f25ffb640 10 client.15693 handle_command_reply: tid=0 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f4741e740 1 client.15693 shutdown 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f4741e740 1 -- 172.21.15.60:0/1867960350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5c1aa0 msgr2=0x55f62c5cd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f4741e740 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cd370 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8f38012d20 tx=0x7f8f3800f790 comp rx=0 tx=0).stop 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f4741e740 1 -- 172.21.15.60:0/1867960350 shutdown_connections 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f4741e740 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55f62c5d1cc0 0x55f62c5d4100 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f4741e740 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cd370 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f4741e740 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bc930 0x55f62c5c6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f4741e740 1 --2- 172.21.15.60:0/1867960350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5bbf40 0x55f62c5c5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f4741e740 1 -- 172.21.15.60:0/1867960350 >> 172.21.15.60:0/1867960350 conn(0x55f62c5b74b0 msgr2=0x55f62c5b6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f25ffb640 0 client.15693 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f4741e740 1 -- 172.21.15.60:0/1867960350 shutdown_connections 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f4741e740 1 -- 172.21.15.60:0/1867960350 wait complete. 2024-03-04T12:25:45.289 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.277+0000 7f8f4741e740 20 client.15693 trim_cache size 0 max 16384 2024-03-04T12:25:45.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.292+0000 7f8f4741e740 1 Processor -- start 2024-03-04T12:25:45.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.292+0000 7f8f4741e740 1 -- start start 2024-03-04T12:25:45.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.292+0000 7f8f4741e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bbf40 0x55f62c5bdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f4741e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 0x55f62c5be0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f4741e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f4741e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f401a9c80 con 0x55f62c5bc930 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f4741e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f401ab600 con 0x55f62c5bbf40 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f4741e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f401aa2e0 con 0x55f62c5c1aa0 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f37fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 0x55f62c5be0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f37fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 0x55f62c5be0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56128/0 (socket says 172.21.15.60:56128) 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f37fff640 1 -- 172.21.15.60:0/3194080167 learned_addr learned my addr 172.21.15.60:0/3194080167 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/3194080167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bbf40 0x55f62c5bdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/3194080167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f37fff640 1 -- 172.21.15.60:0/3194080167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5c1aa0 msgr2=0x55f62c5cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f37fff640 1 --2- 172.21.15.60:0/3194080167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f37fff640 1 -- 172.21.15.60:0/3194080167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bbf40 msgr2=0x55f62c5bdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.305 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f37fff640 1 --2- 172.21.15.60:0/3194080167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bbf40 0x55f62c5bdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f37fff640 1 -- 172.21.15.60:0/3194080167 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f62c5d9170 con 0x55f62c5bc930 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/3194080167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bbf40 0x55f62c5bdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/3194080167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f37fff640 1 --2- 172.21.15.60:0/3194080167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 0x55f62c5be0b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8f280bca90 tx=0x7f8f280b8e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c93486fa0a49f57 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f27fff640 1 -- 172.21.15.60:0/3194080167 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280baba0 con 0x55f62c5bc930 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f27fff640 1 -- 172.21.15.60:0/3194080167 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f280bd200 con 0x55f62c5bc930 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.293+0000 7f8f27fff640 1 -- 172.21.15.60:0/3194080167 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f280bae90 con 0x55f62c5bc930 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.294+0000 7f8f4741e740 1 -- 172.21.15.60:0/3194080167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 msgr2=0x55f62c5be0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.294+0000 7f8f4741e740 1 --2- 172.21.15.60:0/3194080167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 0x55f62c5be0b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8f280bca90 tx=0x7f8f280b8e70 comp rx=0 tx=0).stop 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.294+0000 7f8f4741e740 1 -- 172.21.15.60:0/3194080167 shutdown_connections 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.294+0000 7f8f4741e740 1 --2- 172.21.15.60:0/3194080167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.294+0000 7f8f4741e740 1 --2- 172.21.15.60:0/3194080167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bc930 0x55f62c5be0b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.294+0000 7f8f4741e740 1 --2- 172.21.15.60:0/3194080167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5bbf40 0x55f62c5bdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.294+0000 7f8f4741e740 1 -- 172.21.15.60:0/3194080167 >> 172.21.15.60:0/3194080167 conn(0x55f62c5b74b0 msgr2=0x55f62c5d8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.294+0000 7f8f4741e740 1 -- 172.21.15.60:0/3194080167 shutdown_connections 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.294+0000 7f8f4741e740 1 -- 172.21.15.60:0/3194080167 wait complete. 2024-03-04T12:25:45.306 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.294+0000 7f8f4741e740 1 -- start start 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.294+0000 7f8f4741e740 1 Processor -- start 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f4741e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bbf40 0x55f62c5cd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f4741e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5bc930 0x55f62c5cb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f4741e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f4741e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f401aafa0 con 0x55f62c5bbf40 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f4741e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f401aa940 con 0x55f62c5c1aa0 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f4741e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f401a7f20 con 0x55f62c5bc930 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f4741e740 10 client.0 _ll_register_callbacks cb 0x7f8f28005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f3e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bbf40 0x55f62c5cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f3e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bbf40 0x55f62c5cd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:56144/0 (socket says 172.21.15.60:56144) 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f3e59c640 1 -- 172.21.15.60:0/936269741 learned_addr learned my addr 172.21.15.60:0/936269741 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f37fff640 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5bc930 0x55f62c5cb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f3e59c640 1 -- 172.21.15.60:0/936269741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5bc930 msgr2=0x55f62c5cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5bc930 0x55f62c5cb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f3e59c640 1 -- 172.21.15.60:0/936269741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5c1aa0 msgr2=0x55f62c5cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f3e59c640 1 -- 172.21.15.60:0/936269741 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f62c5cc1e0 con 0x55f62c5bbf40 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f3ed9d640 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f3e59c640 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bbf40 0x55f62c5cd050 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8f30009fd0 tx=0x7f8f30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d20ce3ad51e248 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.295+0000 7f8f25ffb640 1 -- 172.21.15.60:0/936269741 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f30005020 con 0x55f62c5bbf40 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f25ffb640 1 -- 172.21.15.60:0/936269741 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f8f300051c0 con 0x55f62c5bbf40 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f4741e740 1 -- 172.21.15.60:0/936269741 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f280bd200 con 0x55f62c5bbf40 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f25ffb640 1 -- 172.21.15.60:0/936269741 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f30038050 con 0x55f62c5bbf40 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f4741e740 10 client.15699 fetch_fsmap learned FSMap version 29 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f4741e740 1 -- 172.21.15.60:0/936269741 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55f62c6290c0 con 0x55f62c5bbf40 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f25ffb640 1 -- 172.21.15.60:0/936269741 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f280bd200 con 0x55f62c5bbf40 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f25ffb640 1 -- 172.21.15.60:0/936269741 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8f3001c950 con 0x55f62c5bbf40 2024-03-04T12:25:45.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f4741e740 10 client.15699 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:25:45.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f4741e740 10 client.15699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:25:45.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f4741e740 20 client.15699 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:45.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f4741e740 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55f62c629a80 0x55f62c62bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f4741e740 4 client.15699 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T12:25:45.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f4741e740 1 -- 172.21.15.60:0/936269741 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f62c4e2080 con 0x55f62c629a80 2024-03-04T12:25:45.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.296+0000 7f8f37fff640 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55f62c629a80 0x55f62c62bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.297+0000 7f8f37fff640 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55f62c629a80 0x55f62c62bea0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85a524772367c407 server_cookie=fc5330c602054031 in_seq=0 out_seq=0 2024-03-04T12:25:45.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.297+0000 7f8f25ffb640 10 client.15699 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:45.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f25ffb640 1 -- 172.21.15.60:0/936269741 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f62c4e2080 con 0x55f62c629a80 2024-03-04T12:25:45.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f25ffb640 10 client.15699 handle_command_reply: tid=0 2024-03-04T12:25:45.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f4741e740 1 client.15699 shutdown 2024-03-04T12:25:45.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f4741e740 1 -- 172.21.15.60:0/936269741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bbf40 msgr2=0x55f62c5cd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f4741e740 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bbf40 0x55f62c5cd050 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8f30009fd0 tx=0x7f8f30004730 comp rx=0 tx=0).stop 2024-03-04T12:25:45.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f4741e740 1 -- 172.21.15.60:0/936269741 shutdown_connections 2024-03-04T12:25:45.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f4741e740 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55f62c629a80 0x55f62c62bea0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f25ffb640 0 client.15699 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:45.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f4741e740 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f62c5c1aa0 0x55f62c5cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f4741e740 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f62c5bc930 0x55f62c5cb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f4741e740 1 --2- 172.21.15.60:0/936269741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f62c5bbf40 0x55f62c5cd050 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f4741e740 1 -- 172.21.15.60:0/936269741 >> 172.21.15.60:0/936269741 conn(0x55f62c5b74b0 msgr2=0x55f62c5bdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f4741e740 1 -- 172.21.15.60:0/936269741 shutdown_connections 2024-03-04T12:25:45.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f4741e740 1 -- 172.21.15.60:0/936269741 wait complete. 2024-03-04T12:25:45.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.298+0000 7f8f4741e740 20 client.15699 trim_cache size 0 max 16384 2024-03-04T12:25:45.311 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:25:45.311 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": 0, 2024-03-04T12:25:45.311 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_tag": "31225d3a-a6a5-4ac2-8fd9-256dfcc051bd", 2024-03-04T12:25:45.311 INFO:teuthology.orchestra.run.smithi060.stdout: "mode": "asynchronous" 2024-03-04T12:25:45.311 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:25:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.299+0000 7f8f4483d640 1 -- 172.21.15.60:0/2309555376 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8f2c0a1e10 msgr2=0x7f8f2c0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.299+0000 7f8f4483d640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8f2c0a1e10 0x7f8f2c0a42d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8f300058c0 tx=0x7f8f30005f90 comp rx=0 tx=0).stop 2024-03-04T12:25:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.299+0000 7f8f4483d640 1 -- 172.21.15.60:0/2309555376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40101460 msgr2=0x7f8f40197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.299+0000 7f8f4483d640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40101460 0x7f8f40197bc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8f28036040 tx=0x7f8f2802c650 comp rx=0 tx=0).stop 2024-03-04T12:25:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.299+0000 7f8f4483d640 1 -- 172.21.15.60:0/2309555376 shutdown_connections 2024-03-04T12:25:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.299+0000 7f8f4483d640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f8f2c0a1e10 0x7f8f2c0a42d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.299+0000 7f8f4483d640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f40101e20 0x7f8f4019ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.299+0000 7f8f4483d640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8f40101460 0x7f8f40197bc0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.299+0000 7f8f4483d640 1 --2- 172.21.15.60:0/2309555376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8f40100270 0x7f8f40197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.299+0000 7f8f4483d640 1 -- 172.21.15.60:0/2309555376 >> 172.21.15.60:0/2309555376 conn(0x7f8f400fb800 msgr2=0x7f8f400fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.299+0000 7f8f4483d640 1 -- 172.21.15.60:0/2309555376 shutdown_connections 2024-03-04T12:25:45.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.299+0000 7f8f4483d640 1 -- 172.21.15.60:0/2309555376 wait complete. 2024-03-04T12:25:45.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T12:25:45.317 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T12:25:45.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:45 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1243206269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:45.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:45 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2827457817' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:45.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:45 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3030684530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:45.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:45 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1243206269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:25:45.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:45 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2827457817' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:25:45.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:45 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3030684530' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:25:45.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.415+0000 7efe0791b640 1 -- 172.21.15.60:0/2581903299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe00101420 msgr2=0x7efe001018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.415+0000 7efe0791b640 1 --2- 172.21.15.60:0/2581903299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe00101420 0x7efe001018a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009900 tx=0x7efdf002bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:45.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.415+0000 7efe0791b640 1 -- 172.21.15.60:0/2581903299 shutdown_connections 2024-03-04T12:25:45.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.415+0000 7efe0791b640 1 --2- 172.21.15.60:0/2581903299 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe00101de0 0x7efe00108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.415+0000 7efe0791b640 1 --2- 172.21.15.60:0/2581903299 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe00101420 0x7efe001018a0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.415+0000 7efe0791b640 1 --2- 172.21.15.60:0/2581903299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe00100230 0x7efe00100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.415+0000 7efe0791b640 1 -- 172.21.15.60:0/2581903299 >> 172.21.15.60:0/2581903299 conn(0x7efe000fb760 msgr2=0x7efe000fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.415+0000 7efe0791b640 1 -- 172.21.15.60:0/2581903299 shutdown_connections 2024-03-04T12:25:45.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.415+0000 7efe0791b640 1 -- 172.21.15.60:0/2581903299 wait complete. 2024-03-04T12:25:45.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe0791b640 1 Processor -- start 2024-03-04T12:25:45.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe0791b640 1 -- start start 2024-03-04T12:25:45.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe0791b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe00100230 0x7efe00197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe0791b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe00101420 0x7efe00197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe0791b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe00101de0 0x7efe0019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe0791b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0010bec0 con 0x7efe00101de0 2024-03-04T12:25:45.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe0791b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0010bd40 con 0x7efe00101420 2024-03-04T12:25:45.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe0791b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0010c040 con 0x7efe00100230 2024-03-04T12:25:45.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe05eb8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe00101de0 0x7efe0019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe056b7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe00100230 0x7efe00197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe05eb8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe00101de0 0x7efe0019ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40520/0 (socket says 172.21.15.60:40520) 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe05eb8640 1 -- 172.21.15.60:0/3742048205 learned_addr learned my addr 172.21.15.60:0/3742048205 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe04eb6640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe00101420 0x7efe00197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe05eb8640 1 -- 172.21.15.60:0/3742048205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe00100230 msgr2=0x7efe00197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe05eb8640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe00100230 0x7efe00197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe05eb8640 1 -- 172.21.15.60:0/3742048205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe00101420 msgr2=0x7efe00197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe05eb8640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe00101420 0x7efe00197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe05eb8640 1 -- 172.21.15.60:0/3742048205 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0019f270 con 0x7efe00101de0 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe056b7640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe00100230 0x7efe00197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.416+0000 7efe05eb8640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe00101de0 0x7efe0019ec00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009fd0 tx=0x7efdf800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.417+0000 7efdf67fc640 1 -- 172.21.15.60:0/3742048205 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf8017070 con 0x7efe00101de0 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.417+0000 7efdf67fc640 1 -- 172.21.15.60:0/3742048205 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf8003e50 con 0x7efe00101de0 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.417+0000 7efe0791b640 1 -- 172.21.15.60:0/3742048205 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0019f560 con 0x7efe00101de0 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.417+0000 7efdf67fc640 1 -- 172.21.15.60:0/3742048205 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf80106c0 con 0x7efe00101de0 2024-03-04T12:25:45.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.417+0000 7efe0791b640 1 -- 172.21.15.60:0/3742048205 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0019fa30 con 0x7efe00101de0 2024-03-04T12:25:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.419+0000 7efe0a5db740 1 Processor -- start 2024-03-04T12:25:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.420+0000 7efdf67fc640 1 -- 172.21.15.60:0/3742048205 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 19) v1 ==== 136804+0+0 (secure 0 0 0) 0x7efdf8003970 con 0x7efe00101de0 2024-03-04T12:25:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.420+0000 7efe0a5db740 1 -- start start 2024-03-04T12:25:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.420+0000 7efe0a5db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb4817b0 0x556eeb481bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.420+0000 7efe0a5db740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 0x556eeb486ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.420+0000 7efe0a5db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb487310 0x556eeb4897a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.420+0000 7efe0a5db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x556eeb3e5e00 con 0x556eeb487310 2024-03-04T12:25:45.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.420+0000 7efe0a5db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x556eeb38eb10 con 0x556eeb4817b0 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.420+0000 7efe0a5db740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x556eeb3e1ea0 con 0x556eeb4821a0 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.420+0000 7efe04eb6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb487310 0x556eeb4897a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.420+0000 7efe04eb6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb487310 0x556eeb4897a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40532/0 (socket says 172.21.15.60:40532) 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.420+0000 7efe04eb6640 1 -- 172.21.15.60:0/1867976152 learned_addr learned my addr 172.21.15.60:0/1867976152 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efdf67fc640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efdec0a1e10 0x7efdec0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efe056b7640 1 --2- 172.21.15.60:0/1867976152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 0x556eeb486ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efe04eb6640 1 --2- 172.21.15.60:0/1867976152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb4817b0 0x556eeb481bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efe05eb8640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efdec0a1e10 0x7efdec0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efe04eb6640 1 -- 172.21.15.60:0/1867976152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 msgr2=0x556eeb486ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efe04eb6640 1 --2- 172.21.15.60:0/1867976152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 0x556eeb486ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efe04eb6640 1 -- 172.21.15.60:0/1867976152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb4817b0 msgr2=0x556eeb481bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efe04eb6640 1 --2- 172.21.15.60:0/1867976152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb4817b0 0x556eeb481bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efe04eb6640 1 -- 172.21.15.60:0/1867976152 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556eeb1c47d0 con 0x556eeb487310 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efdf67fc640 1 -- 172.21.15.60:0/3742048205 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7efdf80ce020 con 0x7efe00101de0 2024-03-04T12:25:45.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efe04eb6640 1 --2- 172.21.15.60:0/1867976152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb487310 0x556eeb4897a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efdf0005ab0 tx=0x7efdf0005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21768a6f5547924 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efdeb7fe640 1 -- 172.21.15.60:0/1867976152 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0004030 con 0x556eeb487310 2024-03-04T12:25:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efdeb7fe640 1 -- 172.21.15.60:0/1867976152 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf0005530 con 0x556eeb487310 2024-03-04T12:25:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.421+0000 7efdeb7fe640 1 -- 172.21.15.60:0/1867976152 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0004bc0 con 0x556eeb487310 2024-03-04T12:25:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.422+0000 7efe0a5db740 1 -- 172.21.15.60:0/1867976152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb487310 msgr2=0x556eeb4897a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.422+0000 7efe05eb8640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efdec0a1e10 0x7efdec0a42d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efdf8097230 tx=0x7efdf8016b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.422+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1867976152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb487310 0x556eeb4897a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efdf0005ab0 tx=0x7efdf0005a20 comp rx=0 tx=0).stop 2024-03-04T12:25:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.422+0000 7efe0a5db740 1 -- 172.21.15.60:0/1867976152 shutdown_connections 2024-03-04T12:25:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.422+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1867976152 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb487310 0x556eeb4897a0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.422+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1867976152 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 0x556eeb486ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.422+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1867976152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb4817b0 0x556eeb481bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.422+0000 7efe0a5db740 1 -- 172.21.15.60:0/1867976152 >> 172.21.15.60:0/1867976152 conn(0x556eeb47cd20 msgr2=0x556eeb47d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.422+0000 7efe0a5db740 1 -- 172.21.15.60:0/1867976152 shutdown_connections 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.422+0000 7efe0a5db740 1 -- 172.21.15.60:0/1867976152 wait complete. 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.422+0000 7efe0a5db740 1 -- start start 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.422+0000 7efe0a5db740 1 Processor -- start 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe0a5db740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4817b0 0x556eeb48b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe0a5db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4821a0 0x556eeb48bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe0a5db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 0x556eeb492c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe0a5db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x556eeb48a200 con 0x556eeb4821a0 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe0a5db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x556eeb3a20a0 con 0x556eeb487310 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe0a5db740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x556eeb48a380 con 0x556eeb4817b0 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe0a5db740 10 client.0 _ll_register_callbacks cb 0x556eeb3a1a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe056b7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4821a0 0x556eeb48bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe056b7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4821a0 0x556eeb48bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40540/0 (socket says 172.21.15.60:40540) 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe056b7640 1 -- 172.21.15.60:0/4022107632 learned_addr learned my addr 172.21.15.60:0/4022107632 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:45.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe04eb6640 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4817b0 0x556eeb48b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe04eb6640 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 0x556eeb492c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe056b7640 1 -- 172.21.15.60:0/4022107632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4817b0 msgr2=0x556eeb48b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe056b7640 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4817b0 0x556eeb48b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe056b7640 1 -- 172.21.15.60:0/4022107632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 msgr2=0x556eeb492c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe056b7640 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 0x556eeb492c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe056b7640 1 -- 172.21.15.60:0/4022107632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556eeb0fa3f0 con 0x556eeb4821a0 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe04eb6640 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 0x556eeb492c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efe056b7640 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4821a0 0x556eeb48bcc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7efdfc004490 tx=0x7efdfc0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389b87c973721a1c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.423+0000 7efde8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efde8ff9640 1 -- 172.21.15.60:0/4022107632 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc0117d0 con 0x556eeb4821a0 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efde8ff9640 1 -- 172.21.15.60:0/4022107632 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7efdfc011970 con 0x556eeb4821a0 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efe0a5db740 1 -- 172.21.15.60:0/4022107632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdf0005530 con 0x556eeb4821a0 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efde8ff9640 1 -- 172.21.15.60:0/4022107632 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdfc011b70 con 0x556eeb4821a0 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efe0a5db740 10 client.15711 fetch_fsmap learned FSMap version 29 2024-03-04T12:25:45.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efe0a5db740 1 -- 172.21.15.60:0/4022107632 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x556eeb0ba0f0 con 0x556eeb4821a0 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efde8ff9640 1 -- 172.21.15.60:0/4022107632 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf0005530 con 0x556eeb4821a0 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efde8ff9640 1 -- 172.21.15.60:0/4022107632 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7efdfc010440 con 0x556eeb4821a0 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efe0a5db740 10 client.15711 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efe0a5db740 10 client.15711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efe0a5db740 20 client.15711 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efe0a5db740 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x556eeb497330 0x556eeb499750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efe0a5db740 4 client.15711 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.424+0000 7efe0a5db740 1 -- 172.21.15.60:0/4022107632 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556eeb3f3340 con 0x556eeb497330 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.425+0000 7efe04eb6640 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x556eeb497330 0x556eeb499750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.425+0000 7efde8ff9640 10 client.15711 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.425+0000 7efe04eb6640 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x556eeb497330 0x556eeb499750 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51c542c39430b17c server_cookie=376e0ec3575f0b2b in_seq=0 out_seq=0 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efde8ff9640 1 -- 172.21.15.60:0/4022107632 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556eeb3f3340 con 0x556eeb497330 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efde8ff9640 10 client.15711 handle_command_reply: tid=0 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efe0a5db740 1 client.15711 shutdown 2024-03-04T12:25:45.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efe0a5db740 1 -- 172.21.15.60:0/4022107632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4821a0 msgr2=0x556eeb48bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efe0a5db740 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4821a0 0x556eeb48bcc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7efdfc004490 tx=0x7efdfc0039c0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efe0a5db740 1 -- 172.21.15.60:0/4022107632 shutdown_connections 2024-03-04T12:25:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efe0a5db740 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x556eeb497330 0x556eeb499750 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efe0a5db740 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 0x556eeb492c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efe0a5db740 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4821a0 0x556eeb48bcc0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efe0a5db740 1 --2- 172.21.15.60:0/4022107632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4817b0 0x556eeb48b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efde8ff9640 0 client.15711 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efe0a5db740 1 -- 172.21.15.60:0/4022107632 >> 172.21.15.60:0/4022107632 conn(0x556eeb47cd20 msgr2=0x556eeb47bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efe0a5db740 1 -- 172.21.15.60:0/4022107632 shutdown_connections 2024-03-04T12:25:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efe0a5db740 1 -- 172.21.15.60:0/4022107632 wait complete. 2024-03-04T12:25:45.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.426+0000 7efe0a5db740 20 client.15711 trim_cache size 0 max 16384 2024-03-04T12:25:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.441+0000 7efe0a5db740 1 Processor -- start 2024-03-04T12:25:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.441+0000 7efe0a5db740 1 -- start start 2024-03-04T12:25:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.441+0000 7efe0a5db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 0x556eeb48b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.441+0000 7efe0a5db740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 0x556eeb48bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.441+0000 7efe0a5db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 0x556eeb47b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.441+0000 7efe0a5db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe001a9c10 con 0x556eeb4817b0 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.441+0000 7efe0a5db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe001ab590 con 0x556eeb487310 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.441+0000 7efe0a5db740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe001aa270 con 0x556eeb4821a0 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.441+0000 7efe04eb6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 0x556eeb48b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.441+0000 7efe04eb6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 0x556eeb48b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40548/0 (socket says 172.21.15.60:40548) 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.441+0000 7efe04eb6640 1 -- 172.21.15.60:0/1962040209 learned_addr learned my addr 172.21.15.60:0/1962040209 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efe04eb6640 1 -- 172.21.15.60:0/1962040209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 msgr2=0x556eeb48bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efe04eb6640 1 --2- 172.21.15.60:0/1962040209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 0x556eeb48bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efe04eb6640 1 -- 172.21.15.60:0/1962040209 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 msgr2=0x556eeb47b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efe04eb6640 1 --2- 172.21.15.60:0/1962040209 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 0x556eeb47b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efe04eb6640 1 -- 172.21.15.60:0/1962040209 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556eeb482580 con 0x556eeb4817b0 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efe04eb6640 1 --2- 172.21.15.60:0/1962040209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 0x556eeb48b780 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efdf002c7b0 tx=0x7efdf0005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d980e0e35258dbdb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efdeaffd640 1 -- 172.21.15.60:0/1962040209 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0005230 con 0x556eeb4817b0 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efdeaffd640 1 -- 172.21.15.60:0/1962040209 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf00053d0 con 0x556eeb4817b0 2024-03-04T12:25:45.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efdeaffd640 1 -- 172.21.15.60:0/1962040209 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0053360 con 0x556eeb4817b0 2024-03-04T12:25:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efe0a5db740 1 -- 172.21.15.60:0/1962040209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 msgr2=0x556eeb48b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1962040209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 0x556eeb48b780 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efdf002c7b0 tx=0x7efdf0005a20 comp rx=0 tx=0).stop 2024-03-04T12:25:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efe0a5db740 1 -- 172.21.15.60:0/1962040209 shutdown_connections 2024-03-04T12:25:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1962040209 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 0x556eeb47b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.442+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1962040209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 0x556eeb48bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1962040209 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 0x556eeb48b780 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 1 -- 172.21.15.60:0/1962040209 >> 172.21.15.60:0/1962040209 conn(0x556eeb47cd20 msgr2=0x556eeb49e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 1 -- 172.21.15.60:0/1962040209 shutdown_connections 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 1 -- 172.21.15.60:0/1962040209 wait complete. 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 1 -- start start 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 1 Processor -- start 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 0x556eeb4e9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 0x556eeb4e9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 0x556eeb4f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe001aaf30 con 0x556eeb4817b0 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe001aa8d0 con 0x556eeb487310 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe001a7eb0 con 0x556eeb4821a0 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe0a5db740 10 client.0 _ll_register_callbacks cb 0x7efdfc0040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe04eb6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 0x556eeb4e9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe04eb6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 0x556eeb4e9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40564/0 (socket says 172.21.15.60:40564) 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe04eb6640 1 -- 172.21.15.60:0/1664279736 learned_addr learned my addr 172.21.15.60:0/1664279736 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.443+0000 7efe056b7640 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 0x556eeb4e9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.456 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efe04eb6640 1 -- 172.21.15.60:0/1664279736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 msgr2=0x556eeb4e9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efe04eb6640 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 0x556eeb4e9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efe04eb6640 1 -- 172.21.15.60:0/1664279736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 msgr2=0x556eeb4f0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efe04eb6640 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 0x556eeb4f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efe04eb6640 1 -- 172.21.15.60:0/1664279736 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556eeb4f12a0 con 0x556eeb4817b0 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efe056b7640 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 0x556eeb4e9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efe04eb6640 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 0x556eeb4e9730 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009900 tx=0x7efdf00092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f4889a289674865 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efde8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efde8ff9640 1 -- 172.21.15.60:0/1664279736 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0060060 con 0x556eeb4817b0 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efde8ff9640 1 -- 172.21.15.60:0/1664279736 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7efdf00046c0 con 0x556eeb4817b0 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efe0a5db740 1 -- 172.21.15.60:0/1664279736 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdf00053d0 con 0x556eeb4817b0 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efde8ff9640 1 -- 172.21.15.60:0/1664279736 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdf00048c0 con 0x556eeb4817b0 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efe0a5db740 10 client.15717 fetch_fsmap learned FSMap version 29 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efe0a5db740 1 -- 172.21.15.60:0/1664279736 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x556eeb4f25b0 con 0x556eeb4817b0 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.444+0000 7efde8ff9640 1 -- 172.21.15.60:0/1664279736 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf00053d0 con 0x556eeb4817b0 2024-03-04T12:25:45.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.445+0000 7efde8ff9640 1 -- 172.21.15.60:0/1664279736 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7efdf0040670 con 0x556eeb4817b0 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.445+0000 7efe0a5db740 10 client.15717 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.445+0000 7efe0a5db740 10 client.15717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.445+0000 7efe0a5db740 20 client.15717 populate_metadata read hostname 'smithi060' 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.445+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x556eeb4f2ba0 0x556eeb4f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.445+0000 7efe0a5db740 4 client.15717 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.445+0000 7efe0a5db740 1 -- 172.21.15.60:0/1664279736 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556eeb3ed690 con 0x556eeb4f2ba0 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.445+0000 7efe056b7640 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x556eeb4f2ba0 0x556eeb4f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.445+0000 7efde8ff9640 10 client.15717 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.445+0000 7efe056b7640 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x556eeb4f2ba0 0x556eeb4f4fc0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52f7d327ccb1dd44 server_cookie=16901fd645f4504d in_seq=0 out_seq=0 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efde8ff9640 1 -- 172.21.15.60:0/1664279736 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556eeb3ed690 con 0x556eeb4f2ba0 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efde8ff9640 10 client.15717 handle_command_reply: tid=0 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0a5db740 1 client.15717 shutdown 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0a5db740 1 -- 172.21.15.60:0/1664279736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 msgr2=0x556eeb4e9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 0x556eeb4e9730 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009900 tx=0x7efdf00092f0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0a5db740 1 -- 172.21.15.60:0/1664279736 shutdown_connections 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x556eeb4f2ba0 0x556eeb4f4fc0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556eeb487310 0x556eeb4f0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556eeb4821a0 0x556eeb4e9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efde8ff9640 0 client.15717 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:25:45.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0a5db740 1 --2- 172.21.15.60:0/1664279736 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x556eeb4817b0 0x556eeb4e9730 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0a5db740 1 -- 172.21.15.60:0/1664279736 >> 172.21.15.60:0/1664279736 conn(0x556eeb47cd20 msgr2=0x556eeb48b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0a5db740 1 -- 172.21.15.60:0/1664279736 shutdown_connections 2024-03-04T12:25:45.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0a5db740 1 -- 172.21.15.60:0/1664279736 wait complete. 2024-03-04T12:25:45.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0a5db740 20 client.15717 trim_cache size 0 max 16384 2024-03-04T12:25:45.459 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:25:45.459 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T12:25:45.459 INFO:teuthology.orchestra.run.smithi060.stdout: "scrubs": { 2024-03-04T12:25:45.459 INFO:teuthology.orchestra.run.smithi060.stdout: "31225d3a-a6a5-4ac2-8fd9-256dfcc051bd": { 2024-03-04T12:25:45.459 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/", 2024-03-04T12:25:45.460 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "31225d3a-a6a5-4ac2-8fd9-256dfcc051bd", 2024-03-04T12:25:45.460 INFO:teuthology.orchestra.run.smithi060.stdout: "options": "recursive,force" 2024-03-04T12:25:45.460 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:25:45.460 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:25:45.460 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:25:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0791b640 1 -- 172.21.15.60:0/3742048205 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efdec0a1e10 msgr2=0x7efdec0a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0791b640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efdec0a1e10 0x7efdec0a42d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efdf8097230 tx=0x7efdf8016b40 comp rx=0 tx=0).stop 2024-03-04T12:25:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0791b640 1 -- 172.21.15.60:0/3742048205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe00101de0 msgr2=0x7efe0019ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.446+0000 7efe0791b640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe00101de0 0x7efe0019ec00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009fd0 tx=0x7efdf800b260 comp rx=0 tx=0).stop 2024-03-04T12:25:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.447+0000 7efe0791b640 1 -- 172.21.15.60:0/3742048205 shutdown_connections 2024-03-04T12:25:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.447+0000 7efe0791b640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7efdec0a1e10 0x7efdec0a42d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.447+0000 7efe0791b640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efe00101de0 0x7efe0019ec00 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.447+0000 7efe0791b640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe00101420 0x7efe00197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.447+0000 7efe0791b640 1 --2- 172.21.15.60:0/3742048205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe00100230 0x7efe00197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.447+0000 7efe0791b640 1 -- 172.21.15.60:0/3742048205 >> 172.21.15.60:0/3742048205 conn(0x7efe000fb760 msgr2=0x7efe000fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.447+0000 7efe0791b640 1 -- 172.21.15.60:0/3742048205 shutdown_connections 2024-03-04T12:25:45.461 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.447+0000 7efe0791b640 1 -- 172.21.15.60:0/3742048205 wait complete. 2024-03-04T12:25:45.464 INFO:tasks.cephfs.filesystem:scrub status for tag:31225d3a-a6a5-4ac2-8fd9-256dfcc051bd - {'path': '/', 'tag': '31225d3a-a6a5-4ac2-8fd9-256dfcc051bd', 'options': 'recursive,force'} 2024-03-04T12:25:45.520 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:45.513+0000 7fa722ed1640 1 -- 172.21.15.60:0/1786069285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fa71c19ff10 con 0x7fa71c101480 2024-03-04T12:25:46.046 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.039+0000 7fa719ffb640 1 -- 172.21.15.60:0/1786069285 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v20) v1 ==== 92+0+0 (secure 0 0 0) 0x7fa7080c0d20 con 0x7fa71c101480 2024-03-04T12:25:46.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.042+0000 7fa722ed1640 1 -- 172.21.15.60:0/1786069285 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa70c0a1e00 msgr2=0x7fa70c0a42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.042+0000 7fa722ed1640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa70c0a1e00 0x7fa70c0a42c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa71c1012e0 tx=0x7fa71000b390 comp rx=0 tx=0).stop 2024-03-04T12:25:46.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.042+0000 7fa722ed1640 1 -- 172.21.15.60:0/1786069285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa71c101480 msgr2=0x7fa71c197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.042+0000 7fa722ed1640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa71c101480 0x7fa71c197c80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa708009230 tx=0x7fa708005dd0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.042+0000 7fa722ed1640 1 -- 172.21.15.60:0/1786069285 shutdown_connections 2024-03-04T12:25:46.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.042+0000 7fa722ed1640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7fa70c0a1e00 0x7fa70c0a42c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.042+0000 7fa722ed1640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa71c101e40 0x7fa71c19eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.042+0000 7fa722ed1640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa71c101480 0x7fa71c197c80 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.042+0000 7fa722ed1640 1 --2- 172.21.15.60:0/1786069285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa71c100290 0x7fa71c197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.042+0000 7fa722ed1640 1 -- 172.21.15.60:0/1786069285 >> 172.21.15.60:0/1786069285 conn(0x7fa71c0fb840 msgr2=0x7fa71c0694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:46.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.044+0000 7fa722ed1640 1 -- 172.21.15.60:0/1786069285 shutdown_connections 2024-03-04T12:25:46.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.044+0000 7fa722ed1640 1 -- 172.21.15.60:0/1786069285 wait complete. 2024-03-04T12:25:46.070 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-04T12:25:46.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.174+0000 7f2c1950a640 1 -- 172.21.15.60:0/3382583053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c14102570 msgr2=0x7f2c141029f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.174+0000 7f2c1950a640 1 --2- 172.21.15.60:0/3382583053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c14102570 0x7f2c141029f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc009900 tx=0x7f2bfc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:46.181 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.174+0000 7f2c1950a640 1 -- 172.21.15.60:0/3382583053 shutdown_connections 2024-03-04T12:25:46.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.174+0000 7f2c1950a640 1 --2- 172.21.15.60:0/3382583053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c14102f30 0x7f2c141097c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.174+0000 7f2c1950a640 1 --2- 172.21.15.60:0/3382583053 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c14102570 0x7f2c141029f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.174+0000 7f2c1950a640 1 --2- 172.21.15.60:0/3382583053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c14101380 0x7f2c141017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.174+0000 7f2c1950a640 1 -- 172.21.15.60:0/3382583053 >> 172.21.15.60:0/3382583053 conn(0x7f2c140fc8f0 msgr2=0x7f2c140fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:46.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.174+0000 7f2c1950a640 1 -- 172.21.15.60:0/3382583053 shutdown_connections 2024-03-04T12:25:46.183 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.174+0000 7f2c1950a640 1 -- 172.21.15.60:0/3382583053 wait complete. 2024-03-04T12:25:46.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.174+0000 7f2c1950a640 1 Processor -- start 2024-03-04T12:25:46.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c1950a640 1 -- start start 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c1950a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c14101380 0x7f2c141976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c1950a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c14102570 0x7f2c14197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c1950a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c14102f30 0x7f2c1419ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c1950a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1410d060 con 0x7f2c14102f30 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c1950a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1410cee0 con 0x7f2c14101380 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c1950a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1410d1e0 con 0x7f2c14102570 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c137fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c14102f30 0x7f2c1419ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c137fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c14102f30 0x7f2c1419ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40602/0 (socket says 172.21.15.60:40602) 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c137fe640 1 -- 172.21.15.60:0/292828310 learned_addr learned my addr 172.21.15.60:0/292828310 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c12ffd640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c14101380 0x7f2c141976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c127fc640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c14102570 0x7f2c14197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c137fe640 1 -- 172.21.15.60:0/292828310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c14102570 msgr2=0x7f2c14197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c137fe640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c14102570 0x7f2c14197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c137fe640 1 -- 172.21.15.60:0/292828310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c14101380 msgr2=0x7f2c141976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c137fe640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c14101380 0x7f2c141976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c137fe640 1 -- 172.21.15.60:0/292828310 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1419f2d0 con 0x7f2c14102f30 2024-03-04T12:25:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c12ffd640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c14101380 0x7f2c141976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:25:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c127fc640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c14102570 0x7f2c14197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:25:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.175+0000 7f2c137fe640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c14102f30 0x7f2c1419ec60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400efd0 tx=0x7f2c0400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.176+0000 7f2c0bfff640 1 -- 172.21.15.60:0/292828310 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c04003b70 con 0x7f2c14102f30 2024-03-04T12:25:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.176+0000 7f2c1950a640 1 -- 172.21.15.60:0/292828310 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1419f5c0 con 0x7f2c14102f30 2024-03-04T12:25:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.176+0000 7f2c0bfff640 1 -- 172.21.15.60:0/292828310 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c04003d10 con 0x7f2c14102f30 2024-03-04T12:25:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.176+0000 7f2c0bfff640 1 -- 172.21.15.60:0/292828310 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c040109f0 con 0x7f2c14102f30 2024-03-04T12:25:46.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.176+0000 7f2c1950a640 1 -- 172.21.15.60:0/292828310 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1419fae0 con 0x7f2c14102f30 2024-03-04T12:25:46.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.177+0000 7f2c1950a640 1 -- 172.21.15.60:0/292828310 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c14108720 con 0x7f2c14102f30 2024-03-04T12:25:46.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.178+0000 7f2c0bfff640 1 -- 172.21.15.60:0/292828310 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) v1 ==== 136821+0+0 (secure 0 0 0) 0x7f2c04004020 con 0x7f2c14102f30 2024-03-04T12:25:46.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.178+0000 7f2c0bfff640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2c000a1e50 0x7f2c000a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:46.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.179+0000 7f2c12ffd640 1 -- 172.21.15.60:0/292828310 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2c000a1e50 msgr2=0x7f2c000a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1914863179 2024-03-04T12:25:46.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.179+0000 7f2c0bfff640 1 -- 172.21.15.60:0/292828310 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f2c040ce760 con 0x7f2c14102f30 2024-03-04T12:25:46.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.179+0000 7f2c12ffd640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2c000a1e50 0x7f2c000a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T12:25:46.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.183+0000 7f2c0bfff640 1 -- 172.21.15.60:0/292828310 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2c0409b6a0 con 0x7f2c14102f30 2024-03-04T12:25:46.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:46 smithi086 ceph-mon[36851]: pgmap v373: 129 pgs: 129 active+clean; 651 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-04T12:25:46.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:46 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/54006971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:46.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:46 smithi086 ceph-mon[36851]: scrub queued for path: / 2024-03-04T12:25:46.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:46 smithi086 ceph-mon[36851]: scrub summary: idle+waiting paths [/] 2024-03-04T12:25:46.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:46 smithi086 ceph-mon[36851]: scrub summary: active paths [/] 2024-03-04T12:25:46.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:46 smithi086 ceph-mon[36851]: scrub summary: idle+waiting paths [/] 2024-03-04T12:25:46.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:46 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1786069285' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-04T12:25:46.349 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:46 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: ignoring --setuser ceph since I am not root 2024-03-04T12:25:46.349 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:46 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: ignoring --setgroup ceph since I am not root 2024-03-04T12:25:46.349 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:46 smithi086 ceph-mgr[38710]: -- 172.21.15.86:0/808199160 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5630b65c85a0 con 0x5630b659d000 2024-03-04T12:25:46.351 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:46 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: ignoring --setuser ceph since I am not root 2024-03-04T12:25:46.352 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:46 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: ignoring --setgroup ceph since I am not root 2024-03-04T12:25:46.352 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:46 smithi063 ceph-mgr[38103]: -- 172.21.15.63:0/3020389054 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fa2b9e25a0 con 0x55fa2b9b6800 2024-03-04T12:25:46.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:46 smithi063 ceph-mon[36717]: pgmap v373: 129 pgs: 129 active+clean; 651 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-04T12:25:46.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:46 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/54006971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:46.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:46 smithi063 ceph-mon[36717]: scrub queued for path: / 2024-03-04T12:25:46.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:46 smithi063 ceph-mon[36717]: scrub summary: idle+waiting paths [/] 2024-03-04T12:25:46.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:46 smithi063 ceph-mon[36717]: scrub summary: active paths [/] 2024-03-04T12:25:46.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:46 smithi063 ceph-mon[36717]: scrub summary: idle+waiting paths [/] 2024-03-04T12:25:46.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:46 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1786069285' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-04T12:25:46.374 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.367+0000 7f2c1950a640 1 -- 172.21.15.60:0/292828310 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f2c14061f30 con 0x7f2c14102f30 2024-03-04T12:25:46.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.370+0000 7f2c0bfff640 1 -- 172.21.15.60:0/292828310 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2c0409afa0 con 0x7f2c14102f30 2024-03-04T12:25:46.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.373+0000 7f2c1950a640 1 -- 172.21.15.60:0/292828310 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2c000a1e50 msgr2=0x7f2c000a4310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T12:25:46.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.373+0000 7f2c1950a640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2c000a1e50 0x7f2c000a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.373+0000 7f2c1950a640 1 -- 172.21.15.60:0/292828310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c14102f30 msgr2=0x7f2c1419ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.373+0000 7f2c1950a640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c14102f30 0x7f2c1419ec60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400efd0 tx=0x7f2c0400c6a0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.373+0000 7f2c1950a640 1 -- 172.21.15.60:0/292828310 shutdown_connections 2024-03-04T12:25:46.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.373+0000 7f2c1950a640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f2c000a1e50 0x7f2c000a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.373+0000 7f2c1950a640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c14102f30 0x7f2c1419ec60 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.373+0000 7f2c1950a640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c14102570 0x7f2c14197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.373+0000 7f2c1950a640 1 --2- 172.21.15.60:0/292828310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c14101380 0x7f2c141976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.373+0000 7f2c1950a640 1 -- 172.21.15.60:0/292828310 >> 172.21.15.60:0/292828310 conn(0x7f2c140fc8f0 msgr2=0x7f2c141058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:46.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.373+0000 7f2c1950a640 1 -- 172.21.15.60:0/292828310 shutdown_connections 2024-03-04T12:25:46.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.373+0000 7f2c1950a640 1 -- 172.21.15.60:0/292828310 wait complete. 2024-03-04T12:25:46.390 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:46 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: ignoring --setuser ceph since I am not root 2024-03-04T12:25:46.390 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:46 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: ignoring --setgroup ceph since I am not root 2024-03-04T12:25:46.391 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:46 smithi060 ceph-mon[32745]: pgmap v373: 129 pgs: 129 active+clean; 651 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-04T12:25:46.391 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:46 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/54006971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:25:46.391 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:46 smithi060 ceph-mon[32745]: scrub queued for path: / 2024-03-04T12:25:46.391 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:46 smithi060 ceph-mon[32745]: scrub summary: idle+waiting paths [/] 2024-03-04T12:25:46.391 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:46 smithi060 ceph-mon[32745]: scrub summary: active paths [/] 2024-03-04T12:25:46.391 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:46 smithi060 ceph-mon[32745]: scrub summary: idle+waiting paths [/] 2024-03-04T12:25:46.391 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:46 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1786069285' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-04T12:25:46.392 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-04T12:25:46.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.489+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/2089613415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d08101e10 msgr2=0x7f6d081086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.489+0000 7f6d0f6f1640 1 --2- 172.21.15.60:0/2089613415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d08101e10 0x7f6d081086a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009ca0 tx=0x7f6d0002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.489+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/2089613415 shutdown_connections 2024-03-04T12:25:46.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.489+0000 7f6d0f6f1640 1 --2- 172.21.15.60:0/2089613415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d08101e10 0x7f6d081086a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.489+0000 7f6d0f6f1640 1 --2- 172.21.15.60:0/2089613415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d08101450 0x7f6d081018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.489+0000 7f6d0f6f1640 1 --2- 172.21.15.60:0/2089613415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d08100260 0x7f6d08100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.489+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/2089613415 >> 172.21.15.60:0/2089613415 conn(0x7f6d080fb7d0 msgr2=0x7f6d080fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:46.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.489+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/2089613415 shutdown_connections 2024-03-04T12:25:46.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.489+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/2089613415 wait complete. 2024-03-04T12:25:46.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0f6f1640 1 Processor -- start 2024-03-04T12:25:46.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0f6f1640 1 -- start start 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0f6f1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d08100260 0x7f6d08197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0f6f1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d08101450 0x7f6d08197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0f6f1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d08101e10 0x7f6d0819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0f6f1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0810bef0 con 0x7f6d08101e10 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0f6f1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0810bd70 con 0x7f6d08100260 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0f6f1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d0810c070 con 0x7f6d08101450 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0dc8e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d08101e10 0x7f6d0819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0dc8e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d08101e10 0x7f6d0819ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40622/0 (socket says 172.21.15.60:40622) 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0dc8e640 1 -- 172.21.15.60:0/414309807 learned_addr learned my addr 172.21.15.60:0/414309807 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0d48d640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d08100260 0x7f6d08197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0cc8c640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d08101450 0x7f6d08197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0dc8e640 1 -- 172.21.15.60:0/414309807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d08101450 msgr2=0x7f6d08197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0dc8e640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d08101450 0x7f6d08197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0dc8e640 1 -- 172.21.15.60:0/414309807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d08100260 msgr2=0x7f6d08197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.499 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0dc8e640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d08100260 0x7f6d08197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0dc8e640 1 -- 172.21.15.60:0/414309807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d0819f260 con 0x7f6d08101e10 2024-03-04T12:25:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0d48d640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d08100260 0x7f6d08197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:25:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6d0dc8e640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d08101e10 0x7f6d0819ebf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6d00037040 tx=0x7f6d00032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.490+0000 7f6cfe7fc640 1 -- 172.21.15.60:0/414309807 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00041020 con 0x7f6d08101e10 2024-03-04T12:25:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.491+0000 7f6cfe7fc640 1 -- 172.21.15.60:0/414309807 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d00034cd0 con 0x7f6d08101e10 2024-03-04T12:25:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.491+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/414309807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d0819f4f0 con 0x7f6d08101e10 2024-03-04T12:25:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.491+0000 7f6cfe7fc640 1 -- 172.21.15.60:0/414309807 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d00039040 con 0x7f6d08101e10 2024-03-04T12:25:46.500 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.491+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/414309807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d0819fa30 con 0x7f6d08101e10 2024-03-04T12:25:46.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.491+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/414309807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d0819fd50 con 0x7f6d08101e10 2024-03-04T12:25:46.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.493+0000 7f6cfe7fc640 1 -- 172.21.15.60:0/414309807 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) v1 ==== 136821+0+0 (secure 0 0 0) 0x7f6d00044020 con 0x7f6d08101e10 2024-03-04T12:25:46.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.494+0000 7f6cfe7fc640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6cf40a1ed0 0x7f6cf40a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:46.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.494+0000 7f6d0d48d640 1 -- 172.21.15.60:0/414309807 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6cf40a1ed0 msgr2=0x7f6cf40a4390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1914863179 2024-03-04T12:25:46.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.494+0000 7f6cfe7fc640 1 -- 172.21.15.60:0/414309807 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f6d000f04b0 con 0x7f6d08101e10 2024-03-04T12:25:46.501 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.494+0000 7f6d0d48d640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6cf40a1ed0 0x7f6cf40a4390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T12:25:46.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.497+0000 7f6cfe7fc640 1 -- 172.21.15.60:0/414309807 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6d0004b020 con 0x7f6d08101e10 2024-03-04T12:25:46.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.682+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/414309807 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f6d0819ff20 con 0x7f6d08101e10 2024-03-04T12:25:46.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.685+0000 7f6cfe7fc640 1 -- 172.21.15.60:0/414309807 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) v1 ==== 133+0+0 (secure 0 0 0) 0x7f6d000bd590 con 0x7f6d08101e10 2024-03-04T12:25:46.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.688+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/414309807 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6cf40a1ed0 msgr2=0x7f6cf40a4390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T12:25:46.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.688+0000 7f6d0f6f1640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6cf40a1ed0 0x7f6cf40a4390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.688+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/414309807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d08101e10 msgr2=0x7f6d0819ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.688+0000 7f6d0f6f1640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d08101e10 0x7f6d0819ebf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6d00037040 tx=0x7f6d00032f40 comp rx=0 tx=0).stop 2024-03-04T12:25:46.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.688+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/414309807 shutdown_connections 2024-03-04T12:25:46.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.688+0000 7f6d0f6f1640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f6cf40a1ed0 0x7f6cf40a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.688+0000 7f6d0f6f1640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6d08101e10 0x7f6d0819ebf0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.688+0000 7f6d0f6f1640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d08101450 0x7f6d08197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.688+0000 7f6d0f6f1640 1 --2- 172.21.15.60:0/414309807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d08100260 0x7f6d08197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.688+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/414309807 >> 172.21.15.60:0/414309807 conn(0x7f6d080fb7d0 msgr2=0x7f6d080fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:46.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.688+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/414309807 shutdown_connections 2024-03-04T12:25:46.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.688+0000 7f6d0f6f1640 1 -- 172.21.15.60:0/414309807 wait complete. 2024-03-04T12:25:46.706 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-04T12:25:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.806+0000 7f36f1945640 1 -- 172.21.15.60:0/112144085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36ec107050 msgr2=0x7f36ec1094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.806+0000 7f36f1945640 1 --2- 172.21.15.60:0/112144085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36ec107050 0x7f36ec1094a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f36e4009ca0 tx=0x7f36e402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.806+0000 7f36f1945640 1 -- 172.21.15.60:0/112144085 shutdown_connections 2024-03-04T12:25:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.806+0000 7f36f1945640 1 --2- 172.21.15.60:0/112144085 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36ec107050 0x7f36ec1094a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.806+0000 7f36f1945640 1 --2- 172.21.15.60:0/112144085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36ec104720 0x7f36ec106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.806+0000 7f36f1945640 1 --2- 172.21.15.60:0/112144085 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36ec101df0 0x7f36ec1041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.806+0000 7f36f1945640 1 -- 172.21.15.60:0/112144085 >> 172.21.15.60:0/112144085 conn(0x7f36ec0fb800 msgr2=0x7f36ec0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.806+0000 7f36f1945640 1 -- 172.21.15.60:0/112144085 shutdown_connections 2024-03-04T12:25:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.806+0000 7f36f1945640 1 -- 172.21.15.60:0/112144085 wait complete. 2024-03-04T12:25:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.806+0000 7f36f1945640 1 Processor -- start 2024-03-04T12:25:46.815 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36f1945640 1 -- start start 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36f1945640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36ec101df0 0x7f36ec101760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36f1945640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36ec104720 0x7f36ec0ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36f1945640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36ec107050 0x7f36ec1002e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36f1945640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec10d000 con 0x7f36ec107050 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36f1945640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec10ce80 con 0x7f36ec101df0 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36f1945640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36ec10d180 con 0x7f36ec104720 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36eb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36ec107050 0x7f36ec1002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36eaffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36ec101df0 0x7f36ec101760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36eb7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36ec107050 0x7f36ec1002e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40640/0 (socket says 172.21.15.60:40640) 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36eb7fe640 1 -- 172.21.15.60:0/1930210195 learned_addr learned my addr 172.21.15.60:0/1930210195 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36e3fff640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36ec104720 0x7f36ec0ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36eb7fe640 1 -- 172.21.15.60:0/1930210195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36ec104720 msgr2=0x7f36ec0ffda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36eb7fe640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36ec104720 0x7f36ec0ffda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36eb7fe640 1 -- 172.21.15.60:0/1930210195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36ec101df0 msgr2=0x7f36ec101760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:46.816 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36eb7fe640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36ec101df0 0x7f36ec101760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:46.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36eb7fe640 1 -- 172.21.15.60:0/1930210195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ec100b70 con 0x7f36ec107050 2024-03-04T12:25:46.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36eb7fe640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36ec107050 0x7f36ec1002e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f36e4037040 tx=0x7f36e4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:46.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.807+0000 7f36f0943640 1 -- 172.21.15.60:0/1930210195 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4041020 con 0x7f36ec107050 2024-03-04T12:25:46.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.808+0000 7f36f1945640 1 -- 172.21.15.60:0/1930210195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ec1a1710 con 0x7f36ec107050 2024-03-04T12:25:46.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.808+0000 7f36f0943640 1 -- 172.21.15.60:0/1930210195 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36e4034d90 con 0x7f36ec107050 2024-03-04T12:25:46.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.808+0000 7f36f0943640 1 -- 172.21.15.60:0/1930210195 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4039040 con 0x7f36ec107050 2024-03-04T12:25:46.817 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.808+0000 7f36f1945640 1 -- 172.21.15.60:0/1930210195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ec1a1c60 con 0x7f36ec107050 2024-03-04T12:25:46.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.808+0000 7f36f1945640 1 -- 172.21.15.60:0/1930210195 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ec0fd2f0 con 0x7f36ec107050 2024-03-04T12:25:46.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.810+0000 7f36f0943640 1 -- 172.21.15.60:0/1930210195 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 20) v1 ==== 136821+0+0 (secure 0 0 0) 0x7f36e4044020 con 0x7f36ec107050 2024-03-04T12:25:46.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.810+0000 7f36f0943640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 0x7f36d80a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:46.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.810+0000 7f36eaffd640 1 -- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 msgr2=0x7f36d80a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1914863179 2024-03-04T12:25:46.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.810+0000 7f36f0943640 1 -- 172.21.15.60:0/1930210195 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9150+0+0 (secure 0 0 0) 0x7f36e40f0510 con 0x7f36ec107050 2024-03-04T12:25:46.818 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.810+0000 7f36eaffd640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 0x7f36d80a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-04T12:25:46.821 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:46.814+0000 7f36f0943640 1 -- 172.21.15.60:0/1930210195 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f36e404b020 con 0x7f36ec107050 2024-03-04T12:25:46.928 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:46 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:46.643+0000 7f5039eea200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T12:25:46.928 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:46 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:46.744+0000 7f5039eea200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T12:25:47.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:47.011+0000 7f36eaffd640 1 -- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 msgr2=0x7f36d80a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1914863179 2024-03-04T12:25:47.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:47.011+0000 7f36eaffd640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 0x7f36d80a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-04T12:25:47.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:47.018+0000 7f36f1945640 1 -- 172.21.15.60:0/1930210195 --> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f36ec061770 con 0x7f36d80a1e50 2024-03-04T12:25:47.047 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:46 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:46.603+0000 7f28b18ca200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T12:25:47.047 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:46 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:46.696+0000 7f28b18ca200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T12:25:47.047 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:46 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:46.629+0000 7ff720283200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T12:25:47.047 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:46 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:46.727+0000 7ff720283200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T12:25:47.342 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:47 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1786069285' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-04T12:25:47.342 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:47 smithi086 ceph-mon[36851]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-04T12:25:47.342 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:47 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/292828310' entity='client.admin' 2024-03-04T12:25:47.342 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:47 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/414309807' entity='client.admin' 2024-03-04T12:25:47.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:47 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1786069285' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-04T12:25:47.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:47 smithi063 ceph-mon[36717]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-04T12:25:47.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:47 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/292828310' entity='client.admin' 2024-03-04T12:25:47.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:47 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/414309807' entity='client.admin' 2024-03-04T12:25:47.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:47.412+0000 7f36eaffd640 1 -- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 msgr2=0x7f36d80a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1914863179 2024-03-04T12:25:47.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:47.412+0000 7f36eaffd640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 0x7f36d80a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-04T12:25:47.420 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:47 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1786069285' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-04T12:25:47.420 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:47 smithi060 ceph-mon[32745]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-04T12:25:47.420 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:47 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/292828310' entity='client.admin' 2024-03-04T12:25:47.420 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:47 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/414309807' entity='client.admin' 2024-03-04T12:25:47.598 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:47 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:47.335+0000 7f28b18ca200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T12:25:47.599 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:47 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:47.427+0000 7f28b18ca200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T12:25:47.599 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:47 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:47.567+0000 7f28b18ca200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T12:25:47.678 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:47 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:47.412+0000 7f5039eea200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T12:25:47.678 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:47 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:47.508+0000 7f5039eea200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T12:25:47.678 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:47 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:47.652+0000 7f5039eea200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T12:25:47.742 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:47 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:47.394+0000 7ff720283200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T12:25:47.742 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:47 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:47.490+0000 7ff720283200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T12:25:47.742 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:47 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:47.637+0000 7ff720283200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T12:25:47.742 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:47 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:47.735+0000 7ff720283200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T12:25:48.015 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:47 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:47.663+0000 7f28b18ca200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T12:25:48.015 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:47 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:47.745+0000 7f28b18ca200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T12:25:48.015 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:48 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:48.008+0000 7f28b18ca200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T12:25:48.101 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:47 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:47.821+0000 7ff720283200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T12:25:48.101 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:48.094+0000 7ff720283200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T12:25:48.113 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:47 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:47.750+0000 7f5039eea200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T12:25:48.113 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:47 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:47.836+0000 7f5039eea200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T12:25:48.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:48.213+0000 7f36eaffd640 1 -- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 msgr2=0x7f36d80a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1914863179 2024-03-04T12:25:48.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:48.213+0000 7f36eaffd640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 0x7f36d80a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-04T12:25:48.306 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:48 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:48.119+0000 7f28b18ca200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T12:25:48.306 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:48 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:48.205+0000 7f28b18ca200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T12:25:48.306 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:48 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:48.300+0000 7f28b18ca200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T12:25:48.351 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:48.208+0000 7ff720283200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T12:25:48.352 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:48.297+0000 7ff720283200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T12:25:48.412 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:48 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:48.106+0000 7f5039eea200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T12:25:48.412 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:48 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:48.218+0000 7f5039eea200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T12:25:48.412 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:48 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:48.307+0000 7f5039eea200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T12:25:48.412 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:48 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:48.405+0000 7f5039eea200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T12:25:48.598 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:48 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:48.474+0000 7f28b18ca200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T12:25:48.678 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:48 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:48.585+0000 7f5039eea200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T12:25:48.852 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:48.395+0000 7ff720283200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T12:25:48.852 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:48.577+0000 7ff720283200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T12:25:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:49.815+0000 7f36eaffd640 1 -- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 msgr2=0x7f36d80a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1914863179 2024-03-04T12:25:49.822 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:49.815+0000 7f36eaffd640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 0x7f36d80a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-04T12:25:49.848 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:49 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:49.433+0000 7f28b18ca200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T12:25:49.848 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:49 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:49.564+0000 7f28b18ca200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T12:25:49.851 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:49 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:49.591+0000 7ff720283200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T12:25:49.851 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:49 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:49.726+0000 7ff720283200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T12:25:49.928 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:49 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:49.577+0000 7f5039eea200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T12:25:49.928 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:49 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:49.712+0000 7f5039eea200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T12:25:50.188 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:49 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:49.894+0000 7f28b18ca200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T12:25:50.188 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:49 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:49.984+0000 7f28b18ca200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T12:25:50.188 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:50 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:50.181+0000 7f28b18ca200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:25:50.351 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:50.069+0000 7ff720283200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T12:25:50.352 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:50.164+0000 7ff720283200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T12:25:50.358 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:50 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:50.054+0000 7f5039eea200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T12:25:50.358 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:50 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:50.149+0000 7f5039eea200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T12:25:50.358 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:50 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:50.351+0000 7f5039eea200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:25:50.598 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:50 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:50.363+0000 7f28b18ca200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T12:25:50.678 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:50 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:50.543+0000 7f5039eea200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T12:25:50.851 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:50.364+0000 7ff720283200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:25:50.852 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:50.554+0000 7ff720283200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T12:25:50.911 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:50 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:50.646+0000 7f28b18ca200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:25:51.113 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:50 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:50.840+0000 7f5039eea200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:25:51.114 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:51 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:51.107+0000 7f5039eea200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T12:25:51.122 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:50.847+0000 7ff720283200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T12:25:51.348 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:50 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:50.905+0000 7f28b18ca200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T12:25:51.348 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:51 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:51.141+0000 7f28b18ca200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T12:25:51.428 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:51 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:51.355+0000 7f5039eea200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T12:25:51.602 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:51 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:51.116+0000 7ff720283200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T12:25:51.602 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:51 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:51.359+0000 7ff720283200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T12:25:52.348 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:51 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:51.938+0000 7f28b18ca200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T12:25:52.601 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:52.192+0000 7ff720283200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T12:25:52.678 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:52 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:52.198+0000 7f5039eea200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T12:25:52.823 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:52 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:52.505+0000 7f28b18ca200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T12:25:52.823 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:52 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:52.602+0000 7f28b18ca200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T12:25:52.823 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:25:52 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z[38706]: 2024-03-04T12:25:52.816+0000 7f28b18ca200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T12:25:52.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:52.891+0000 7f36f0943640 1 -- 172.21.15.60:0/1930210195 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mgrmap(e 21) v1 ==== 136821+0+0 (secure 0 0 0) 0x7f36e40b94d0 con 0x7f36ec107050 2024-03-04T12:25:53.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:53.019+0000 7f36eaffd640 1 -- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 msgr2=0x7f36d80a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.60:6800/1914863179 2024-03-04T12:25:53.025 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:53.019+0000 7f36eaffd640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 0x7f36d80a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-04T12:25:53.098 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:52 smithi086 ceph-mon[36851]: Standby manager daemon z restarted 2024-03-04T12:25:53.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:52 smithi086 ceph-mon[36851]: Standby manager daemon z started 2024-03-04T12:25:53.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:52 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.86:0/2386027635' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T12:25:53.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:52 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.86:0/2386027635' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T12:25:53.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:52 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.86:0/2386027635' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T12:25:53.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:52 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.86:0/2386027635' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T12:25:53.101 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:52.785+0000 7ff720283200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T12:25:53.102 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:52.886+0000 7ff720283200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T12:25:53.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:52 smithi063 ceph-mon[36717]: Standby manager daemon z restarted 2024-03-04T12:25:53.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:52 smithi063 ceph-mon[36717]: Standby manager daemon z started 2024-03-04T12:25:53.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:52 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.86:0/2386027635' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T12:25:53.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:52 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.86:0/2386027635' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T12:25:53.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:52 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.86:0/2386027635' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T12:25:53.102 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:52 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.86:0/2386027635' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T12:25:53.131 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:52 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:52.787+0000 7f5039eea200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T12:25:53.131 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:52 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:52.894+0000 7f5039eea200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T12:25:53.131 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:52 smithi060 ceph-mon[32745]: Standby manager daemon z restarted 2024-03-04T12:25:53.132 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:52 smithi060 ceph-mon[32745]: Standby manager daemon z started 2024-03-04T12:25:53.132 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:52 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.86:0/2386027635' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T12:25:53.132 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:52 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.86:0/2386027635' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T12:25:53.132 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:52 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.86:0/2386027635' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T12:25:53.132 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:52 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.86:0/2386027635' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T12:25:53.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:53.148+0000 7f36f0943640 1 -- 172.21.15.60:0/1930210195 <== mon.0 v2:172.21.15.60:3300/0 8 ==== mgrmap(e 22) v1 ==== 136438+0+0 (secure 0 0 0) 0x7f36e40edcc0 con 0x7f36ec107050 2024-03-04T12:25:53.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:53.148+0000 7f36f0943640 1 -- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 msgr2=0x7f36d80a4310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-04T12:25:53.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:53.148+0000 7f36f0943640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/1914863179,v1:172.21.15.60:6801/1914863179] conn(0x7f36d80a1e50 0x7f36d80a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:53.392 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:53.125+0000 7f5039eea200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T12:25:53.425 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y[38099]: 2024-03-04T12:25:53.114+0000 7ff720283200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T12:25:53.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:53.386+0000 7f4f9c992700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:53.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:53.386+0000 7f4f9c992700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:53.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:53.386+0000 7f4f9c992700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:53.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:53.386+0000 7f4f9c992700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:53.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:53.386+0000 7f4f9c992700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:53.679 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:53.386+0000 7f4f988ca700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:53.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:53.386+0000 7f4f988ca700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:53.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:53.386+0000 7f4f988ca700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:53.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:53.386+0000 7f4f988ca700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:53.680 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:25:53.386+0000 7f4f988ca700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:25:54.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.153+0000 7f36f0943640 1 -- 172.21.15.60:0/1930210195 <== mon.0 v2:172.21.15.60:3300/0 9 ==== mgrmap(e 23) v1 ==== 136565+0+0 (secure 0 0 0) 0x7f36e4003970 con 0x7f36ec107050 2024-03-04T12:25:54.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.154+0000 7f36f0943640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f36d801baf0 0x7f36d801bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:54.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.154+0000 7f36f0943640 1 -- 172.21.15.60:0/1930210195 --> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f36ec061770 con 0x7f36d801baf0 2024-03-04T12:25:54.162 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.156+0000 7f36eaffd640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f36d801baf0 0x7f36d801bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:54.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.159+0000 7f36eaffd640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f36d801baf0 0x7f36d801bed0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f36dc005d00 tx=0x7f36dc00b410 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:54.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: mgrmap e21: x(active, since 9m), standbys: y, z 2024-03-04T12:25:54.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: Standby manager daemon y restarted 2024-03-04T12:25:54.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: Standby manager daemon y started 2024-03-04T12:25:54.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.63:0/27454208' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T12:25:54.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.63:0/27454208' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.63:0/27454208' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.? 172.21.15.63:0/27454208' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: Active manager daemon x restarted 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: Activating manager daemon x 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: osdmap e94: 12 total, 12 up, 12 in 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: mgrmap e22: x(active, starting, since 0.0122375s), standbys: y, z 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-04T12:25:54.180 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:25:54.181 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: Manager daemon x is now available 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T12:25:54.182 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:53 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T12:25:54.230 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.224+0000 7f36f0943640 1 -- 172.21.15.60:0/1930210195 <== mgr.15735 v2:172.21.15.60:6800/248765372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f36ec061770 con 0x7f36d801baf0 2024-03-04T12:25:54.230 INFO:teuthology.orchestra.run.smithi060.stdout:Schedule set for path / 2024-03-04T12:25:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.226+0000 7f36f1945640 1 -- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f36d801baf0 msgr2=0x7f36d801bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.226+0000 7f36f1945640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f36d801baf0 0x7f36d801bed0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f36dc005d00 tx=0x7f36dc00b410 comp rx=0 tx=0).stop 2024-03-04T12:25:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.226+0000 7f36f1945640 1 -- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36ec107050 msgr2=0x7f36ec1002e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.226+0000 7f36f1945640 1 --2- 172.21.15.60:0/1930210195 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f36ec107050 0x7f36ec1002e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f36e4037040 tx=0x7f36e4032f20 comp rx=0 tx=0).stop 2024-03-04T12:25:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.226+0000 7f36eb7fe640 1 -- 172.21.15.60:0/1930210195 reap_dead start 2024-03-04T12:25:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.226+0000 7f36f1945640 1 -- 172.21.15.60:0/1930210195 shutdown_connections 2024-03-04T12:25:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.226+0000 7f36f1945640 1 -- 172.21.15.60:0/1930210195 >> 172.21.15.60:0/1930210195 conn(0x7f36ec0fb800 msgr2=0x7f36ec102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.226+0000 7f36f1945640 1 -- 172.21.15.60:0/1930210195 shutdown_connections 2024-03-04T12:25:54.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.226+0000 7f36f1945640 1 -- 172.21.15.60:0/1930210195 wait complete. 2024-03-04T12:25:54.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-04T12:25:54.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: mgrmap e21: x(active, since 9m), standbys: y, z 2024-03-04T12:25:54.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: Standby manager daemon y restarted 2024-03-04T12:25:54.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: Standby manager daemon y started 2024-03-04T12:25:54.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.63:0/27454208' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.63:0/27454208' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.63:0/27454208' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.? 172.21.15.63:0/27454208' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: Active manager daemon x restarted 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: Activating manager daemon x 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: osdmap e94: 12 total, 12 up, 12 in 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: mgrmap e22: x(active, starting, since 0.0122375s), standbys: y, z 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-04T12:25:54.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:25:54.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: Manager daemon x is now available 2024-03-04T12:25:54.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:25:54.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:25:54.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T12:25:54.354 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:53 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T12:25:54.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: mgrmap e21: x(active, since 9m), standbys: y, z 2024-03-04T12:25:54.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: Standby manager daemon y restarted 2024-03-04T12:25:54.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: Standby manager daemon y started 2024-03-04T12:25:54.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.63:0/27454208' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-04T12:25:54.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.63:0/27454208' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T12:25:54.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.63:0/27454208' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-04T12:25:54.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.? 172.21.15.63:0/27454208' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T12:25:54.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: Active manager daemon x restarted 2024-03-04T12:25:54.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: Activating manager daemon x 2024-03-04T12:25:54.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: osdmap e94: 12 total, 12 up, 12 in 2024-03-04T12:25:54.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: mgrmap e22: x(active, starting, since 0.0122375s), standbys: y, z 2024-03-04T12:25:54.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-04T12:25:54.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-04T12:25:54.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-04T12:25:54.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T12:25:54.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T12:25:54.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T12:25:54.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: Manager daemon x is now available 2024-03-04T12:25:54.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:25:54.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:25:54.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-04T12:25:54.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:53 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-04T12:25:54.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.346+0000 7ff01d659640 1 -- 172.21.15.60:0/1555868722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff018101e10 msgr2=0x7ff0181086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.346+0000 7ff01d659640 1 --2- 172.21.15.60:0/1555868722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff018101e10 0x7ff0181086a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff010009860 tx=0x7ff01002bb70 comp rx=0 tx=0).stop 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.346+0000 7ff01d659640 1 -- 172.21.15.60:0/1555868722 shutdown_connections 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.346+0000 7ff01d659640 1 --2- 172.21.15.60:0/1555868722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff018101e10 0x7ff0181086a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.346+0000 7ff01d659640 1 --2- 172.21.15.60:0/1555868722 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff018101450 0x7ff0181018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.346+0000 7ff01d659640 1 --2- 172.21.15.60:0/1555868722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff018100260 0x7ff018100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.346+0000 7ff01d659640 1 -- 172.21.15.60:0/1555868722 >> 172.21.15.60:0/1555868722 conn(0x7ff0180fb7d0 msgr2=0x7ff0180fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.346+0000 7ff01d659640 1 -- 172.21.15.60:0/1555868722 shutdown_connections 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.346+0000 7ff01d659640 1 -- 172.21.15.60:0/1555868722 wait complete. 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.346+0000 7ff01d659640 1 Processor -- start 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.346+0000 7ff01d659640 1 -- start start 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.346+0000 7ff01d659640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff018100260 0x7ff0181975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff01d659640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff018101450 0x7ff018197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff01d659640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff018101e10 0x7ff01819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:54.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff01d659640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01810bef0 con 0x7ff018100260 2024-03-04T12:25:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff01d659640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01810bd70 con 0x7ff018101e10 2024-03-04T12:25:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff01d659640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01810c070 con 0x7ff018101450 2024-03-04T12:25:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff016ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff018100260 0x7ff0181975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff016ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff018100260 0x7ff0181975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40804/0 (socket says 172.21.15.60:40804) 2024-03-04T12:25:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff016ffd640 1 -- 172.21.15.60:0/958535024 learned_addr learned my addr 172.21.15.60:0/958535024 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff0167fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff018101450 0x7ff018197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff0177fe640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff018101e10 0x7ff01819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff0167fc640 1 -- 172.21.15.60:0/958535024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff018101e10 msgr2=0x7ff01819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff0167fc640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff018101e10 0x7ff01819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff0167fc640 1 -- 172.21.15.60:0/958535024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff018100260 msgr2=0x7ff0181975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff0167fc640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff018100260 0x7ff0181975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff0167fc640 1 -- 172.21.15.60:0/958535024 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01819f1c0 con 0x7ff018101450 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff016ffd640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff018100260 0x7ff0181975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff0167fc640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff018101450 0x7ff018197b30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff00c005fd0 tx=0x7ff00c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7feffffff640 1 -- 172.21.15.60:0/958535024 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c00cd00 con 0x7ff018101450 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff01d659640 1 -- 172.21.15.60:0/958535024 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff01819f4b0 con 0x7ff018101450 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7feffffff640 1 -- 172.21.15.60:0/958535024 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff00c0035f0 con 0x7ff018101450 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7feffffff640 1 -- 172.21.15.60:0/958535024 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c010690 con 0x7ff018101450 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.347+0000 7ff01d659640 1 -- 172.21.15.60:0/958535024 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01819fa00 con 0x7ff018101450 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.348+0000 7ff01d659640 1 -- 172.21.15.60:0/958535024 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01819fcc0 con 0x7ff018101450 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.349+0000 7feffffff640 1 -- 172.21.15.60:0/958535024 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 23) v1 ==== 136565+0+0 (secure 0 0 0) 0x7ff00c0037a0 con 0x7ff018101450 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.350+0000 7feffffff640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff0000a1b60 0x7ff0000a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.350+0000 7feffffff640 1 -- 172.21.15.60:0/958535024 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7ff00c0ce0a0 con 0x7ff018101450 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.350+0000 7ff016ffd640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff0000a1b60 0x7ff0000a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:54.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.350+0000 7ff016ffd640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff0000a1b60 0x7ff0000a4020 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff008007fd0 tx=0x7ff0080076a0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:54.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.353+0000 7feffffff640 1 -- 172.21.15.60:0/958535024 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff00c009510 con 0x7ff018101450 2024-03-04T12:25:54.564 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.558+0000 7ff01d659640 1 -- 172.21.15.60:0/958535024 --> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0180fd8e0 con 0x7ff0000a1b60 2024-03-04T12:25:54.626 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.619+0000 7feffffff640 1 -- 172.21.15.60:0/958535024 <== mgr.15735 v2:172.21.15.60:6800/248765372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7ff0180fd8e0 con 0x7ff0000a1b60 2024-03-04T12:25:54.626 INFO:teuthology.orchestra.run.smithi060.stdout:Retention added to path / 2024-03-04T12:25:54.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.622+0000 7ff01d659640 1 -- 172.21.15.60:0/958535024 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff0000a1b60 msgr2=0x7ff0000a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:54.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.622+0000 7ff01d659640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff0000a1b60 0x7ff0000a4020 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff008007fd0 tx=0x7ff0080076a0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.622+0000 7ff01d659640 1 -- 172.21.15.60:0/958535024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff018101450 msgr2=0x7ff018197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:54.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.622+0000 7ff01d659640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff018101450 0x7ff018197b30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff00c005fd0 tx=0x7ff00c00ee50 comp rx=0 tx=0).stop 2024-03-04T12:25:54.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.622+0000 7ff01d659640 1 -- 172.21.15.60:0/958535024 shutdown_connections 2024-03-04T12:25:54.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.622+0000 7ff01d659640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff018101e10 0x7ff01819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.622+0000 7ff01d659640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff018101450 0x7ff018197b30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.622+0000 7ff01d659640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff0000a1b60 0x7ff0000a4020 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.622+0000 7ff01d659640 1 --2- 172.21.15.60:0/958535024 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff018100260 0x7ff0181975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.622+0000 7ff01d659640 1 -- 172.21.15.60:0/958535024 >> 172.21.15.60:0/958535024 conn(0x7ff0180fb7d0 msgr2=0x7ff0180fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:54.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.622+0000 7ff01d659640 1 -- 172.21.15.60:0/958535024 shutdown_connections 2024-03-04T12:25:54.630 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.622+0000 7ff01d659640 1 -- 172.21.15.60:0/958535024 wait complete. 2024-03-04T12:25:54.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-04T12:25:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.746+0000 7f2168201640 1 -- 172.21.15.60:0/1423322145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160101380 msgr2=0x7f21601017a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.746+0000 7f2168201640 1 --2- 172.21.15.60:0/1423322145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160101380 0x7f21601017a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f215c009900 tx=0x7f215c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:25:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.746+0000 7f2168201640 1 -- 172.21.15.60:0/1423322145 shutdown_connections 2024-03-04T12:25:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.746+0000 7f2168201640 1 --2- 172.21.15.60:0/1423322145 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2160102f30 0x7f21601097c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.746+0000 7f2168201640 1 --2- 172.21.15.60:0/1423322145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2160102570 0x7f21601029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.746+0000 7f2168201640 1 --2- 172.21.15.60:0/1423322145 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160101380 0x7f21601017a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.746+0000 7f2168201640 1 -- 172.21.15.60:0/1423322145 >> 172.21.15.60:0/1423322145 conn(0x7f21600fc8f0 msgr2=0x7f21600fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.747+0000 7f2168201640 1 -- 172.21.15.60:0/1423322145 shutdown_connections 2024-03-04T12:25:54.753 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.747+0000 7f2168201640 1 -- 172.21.15.60:0/1423322145 wait complete. 2024-03-04T12:25:54.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.747+0000 7f2168201640 1 Processor -- start 2024-03-04T12:25:54.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.747+0000 7f2168201640 1 -- start start 2024-03-04T12:25:54.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.747+0000 7f2168201640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2160101380 0x7f2160197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:54.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.747+0000 7f2168201640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2160102570 0x7f2160197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:54.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.747+0000 7f2168201640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102f30 0x7f216019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:54.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.747+0000 7f2168201640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216010d060 con 0x7f2160102f30 2024-03-04T12:25:54.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.747+0000 7f2168201640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216010cee0 con 0x7f2160101380 2024-03-04T12:25:54.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.747+0000 7f2168201640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216010d1e0 con 0x7f2160102570 2024-03-04T12:25:54.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f216679e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102f30 0x7f216019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:54.756 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f216679e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102f30 0x7f216019ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40832/0 (socket says 172.21.15.60:40832) 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f216679e640 1 -- 172.21.15.60:0/1392411776 learned_addr learned my addr 172.21.15.60:0/1392411776 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f2165f9d640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2160101380 0x7f2160197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f216579c640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2160102570 0x7f2160197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f216679e640 1 -- 172.21.15.60:0/1392411776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2160102570 msgr2=0x7f2160197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f216679e640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2160102570 0x7f2160197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f216679e640 1 -- 172.21.15.60:0/1392411776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2160101380 msgr2=0x7f2160197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f216679e640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2160101380 0x7f2160197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f216679e640 1 -- 172.21.15.60:0/1392411776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216019f240 con 0x7f2160102f30 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f2165f9d640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2160101380 0x7f2160197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f216679e640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102f30 0x7f216019ebd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f215800eaa0 tx=0x7f215800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.748+0000 7f2156ffd640 1 -- 172.21.15.60:0/1392411776 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2158004d20 con 0x7f2160102f30 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.749+0000 7f2168201640 1 -- 172.21.15.60:0/1392411776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216019f530 con 0x7f2160102f30 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.749+0000 7f2156ffd640 1 -- 172.21.15.60:0/1392411776 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2158005020 con 0x7f2160102f30 2024-03-04T12:25:54.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.749+0000 7f2156ffd640 1 -- 172.21.15.60:0/1392411776 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21580116d0 con 0x7f2160102f30 2024-03-04T12:25:54.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.749+0000 7f2168201640 1 -- 172.21.15.60:0/1392411776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216019fa80 con 0x7f2160102f30 2024-03-04T12:25:54.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.750+0000 7f2168201640 1 -- 172.21.15.60:0/1392411776 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21601086a0 con 0x7f2160102f30 2024-03-04T12:25:54.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.751+0000 7f2156ffd640 1 -- 172.21.15.60:0/1392411776 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 23) v1 ==== 136565+0+0 (secure 0 0 0) 0x7f21580051d0 con 0x7f2160102f30 2024-03-04T12:25:54.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.752+0000 7f2156ffd640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f214c0a1b60 0x7f214c0a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:54.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.752+0000 7f2165f9d640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f214c0a1b60 0x7f214c0a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:54.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.752+0000 7f2156ffd640 1 -- 172.21.15.60:0/1392411776 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f21580cdf90 con 0x7f2160102f30 2024-03-04T12:25:54.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.753+0000 7f2165f9d640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f214c0a1b60 0x7f214c0a4020 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f215c02c170 tx=0x7f215c02df90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:54.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.757+0000 7f2156ffd640 1 -- 172.21.15.60:0/1392411776 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2158016030 con 0x7f2160102f30 2024-03-04T12:25:54.968 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.962+0000 7f2168201640 1 -- 172.21.15.60:0/1392411776 --> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f2160061770 con 0x7f214c0a1b60 2024-03-04T12:25:54.977 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.971+0000 7f2156ffd640 1 -- 172.21.15.60:0/1392411776 <== mgr.15735 v2:172.21.15.60:6800/248765372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f2160061770 con 0x7f214c0a1b60 2024-03-04T12:25:54.978 INFO:teuthology.orchestra.run.smithi060.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-04T00:00:00", "created": "2024-03-04T12:25:54", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-04T12:25:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.974+0000 7f2168201640 1 -- 172.21.15.60:0/1392411776 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f214c0a1b60 msgr2=0x7f214c0a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.974+0000 7f2168201640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f214c0a1b60 0x7f214c0a4020 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f215c02c170 tx=0x7f215c02df90 comp rx=0 tx=0).stop 2024-03-04T12:25:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.974+0000 7f2168201640 1 -- 172.21.15.60:0/1392411776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102f30 msgr2=0x7f216019ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.974+0000 7f2168201640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102f30 0x7f216019ebd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f215800eaa0 tx=0x7f215800eea0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.974+0000 7f2168201640 1 -- 172.21.15.60:0/1392411776 shutdown_connections 2024-03-04T12:25:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.974+0000 7f2168201640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f214c0a1b60 0x7f214c0a4020 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.974+0000 7f2168201640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2160102f30 0x7f216019ebd0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.974+0000 7f2168201640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2160102570 0x7f2160197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.974+0000 7f2168201640 1 --2- 172.21.15.60:0/1392411776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2160101380 0x7f2160197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.974+0000 7f2168201640 1 -- 172.21.15.60:0/1392411776 >> 172.21.15.60:0/1392411776 conn(0x7f21600fc8f0 msgr2=0x7f2160105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.974+0000 7f2168201640 1 -- 172.21.15.60:0/1392411776 shutdown_connections 2024-03-04T12:25:54.981 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:54.974+0000 7f2168201640 1 -- 172.21.15.60:0/1392411776 wait complete. 2024-03-04T12:25:54.991 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-04T12:25:55.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.094+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4001868231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638102f30 msgr2=0x7ff6381097c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:55.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.094+0000 7ff63cbdf640 1 --2- 172.21.15.60:0/4001868231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638102f30 0x7ff6381097c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff630009ca0 tx=0x7ff63002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.094+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4001868231 shutdown_connections 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.094+0000 7ff63cbdf640 1 --2- 172.21.15.60:0/4001868231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638102f30 0x7ff6381097c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.094+0000 7ff63cbdf640 1 --2- 172.21.15.60:0/4001868231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff638102570 0x7ff6381029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.094+0000 7ff63cbdf640 1 --2- 172.21.15.60:0/4001868231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff638101380 0x7ff6381017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.094+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4001868231 >> 172.21.15.60:0/4001868231 conn(0x7ff6380fc8f0 msgr2=0x7ff6380fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.094+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4001868231 shutdown_connections 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.094+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4001868231 wait complete. 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63cbdf640 1 Processor -- start 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63cbdf640 1 -- start start 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63cbdf640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638101380 0x7ff638197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63cbdf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff638102570 0x7ff638197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63cbdf640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff638102f30 0x7ff63819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63cbdf640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63810cfe0 con 0x7ff638101380 2024-03-04T12:25:55.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63cbdf640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63810ce60 con 0x7ff638102f30 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63cbdf640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63810d160 con 0x7ff638102570 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638101380 0x7ff638197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638101380 0x7ff638197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:40854/0 (socket says 172.21.15.60:40854) 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63659c640 1 -- 172.21.15.60:0/4294019809 learned_addr learned my addr 172.21.15.60:0/4294019809 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff62ffff640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff638102570 0x7ff638197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff636d9d640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff638102f30 0x7ff63819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63659c640 1 -- 172.21.15.60:0/4294019809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff638102570 msgr2=0x7ff638197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63659c640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff638102570 0x7ff638197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63659c640 1 -- 172.21.15.60:0/4294019809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff638102f30 msgr2=0x7ff63819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63659c640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff638102f30 0x7ff63819ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.095+0000 7ff63659c640 1 -- 172.21.15.60:0/4294019809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63819f220 con 0x7ff638101380 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.096+0000 7ff63659c640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638101380 0x7ff638197650 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff62800d950 tx=0x7ff62800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.096+0000 7ff62f7fe640 1 -- 172.21.15.60:0/4294019809 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff628017020 con 0x7ff638101380 2024-03-04T12:25:55.104 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.096+0000 7ff62f7fe640 1 -- 172.21.15.60:0/4294019809 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff628009c30 con 0x7ff638101380 2024-03-04T12:25:55.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.096+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4294019809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63819f510 con 0x7ff638101380 2024-03-04T12:25:55.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.096+0000 7ff62f7fe640 1 -- 172.21.15.60:0/4294019809 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62800f740 con 0x7ff638101380 2024-03-04T12:25:55.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.096+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4294019809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63819fa60 con 0x7ff638101380 2024-03-04T12:25:55.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.097+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4294019809 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6381086a0 con 0x7ff638101380 2024-03-04T12:25:55.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.098+0000 7ff62f7fe640 1 -- 172.21.15.60:0/4294019809 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 23) v1 ==== 136565+0+0 (secure 0 0 0) 0x7ff62801a020 con 0x7ff638101380 2024-03-04T12:25:55.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.098+0000 7ff62f7fe640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff6240a1b60 0x7ff6240a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:25:55.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.099+0000 7ff62f7fe640 1 -- 172.21.15.60:0/4294019809 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7ff6280cd810 con 0x7ff638101380 2024-03-04T12:25:55.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.099+0000 7ff62ffff640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff6240a1b60 0x7ff6240a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:25:55.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.099+0000 7ff62ffff640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff6240a1b60 0x7ff6240a4020 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff638198b10 tx=0x7ff620005f90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:25:55.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.103+0000 7ff62f7fe640 1 -- 172.21.15.60:0/4294019809 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff628021070 con 0x7ff638101380 2024-03-04T12:25:55.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.315+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4294019809 --> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff638061770 con 0x7ff6240a1b60 2024-03-04T12:25:55.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.321+0000 7ff62f7fe640 1 -- 172.21.15.60:0/4294019809 <== mgr.15735 v2:172.21.15.60:6800/248765372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7ff638061770 con 0x7ff6240a1b60 2024-03-04T12:25:55.327 INFO:teuthology.orchestra.run.smithi060.stdout:/ 1m 6m3h 2024-03-04T12:25:55.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.323+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4294019809 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff6240a1b60 msgr2=0x7ff6240a4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:55.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.323+0000 7ff63cbdf640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff6240a1b60 0x7ff6240a4020 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff638198b10 tx=0x7ff620005f90 comp rx=0 tx=0).stop 2024-03-04T12:25:55.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.323+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4294019809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638101380 msgr2=0x7ff638197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:25:55.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.323+0000 7ff63cbdf640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638101380 0x7ff638197650 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff62800d950 tx=0x7ff62800de10 comp rx=0 tx=0).stop 2024-03-04T12:25:55.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.323+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4294019809 shutdown_connections 2024-03-04T12:25:55.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.323+0000 7ff63cbdf640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff638102f30 0x7ff63819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:55.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.323+0000 7ff63cbdf640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff638102570 0x7ff638197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:55.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.323+0000 7ff63cbdf640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff6240a1b60 0x7ff6240a4020 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:55.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.323+0000 7ff63cbdf640 1 --2- 172.21.15.60:0/4294019809 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff638101380 0x7ff638197650 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:25:55.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.323+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4294019809 >> 172.21.15.60:0/4294019809 conn(0x7ff6380fc8f0 msgr2=0x7ff638105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:25:55.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.323+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4294019809 shutdown_connections 2024-03-04T12:25:55.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:25:55.323+0000 7ff63cbdf640 1 -- 172.21.15.60:0/4294019809 wait complete. 2024-03-04T12:25:55.340 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:55 smithi060 ceph-mon[32745]: [04/Mar/2024:12:25:53] ENGINE Bus STARTING 2024-03-04T12:25:55.340 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:55 smithi060 ceph-mon[32745]: [04/Mar/2024:12:25:53] ENGINE Serving on http://172.21.15.60:8765 2024-03-04T12:25:55.340 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:55 smithi060 ceph-mon[32745]: [04/Mar/2024:12:25:53] ENGINE Serving on https://172.21.15.60:7150 2024-03-04T12:25:55.340 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:55 smithi060 ceph-mon[32745]: [04/Mar/2024:12:25:53] ENGINE Bus STARTED 2024-03-04T12:25:55.340 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:55 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:55.340 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:55 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:55.340 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:55 smithi060 ceph-mon[32745]: mgrmap e23: x(active, since 1.0178s), standbys: y, z 2024-03-04T12:25:55.341 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-04T12:25:55.369 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-04T12:25:55.381 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-04T12:25:55.400 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi060.front.sepia.ceph.com 2024-03-04T12:25:55.401 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-04T12:25:55.412 INFO:teuthology.run_tasks:Running task workunit... 2024-03-04T12:25:55.426 INFO:tasks.workunit:Pulling workunits from ref 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:25:55.426 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-04T12:25:55.426 INFO:tasks.workunit:timeout=3h 2024-03-04T12:25:55.426 INFO:tasks.workunit:cleanup=True 2024-03-04T12:25:55.426 DEBUG:teuthology.orchestra.run.smithi060:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-04T12:25:55.442 INFO:teuthology.orchestra.run.smithi060.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-04T12:25:55.442 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-04T12:25:55.442 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-04T12:25:55.442 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T12:25:55.442 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-04T12:25:55.443 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-04 12:23:41.328932481 +0000 2024-03-04T12:25:55.443 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-04 12:23:41.328932481 +0000 2024-03-04T12:25:55.443 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-04 12:25:36.844903283 +0000 2024-03-04T12:25:55.443 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-04 12:23:41.328932481 +0000 2024-03-04T12:25:55.443 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-04T12:25:55.443 DEBUG:teuthology.orchestra.run.smithi060:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-04T12:25:55.520 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 717ce3cce8d0166e4e577faa8004e6f9cb4128c0 2024-03-04T12:25:55.575 INFO:tasks.workunit.client.0.smithi060.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-04T12:25:55.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:55 smithi086 ceph-mon[36851]: [04/Mar/2024:12:25:53] ENGINE Bus STARTING 2024-03-04T12:25:55.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:55 smithi086 ceph-mon[36851]: [04/Mar/2024:12:25:53] ENGINE Serving on http://172.21.15.60:8765 2024-03-04T12:25:55.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:55 smithi086 ceph-mon[36851]: [04/Mar/2024:12:25:53] ENGINE Serving on https://172.21.15.60:7150 2024-03-04T12:25:55.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:55 smithi086 ceph-mon[36851]: [04/Mar/2024:12:25:53] ENGINE Bus STARTED 2024-03-04T12:25:55.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:55 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:55.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:55 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:55.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:55 smithi086 ceph-mon[36851]: mgrmap e23: x(active, since 1.0178s), standbys: y, z 2024-03-04T12:25:55.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:55 smithi063 ceph-mon[36717]: [04/Mar/2024:12:25:53] ENGINE Bus STARTING 2024-03-04T12:25:55.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:55 smithi063 ceph-mon[36717]: [04/Mar/2024:12:25:53] ENGINE Serving on http://172.21.15.60:8765 2024-03-04T12:25:55.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:55 smithi063 ceph-mon[36717]: [04/Mar/2024:12:25:53] ENGINE Serving on https://172.21.15.60:7150 2024-03-04T12:25:55.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:55 smithi063 ceph-mon[36717]: [04/Mar/2024:12:25:53] ENGINE Bus STARTED 2024-03-04T12:25:55.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:55 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:55.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:55 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:55.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:55 smithi063 ceph-mon[36717]: mgrmap e23: x(active, since 1.0178s), standbys: y, z 2024-03-04T12:25:56.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:56 smithi060 ceph-mon[32745]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:56.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:56 smithi060 ceph-mon[32745]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:25:56.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:56 smithi060 ceph-mon[32745]: from='client.25363 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:56.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:56 smithi060 ceph-mon[32745]: from='client.15801 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:56.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:56 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:56 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:56 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:56 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:56 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:56 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:56 smithi086 ceph-mon[36851]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:56.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:56 smithi086 ceph-mon[36851]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:25:56.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:56 smithi086 ceph-mon[36851]: from='client.25363 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:56.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:56 smithi086 ceph-mon[36851]: from='client.15801 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:56.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:56 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:56 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:56 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:56 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:56 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:56 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:56 smithi063 ceph-mon[36717]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:56.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:56 smithi063 ceph-mon[36717]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:25:56.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:56 smithi063 ceph-mon[36717]: from='client.25363 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:56.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:56 smithi063 ceph-mon[36717]: from='client.15801 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:56.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:56 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:56 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:56 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:56 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:56 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:56.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:56 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:25:57.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:57.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: mgrmap e24: x(active, since 3s), standbys: y, z 2024-03-04T12:25:57.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi086 to 174.1M 2024-03-04T12:25:57.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: Unable to set osd_memory_target on smithi086 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-04T12:25:57.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: scrub summary: idle 2024-03-04T12:25:57.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi063 to 174.1M 2024-03-04T12:25:57.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: Unable to set osd_memory_target on smithi063 to 182642073: error parsing value: Value '182642073' is below minimum 939524096 2024-03-04T12:25:57.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi060 to 177.2M 2024-03-04T12:25:57.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: Unable to set osd_memory_target on smithi060 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-03-04T12:25:57.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:25:57.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:25:57.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: Updating smithi060:/etc/ceph/ceph.conf 2024-03-04T12:25:57.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-04T12:25:57.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:57 smithi060 ceph-mon[32745]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T12:25:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:25:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: mgrmap e24: x(active, since 3s), standbys: y, z 2024-03-04T12:25:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi086 to 174.1M 2024-03-04T12:25:57.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: Unable to set osd_memory_target on smithi086 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-04T12:25:57.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: scrub summary: idle 2024-03-04T12:25:57.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi063 to 174.1M 2024-03-04T12:25:57.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: Unable to set osd_memory_target on smithi063 to 182642073: error parsing value: Value '182642073' is below minimum 939524096 2024-03-04T12:25:57.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi060 to 177.2M 2024-03-04T12:25:57.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: Unable to set osd_memory_target on smithi060 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-03-04T12:25:57.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:25:57.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:25:57.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: Updating smithi060:/etc/ceph/ceph.conf 2024-03-04T12:25:57.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-04T12:25:57.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:57 smithi086 ceph-mon[36851]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T12:25:57.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:25:57.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='client.15807 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:25:57.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: mgrmap e24: x(active, since 3s), standbys: y, z 2024-03-04T12:25:57.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi086 to 174.1M 2024-03-04T12:25:57.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: Unable to set osd_memory_target on smithi086 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-04T12:25:57.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: scrub summary: idle 2024-03-04T12:25:57.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi063 to 174.1M 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: Unable to set osd_memory_target on smithi063 to 182642073: error parsing value: Value '182642073' is below minimum 939524096 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi060 to 177.2M 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: Unable to set osd_memory_target on smithi060 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:25:57.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: Updating smithi060:/etc/ceph/ceph.conf 2024-03-04T12:25:57.606 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: Updating smithi063:/etc/ceph/ceph.conf 2024-03-04T12:25:57.606 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:57 smithi063 ceph-mon[36717]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-04T12:25:59.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:25:59.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: Updating smithi086:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:25:59.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: Updating smithi060:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:25:59.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: Updating smithi063:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:25:59.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:25:59.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:25:59.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:25:59.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: Updating smithi086:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: Updating smithi063:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: Updating smithi060:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:25:59.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:25:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:25:59.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: Updating smithi086:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:25:59.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: Updating smithi060:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:25:59.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: Updating smithi063:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:25:59.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:25:59.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:25:59.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: Updating smithi086:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: Updating smithi063:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: Updating smithi060:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:25:59.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:25:59.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:25:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:25:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: Updating smithi086:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:25:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: Updating smithi060:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:25:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: Updating smithi063:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T12:25:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:25:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: Updating smithi060:/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:25:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-03-04T12:25:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: Updating smithi086:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T12:25:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: Updating smithi063:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: Updating smithi060:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:25:59.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:25:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:26:00.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:26:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:26:00.631+0000 7f4f7a50e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:26:00.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:26:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:26:00.631+0000 7f4f7a50e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:26:00.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:26:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:26:00.631+0000 7f4f7a50e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:26:00.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:26:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:26:00.631+0000 7f4f7a50e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:26:00.929 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:26:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:26:00.631+0000 7f4f7a50e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-04T12:26:01.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:01 smithi086 ceph-mon[36851]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:26:01.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:01 smithi063 ceph-mon[36717]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:26:01.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:01 smithi060 ceph-mon[32745]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:26:03.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:03 smithi086 ceph-mon[36851]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 79 op/s 2024-03-04T12:26:03.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:03 smithi086 ceph-mon[36851]: mgrmap e26: x(active, since 8s), standbys: y, z 2024-03-04T12:26:03.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:03 smithi063 ceph-mon[36717]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 79 op/s 2024-03-04T12:26:03.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:03 smithi063 ceph-mon[36717]: mgrmap e26: x(active, since 8s), standbys: y, z 2024-03-04T12:26:03.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:03 smithi060 ceph-mon[32745]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 79 op/s 2024-03-04T12:26:03.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:03 smithi060 ceph-mon[32745]: mgrmap e26: x(active, since 8s), standbys: y, z 2024-03-04T12:26:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:05 smithi086 ceph-mon[36851]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-04T12:26:05.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:05 smithi063 ceph-mon[36717]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-04T12:26:05.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:05 smithi060 ceph-mon[32745]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-04T12:26:07.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:07 smithi086 ceph-mon[36851]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 109 KiB/s wr, 72 op/s 2024-03-04T12:26:07.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:07 smithi063 ceph-mon[36717]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 109 KiB/s wr, 72 op/s 2024-03-04T12:26:07.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:07 smithi060 ceph-mon[32745]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 109 KiB/s wr, 72 op/s 2024-03-04T12:26:08.558 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.552+0000 7fa7960ea640 1 -- 172.21.15.60:0/257156043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa790106640 msgr2=0x7fa79010e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.552+0000 7fa7960ea640 1 --2- 172.21.15.60:0/257156043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa790106640 0x7fa79010e210 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa788009ca0 tx=0x7fa78802c0f0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.552+0000 7fa7960ea640 1 -- 172.21.15.60:0/257156043 shutdown_connections 2024-03-04T12:26:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.552+0000 7fa7960ea640 1 --2- 172.21.15.60:0/257156043 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa790106640 0x7fa79010e210 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.552+0000 7fa7960ea640 1 --2- 172.21.15.60:0/257156043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa790101f20 0x7fa790105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.552+0000 7fa7960ea640 1 --2- 172.21.15.60:0/257156043 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa790101570 0x7fa790101950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.552+0000 7fa7960ea640 1 -- 172.21.15.60:0/257156043 >> 172.21.15.60:0/257156043 conn(0x7fa7900fcdd0 msgr2=0x7fa7900ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.552+0000 7fa7960ea640 1 -- 172.21.15.60:0/257156043 shutdown_connections 2024-03-04T12:26:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.552+0000 7fa7960ea640 1 -- 172.21.15.60:0/257156043 wait complete. 2024-03-04T12:26:08.559 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.552+0000 7fa7960ea640 1 Processor -- start 2024-03-04T12:26:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa7960ea640 1 -- start start 2024-03-04T12:26:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa7960ea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa790101570 0x7fa790073630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa7960ea640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa790106640 0x7fa790073b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa7960ea640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7900740b0 0x7fa7901a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa7960ea640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa790111700 con 0x7fa790101570 2024-03-04T12:26:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa7960ea640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa790111580 con 0x7fa7900740b0 2024-03-04T12:26:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa7960ea640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa790111880 con 0x7fa790106640 2024-03-04T12:26:08.561 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa78f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa790101570 0x7fa790073630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa78f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa790101570 0x7fa790073630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34542/0 (socket says 172.21.15.60:34542) 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa78f7fe640 1 -- 172.21.15.60:0/363701261 learned_addr learned my addr 172.21.15.60:0/363701261 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa78effd640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa790106640 0x7fa790073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa78ffff640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7900740b0 0x7fa7901a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa78f7fe640 1 -- 172.21.15.60:0/363701261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa790106640 msgr2=0x7fa790073b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa790106640 0x7fa790073b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa78f7fe640 1 -- 172.21.15.60:0/363701261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7900740b0 msgr2=0x7fa7901a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7900740b0 0x7fa7901a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.553+0000 7fa78f7fe640 1 -- 172.21.15.60:0/363701261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7901a63d0 con 0x7fa790101570 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.554+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa790101570 0x7fa790073630 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa78000c920 tx=0x7fa78000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.554+0000 7fa78cff9640 1 -- 172.21.15.60:0/363701261 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780007c70 con 0x7fa790101570 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.554+0000 7fa78cff9640 1 -- 172.21.15.60:0/363701261 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa780007e10 con 0x7fa790101570 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.554+0000 7fa78cff9640 1 -- 172.21.15.60:0/363701261 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780018890 con 0x7fa790101570 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.554+0000 7fa7960ea640 1 -- 172.21.15.60:0/363701261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7901a66c0 con 0x7fa790101570 2024-03-04T12:26:08.562 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.554+0000 7fa7960ea640 1 -- 172.21.15.60:0/363701261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7901a6c10 con 0x7fa790101570 2024-03-04T12:26:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.556+0000 7fa78cff9640 1 -- 172.21.15.60:0/363701261 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7fa780018a30 con 0x7fa790101570 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.556+0000 7fa798da5740 1 Processor -- start 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.556+0000 7fa798da5740 1 -- start start 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa798da5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19d0d0 0x55ff6b19be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa798da5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 0x55ff6b19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa798da5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b1a7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa798da5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff6b1005f0 con 0x55ff6b19c360 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa798da5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff6b0a9930 con 0x55ff6b19d0d0 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa798da5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff6b0fc690 con 0x55ff6b1a58e0 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78cff9640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa7780a2030 0x7fa7780a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 0x55ff6b19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 0x55ff6b19c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34554/0 (socket says 172.21.15.60:34554) 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78f7fe640 1 -- 172.21.15.60:0/3029755834 learned_addr learned my addr 172.21.15.60:0/3029755834 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78ffff640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa7780a2030 0x7fa7780a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78effd640 1 --2- 172.21.15.60:0/3029755834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b1a7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78ffff640 1 --2- 172.21.15.60:0/3029755834 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19d0d0 0x55ff6b19be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78cff9640 1 -- 172.21.15.60:0/363701261 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fa7800ce880 con 0x7fa790101570 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78f7fe640 1 -- 172.21.15.60:0/3029755834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 msgr2=0x55ff6b1a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/3029755834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b1a7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78f7fe640 1 -- 172.21.15.60:0/3029755834 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19d0d0 msgr2=0x55ff6b19be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/3029755834 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19d0d0 0x55ff6b19be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78f7fe640 1 -- 172.21.15.60:0/3029755834 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff6ae15210 con 0x55ff6b19c360 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/3029755834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 0x55ff6b19c7e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa7800a35e0 tx=0x7fa780097050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416136af2516a53f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.557+0000 7fa78ffff640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa7780a2030 0x7fa7780a44f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa788004af0 tx=0x7fa78802e040 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.558+0000 7fa775ffb640 1 -- 172.21.15.60:0/3029755834 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78009ac80 con 0x55ff6b19c360 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.558+0000 7fa775ffb640 1 -- 172.21.15.60:0/3029755834 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa78009edd0 con 0x55ff6b19c360 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.558+0000 7fa775ffb640 1 -- 172.21.15.60:0/3029755834 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7800ce6a0 con 0x55ff6b19c360 2024-03-04T12:26:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.558+0000 7fa798da5740 1 -- 172.21.15.60:0/3029755834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 msgr2=0x55ff6b19c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.558+0000 7fa798da5740 1 --2- 172.21.15.60:0/3029755834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 0x55ff6b19c7e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa7800a35e0 tx=0x7fa780097050 comp rx=0 tx=0).stop 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.558+0000 7fa798da5740 1 -- 172.21.15.60:0/3029755834 shutdown_connections 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.558+0000 7fa798da5740 1 --2- 172.21.15.60:0/3029755834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b1a7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.558+0000 7fa798da5740 1 --2- 172.21.15.60:0/3029755834 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 0x55ff6b19c7e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.558+0000 7fa798da5740 1 --2- 172.21.15.60:0/3029755834 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19d0d0 0x55ff6b19be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.558+0000 7fa798da5740 1 -- 172.21.15.60:0/3029755834 >> 172.21.15.60:0/3029755834 conn(0x55ff6b163b90 msgr2=0x55ff6b197050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.558+0000 7fa798da5740 1 -- 172.21.15.60:0/3029755834 shutdown_connections 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.558+0000 7fa798da5740 1 -- 172.21.15.60:0/3029755834 wait complete. 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa798da5740 1 -- start start 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa798da5740 1 Processor -- start 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa798da5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19c360 0x55ff6b1a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa798da5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 0x55ff6b1a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa798da5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b1a3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa798da5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff6b1a85b0 con 0x55ff6b19d0d0 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa798da5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff6b0bcd90 con 0x55ff6b19c360 2024-03-04T12:26:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa798da5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff6b1a8730 con 0x55ff6b1a58e0 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa798da5740 10 client.0 _ll_register_callbacks cb 0x55ff6ae14a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa78effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 0x55ff6b1a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa78effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 0x55ff6b1a2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34560/0 (socket says 172.21.15.60:34560) 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa78effd640 1 -- 172.21.15.60:0/3839210655 learned_addr learned my addr 172.21.15.60:0/3839210655 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa78f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19c360 0x55ff6b1a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.559+0000 7fa78ffff640 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b1a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa78effd640 1 -- 172.21.15.60:0/3839210655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 msgr2=0x55ff6b1a3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa78effd640 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b1a3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa78effd640 1 -- 172.21.15.60:0/3839210655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19c360 msgr2=0x55ff6b1a2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa78effd640 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19c360 0x55ff6b1a2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa78effd640 1 -- 172.21.15.60:0/3839210655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff6aee1620 con 0x55ff6b19d0d0 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19c360 0x55ff6b1a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa78ffff640 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b1a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa78effd640 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 0x55ff6b1a2ec0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa784012700 tx=0x7fa784012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498acd0a490cb4ec server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa76effd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa76effd640 1 -- 172.21.15.60:0/3839210655 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78400f920 con 0x55ff6b19d0d0 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa76effd640 1 -- 172.21.15.60:0/3839210655 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fa78400fac0 con 0x55ff6b19d0d0 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa798da5740 1 -- 172.21.15.60:0/3839210655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa78009edd0 con 0x55ff6b19d0d0 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa76effd640 1 -- 172.21.15.60:0/3839210655 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa784025050 con 0x55ff6b19d0d0 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa798da5740 10 client.15831 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.560+0000 7fa798da5740 1 -- 172.21.15.60:0/3839210655 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ff6b1b4950 con 0x55ff6b19d0d0 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.561+0000 7fa76effd640 1 -- 172.21.15.60:0/3839210655 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa78009edd0 con 0x55ff6b19d0d0 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.561+0000 7fa76effd640 1 -- 172.21.15.60:0/3839210655 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa784010430 con 0x55ff6b19d0d0 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.561+0000 7fa798da5740 10 client.15831 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.561+0000 7fa798da5740 10 client.15831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.561+0000 7fa798da5740 20 client.15831 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.561+0000 7fa798da5740 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ff6b1b5e10 0x55ff6b1b8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.561+0000 7fa798da5740 4 client.15831 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:08.573 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.561+0000 7fa798da5740 1 -- 172.21.15.60:0/3839210655 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ff6b10db30 con 0x55ff6b1b5e10 2024-03-04T12:26:08.573 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.561+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ff6b1b5e10 0x55ff6b1b8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.573 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.561+0000 7fa76effd640 10 client.15831 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:08.573 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.561+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ff6b1b5e10 0x55ff6b1b8200 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4efc388a65d88f27 server_cookie=342b2b8c2bc3f347 in_seq=0 out_seq=0 2024-03-04T12:26:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa76effd640 1 -- 172.21.15.60:0/3839210655 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ff6b10db30 con 0x55ff6b1b5e10 2024-03-04T12:26:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa76effd640 10 client.15831 handle_command_reply: tid=0 2024-03-04T12:26:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa798da5740 1 client.15831 shutdown 2024-03-04T12:26:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa798da5740 1 -- 172.21.15.60:0/3839210655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 msgr2=0x55ff6b1a2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa798da5740 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 0x55ff6b1a2ec0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa784012700 tx=0x7fa784012bc0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa798da5740 1 -- 172.21.15.60:0/3839210655 shutdown_connections 2024-03-04T12:26:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa798da5740 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ff6b1b5e10 0x55ff6b1b8200 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa798da5740 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b1a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa798da5740 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 0x55ff6b1a2ec0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.575 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa76effd640 0 client.15831 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:08.575 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa798da5740 1 --2- 172.21.15.60:0/3839210655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19c360 0x55ff6b1a2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.575 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa798da5740 1 -- 172.21.15.60:0/3839210655 >> 172.21.15.60:0/3839210655 conn(0x55ff6b163b90 msgr2=0x55ff6b198da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:08.575 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa798da5740 1 -- 172.21.15.60:0/3839210655 shutdown_connections 2024-03-04T12:26:08.575 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.562+0000 7fa798da5740 1 -- 172.21.15.60:0/3839210655 wait complete. 2024-03-04T12:26:08.575 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.563+0000 7fa798da5740 20 client.15831 trim_cache size 0 max 16384 2024-03-04T12:26:08.583 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa798da5740 1 Processor -- start 2024-03-04T12:26:08.583 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa798da5740 1 -- start start 2024-03-04T12:26:08.583 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa798da5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 0x55ff6b1a50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.583 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa798da5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b19d0d0 0x55ff6b19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.583 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa798da5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b1a58e0 0x55ff6b1a0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.583 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa798da5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901b0630 con 0x55ff6b19c360 2024-03-04T12:26:08.583 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa798da5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901b1fb0 con 0x55ff6b1a58e0 2024-03-04T12:26:08.583 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa798da5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901b0c90 con 0x55ff6b19d0d0 2024-03-04T12:26:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa78f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 0x55ff6b1a50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa78f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 0x55ff6b1a50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34574/0 (socket says 172.21.15.60:34574) 2024-03-04T12:26:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa78f7fe640 1 -- 172.21.15.60:0/2674855810 learned_addr learned my addr 172.21.15.60:0/2674855810 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa78f7fe640 1 -- 172.21.15.60:0/2674855810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b19d0d0 msgr2=0x55ff6b19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/2674855810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b19d0d0 0x55ff6b19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa78f7fe640 1 -- 172.21.15.60:0/2674855810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b1a58e0 msgr2=0x55ff6b1a0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/2674855810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b1a58e0 0x55ff6b1a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.576+0000 7fa78f7fe640 1 -- 172.21.15.60:0/2674855810 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff6b19b630 con 0x55ff6b19c360 2024-03-04T12:26:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/2674855810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 0x55ff6b1a50b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa7800cbbc0 tx=0x7fa78000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4614e6a2f411629a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa7757fa640 1 -- 172.21.15.60:0/2674855810 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7800b91a0 con 0x55ff6b19c360 2024-03-04T12:26:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa7757fa640 1 -- 172.21.15.60:0/2674855810 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa78007e070 con 0x55ff6b19c360 2024-03-04T12:26:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa7757fa640 1 -- 172.21.15.60:0/2674855810 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7800cbd90 con 0x55ff6b19c360 2024-03-04T12:26:08.588 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa798da5740 1 -- 172.21.15.60:0/2674855810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 msgr2=0x55ff6b1a50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:08.588 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa798da5740 1 --2- 172.21.15.60:0/2674855810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 0x55ff6b1a50b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa7800cbbc0 tx=0x7fa78000cf90 comp rx=0 tx=0).stop 2024-03-04T12:26:08.588 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa798da5740 1 -- 172.21.15.60:0/2674855810 shutdown_connections 2024-03-04T12:26:08.588 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa798da5740 1 --2- 172.21.15.60:0/2674855810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b1a58e0 0x55ff6b1a0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.588 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa798da5740 1 --2- 172.21.15.60:0/2674855810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b19d0d0 0x55ff6b19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.588 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa798da5740 1 --2- 172.21.15.60:0/2674855810 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19c360 0x55ff6b1a50b0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.588 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa798da5740 1 -- 172.21.15.60:0/2674855810 >> 172.21.15.60:0/2674855810 conn(0x55ff6b163b90 msgr2=0x55ff6b1a1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:08.588 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa798da5740 1 -- 172.21.15.60:0/2674855810 shutdown_connections 2024-03-04T12:26:08.588 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.577+0000 7fa798da5740 1 -- 172.21.15.60:0/2674855810 wait complete. 2024-03-04T12:26:08.588 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa798da5740 1 -- start start 2024-03-04T12:26:08.588 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa798da5740 1 Processor -- start 2024-03-04T12:26:08.588 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa798da5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19c360 0x55ff6b1aba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa798da5740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 0x55ff6b1abf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa798da5740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b20c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa798da5740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901b1950 con 0x55ff6b19d0d0 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa798da5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901b12f0 con 0x55ff6b19c360 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa798da5740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7901ae8d0 con 0x55ff6b1a58e0 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa798da5740 10 client.0 _ll_register_callbacks cb 0x7fa784009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa78effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 0x55ff6b1abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa78effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 0x55ff6b1abf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34588/0 (socket says 172.21.15.60:34588) 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa78effd640 1 -- 172.21.15.60:0/1656329425 learned_addr learned my addr 172.21.15.60:0/1656329425 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.578+0000 7fa78ffff640 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b20c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19c360 0x55ff6b1aba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa78effd640 1 -- 172.21.15.60:0/1656329425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 msgr2=0x55ff6b20c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa78effd640 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b20c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa78effd640 1 -- 172.21.15.60:0/1656329425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19c360 msgr2=0x55ff6b1aba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:08.589 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa78effd640 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19c360 0x55ff6b1aba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa78effd640 1 -- 172.21.15.60:0/1656329425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff6b20c810 con 0x55ff6b19d0d0 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa78effd640 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 0x55ff6b1abf50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x55ff6b1a3f50 tx=0x7fa7840096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56eb668d95be0c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa76effd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa76effd640 1 -- 172.21.15.60:0/1656329425 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784022040 con 0x55ff6b19d0d0 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa76effd640 1 -- 172.21.15.60:0/1656329425 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fa784010430 con 0x55ff6b19d0d0 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa798da5740 1 -- 172.21.15.60:0/1656329425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa78007e070 con 0x55ff6b19d0d0 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa76effd640 1 -- 172.21.15.60:0/1656329425 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa784009ca0 con 0x55ff6b19d0d0 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa798da5740 10 client.15837 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa798da5740 1 -- 172.21.15.60:0/1656329425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ff6b20da50 con 0x55ff6b19d0d0 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa76effd640 1 -- 172.21.15.60:0/1656329425 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa78007e070 con 0x55ff6b19d0d0 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa76effd640 1 -- 172.21.15.60:0/1656329425 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa784005410 con 0x55ff6b19d0d0 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa798da5740 10 client.15837 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa798da5740 10 client.15837 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa798da5740 20 client.15837 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:08.590 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.579+0000 7fa798da5740 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ff6b1b5e10 0x55ff6b210150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:08.591 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.580+0000 7fa798da5740 4 client.15837 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T12:26:08.591 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.580+0000 7fa798da5740 1 -- 172.21.15.60:0/1656329425 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ff6b107e80 con 0x55ff6b1b5e10 2024-03-04T12:26:08.591 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.580+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ff6b1b5e10 0x55ff6b210150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:08.591 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.580+0000 7fa78f7fe640 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ff6b1b5e10 0x55ff6b210150 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7959b587fbfea2c server_cookie=a310e354c738a861 in_seq=0 out_seq=0 2024-03-04T12:26:08.591 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.580+0000 7fa76effd640 10 client.15837 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa76effd640 1 -- 172.21.15.60:0/1656329425 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ff6b107e80 con 0x55ff6b1b5e10 2024-03-04T12:26:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa76effd640 10 client.15837 handle_command_reply: tid=0 2024-03-04T12:26:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa798da5740 1 client.15837 shutdown 2024-03-04T12:26:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa798da5740 1 -- 172.21.15.60:0/1656329425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 msgr2=0x55ff6b1abf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa798da5740 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 0x55ff6b1abf50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x55ff6b1a3f50 tx=0x7fa7840096a0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa798da5740 1 -- 172.21.15.60:0/1656329425 shutdown_connections 2024-03-04T12:26:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa798da5740 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ff6b1b5e10 0x55ff6b210150 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa798da5740 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ff6b1a58e0 0x55ff6b20c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa76effd640 0 client.15837 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa798da5740 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ff6b19d0d0 0x55ff6b1abf50 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa798da5740 1 --2- 172.21.15.60:0/1656329425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ff6b19c360 0x55ff6b1aba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.601 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa798da5740 1 -- 172.21.15.60:0/1656329425 >> 172.21.15.60:0/1656329425 conn(0x55ff6b163b90 msgr2=0x55ff6b195940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:08.602 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa798da5740 1 -- 172.21.15.60:0/1656329425 shutdown_connections 2024-03-04T12:26:08.602 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa798da5740 1 -- 172.21.15.60:0/1656329425 wait complete. 2024-03-04T12:26:08.602 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.594+0000 7fa798da5740 20 client.15837 trim_cache size 0 max 16384 2024-03-04T12:26:08.602 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:{ 2024-03-04T12:26:08.602 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "message": "", 2024-03-04T12:26:08.602 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "return_code": 0 2024-03-04T12:26:08.602 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:} 2024-03-04T12:26:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.595+0000 7fa7960ea640 1 -- 172.21.15.60:0/363701261 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa7780a2030 msgr2=0x7fa7780a44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.595+0000 7fa7960ea640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa7780a2030 0x7fa7780a44f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa788004af0 tx=0x7fa78802e040 comp rx=0 tx=0).stop 2024-03-04T12:26:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.595+0000 7fa7960ea640 1 -- 172.21.15.60:0/363701261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa790101570 msgr2=0x7fa790073630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.595+0000 7fa7960ea640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa790101570 0x7fa790073630 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa78000c920 tx=0x7fa78000cde0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.595+0000 7fa7960ea640 1 -- 172.21.15.60:0/363701261 shutdown_connections 2024-03-04T12:26:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.595+0000 7fa7960ea640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7900740b0 0x7fa7901a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.595+0000 7fa7960ea640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa790106640 0x7fa790073b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.595+0000 7fa7960ea640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa7780a2030 0x7fa7780a44f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.603 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.595+0000 7fa7960ea640 1 --2- 172.21.15.60:0/363701261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa790101570 0x7fa790073630 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.595+0000 7fa7960ea640 1 -- 172.21.15.60:0/363701261 >> 172.21.15.60:0/363701261 conn(0x7fa7900fcdd0 msgr2=0x7fa7900fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.595+0000 7fa7960ea640 1 -- 172.21.15.60:0/363701261 shutdown_connections 2024-03-04T12:26:08.604 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:08.595+0000 7fa7960ea640 1 -- 172.21.15.60:0/363701261 wait complete. 2024-03-04T12:26:09.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:09 smithi086 ceph-mon[36851]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 66 op/s 2024-03-04T12:26:09.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:09 smithi063 ceph-mon[36717]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 66 op/s 2024-03-04T12:26:09.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:09 smithi060 ceph-mon[32745]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 66 op/s 2024-03-04T12:26:11.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:11 smithi086 ceph-mon[36851]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 67 op/s 2024-03-04T12:26:11.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:11 smithi063 ceph-mon[36717]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 67 op/s 2024-03-04T12:26:11.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:11 smithi060 ceph-mon[32745]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 67 op/s 2024-03-04T12:26:13.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:13 smithi086 ceph-mon[36851]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 102 KiB/s wr, 75 op/s 2024-03-04T12:26:13.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:13 smithi063 ceph-mon[36717]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 102 KiB/s wr, 75 op/s 2024-03-04T12:26:13.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:13 smithi060 ceph-mon[32745]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 102 KiB/s wr, 75 op/s 2024-03-04T12:26:15.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:15 smithi086 ceph-mon[36851]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-03-04T12:26:15.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:15 smithi063 ceph-mon[36717]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-03-04T12:26:15.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:15 smithi060 ceph-mon[32745]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 28 op/s 2024-03-04T12:26:15.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T12:26:15.465 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T12:26:15.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.579+0000 7f918bc68640 1 -- 172.21.15.60:0/2546360167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9184100a50 msgr2=0x7f918410d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.579+0000 7f918bc68640 1 --2- 172.21.15.60:0/2546360167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9184100a50 0x7f918410d680 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9174009900 tx=0x7f917402bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:15.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.579+0000 7f918bc68640 1 -- 172.21.15.60:0/2546360167 shutdown_connections 2024-03-04T12:26:15.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.579+0000 7f918bc68640 1 --2- 172.21.15.60:0/2546360167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f918410dbc0 0x7f918410ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.579+0000 7f918bc68640 1 --2- 172.21.15.60:0/2546360167 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9184100a50 0x7f918410d680 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.579+0000 7f918bc68640 1 --2- 172.21.15.60:0/2546360167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91841000a0 0x7f9184100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.579+0000 7f918bc68640 1 -- 172.21.15.60:0/2546360167 >> 172.21.15.60:0/2546360167 conn(0x7f91840fbc70 msgr2=0x7f91840fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:15.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.579+0000 7f918bc68640 1 -- 172.21.15.60:0/2546360167 shutdown_connections 2024-03-04T12:26:15.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.579+0000 7f918bc68640 1 -- 172.21.15.60:0/2546360167 wait complete. 2024-03-04T12:26:15.589 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918bc68640 1 Processor -- start 2024-03-04T12:26:15.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918bc68640 1 -- start start 2024-03-04T12:26:15.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918bc68640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91841000a0 0x7f9184199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918bc68640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9184100a50 0x7f918419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918bc68640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918410dbc0 0x7f918419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918bc68640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91841138c0 con 0x7f918410dbc0 2024-03-04T12:26:15.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918bc68640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9184113740 con 0x7f9184100a50 2024-03-04T12:26:15.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918bc68640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9184113a40 con 0x7f91841000a0 2024-03-04T12:26:15.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918a205640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918410dbc0 0x7f918419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918a205640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918410dbc0 0x7f918419e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53438/0 (socket says 172.21.15.60:53438) 2024-03-04T12:26:15.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918a205640 1 -- 172.21.15.60:0/3982681078 learned_addr learned my addr 172.21.15.60:0/3982681078 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:15.590 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f9189a04640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91841000a0 0x7f9184199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f9189203640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9184100a50 0x7f918419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918a205640 1 -- 172.21.15.60:0/3982681078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91841000a0 msgr2=0x7f9184199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918a205640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91841000a0 0x7f9184199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918a205640 1 -- 172.21.15.60:0/3982681078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9184100a50 msgr2=0x7f918419a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918a205640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9184100a50 0x7f918419a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f918a205640 1 -- 172.21.15.60:0/3982681078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918419edf0 con 0x7f918410dbc0 2024-03-04T12:26:15.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f9189203640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9184100a50 0x7f918419a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:15.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.580+0000 7f9189a04640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91841000a0 0x7f9184199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:15.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.581+0000 7f918a205640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918410dbc0 0x7f918419e6c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f917c00e970 tx=0x7f917c00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:15.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.581+0000 7f917affd640 1 -- 172.21.15.60:0/3982681078 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c00ccc0 con 0x7f918410dbc0 2024-03-04T12:26:15.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.581+0000 7f917affd640 1 -- 172.21.15.60:0/3982681078 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f917c005500 con 0x7f918410dbc0 2024-03-04T12:26:15.591 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.581+0000 7f918bc68640 1 -- 172.21.15.60:0/3982681078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f918419f0e0 con 0x7f918410dbc0 2024-03-04T12:26:15.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.581+0000 7f917affd640 1 -- 172.21.15.60:0/3982681078 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c010690 con 0x7f918410dbc0 2024-03-04T12:26:15.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.581+0000 7f918bc68640 1 -- 172.21.15.60:0/3982681078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91841a6970 con 0x7f918410dbc0 2024-03-04T12:26:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.583+0000 7f917affd640 1 -- 172.21.15.60:0/3982681078 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f917c0046c0 con 0x7f918410dbc0 2024-03-04T12:26:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.583+0000 7f918e846740 1 Processor -- start 2024-03-04T12:26:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f918e846740 1 -- start start 2024-03-04T12:26:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f917affd640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f91700a2010 0x7f91700a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f918e846740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 0x55d70e18db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f918e846740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18e0b0 0x55d70e196750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f918e846740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 0x55d70e199080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f918e846740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d70e0f0e00 con 0x55d70e18d790 2024-03-04T12:26:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f918e846740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d70e099b10 con 0x55d70e18e0b0 2024-03-04T12:26:15.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f918e846740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d70e0ecea0 con 0x55d70e196c90 2024-03-04T12:26:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f9189203640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f91700a2010 0x7f91700a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f9189a04640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 0x55d70e18db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f9189a04640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 0x55d70e18db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53444/0 (socket says 172.21.15.60:53444) 2024-03-04T12:26:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f9189a04640 1 -- 172.21.15.60:0/1945431586 learned_addr learned my addr 172.21.15.60:0/1945431586 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f917affd640 1 -- 172.21.15.60:0/3982681078 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f917c0cebc0 con 0x7f918410dbc0 2024-03-04T12:26:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f9189203640 1 --2- 172.21.15.60:0/1945431586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18e0b0 0x55d70e196750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f918a205640 1 --2- 172.21.15.60:0/1945431586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 0x55d70e199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.584+0000 7f9189a04640 1 -- 172.21.15.60:0/1945431586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 msgr2=0x55d70e199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.585+0000 7f9189a04640 1 --2- 172.21.15.60:0/1945431586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 0x55d70e199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.585+0000 7f9189a04640 1 -- 172.21.15.60:0/1945431586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18e0b0 msgr2=0x55d70e196750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.585+0000 7f9189a04640 1 --2- 172.21.15.60:0/1945431586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18e0b0 0x55d70e196750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.585+0000 7f9189a04640 1 -- 172.21.15.60:0/1945431586 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d70decf7d0 con 0x55d70e18d790 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.585+0000 7f9189203640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f91700a2010 0x7f91700a44d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9184101c30 tx=0x7f917402c6e0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.585+0000 7f9189a04640 1 --2- 172.21.15.60:0/1945431586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 0x55d70e18db70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9180005f40 tx=0x7f9180011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e10f15efc6a1503 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.585+0000 7f916ffff640 1 -- 172.21.15.60:0/1945431586 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9180011330 con 0x55d70e18d790 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.585+0000 7f916ffff640 1 -- 172.21.15.60:0/1945431586 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f918000fe80 con 0x55d70e18d790 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.585+0000 7f916ffff640 1 -- 172.21.15.60:0/1945431586 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91800193e0 con 0x55d70e18d790 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.586+0000 7f918e846740 1 -- 172.21.15.60:0/1945431586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 msgr2=0x55d70e18db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.586+0000 7f918e846740 1 --2- 172.21.15.60:0/1945431586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 0x55d70e18db70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9180005f40 tx=0x7f9180011d30 comp rx=0 tx=0).stop 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.586+0000 7f918e846740 1 -- 172.21.15.60:0/1945431586 shutdown_connections 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.586+0000 7f918e846740 1 --2- 172.21.15.60:0/1945431586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 0x55d70e199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.586+0000 7f918e846740 1 --2- 172.21.15.60:0/1945431586 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18e0b0 0x55d70e196750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.586+0000 7f918e846740 1 --2- 172.21.15.60:0/1945431586 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 0x55d70e18db70 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.586+0000 7f918e846740 1 -- 172.21.15.60:0/1945431586 >> 172.21.15.60:0/1945431586 conn(0x55d70e187ec0 msgr2=0x55d70e1882a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:15.601 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.586+0000 7f918e846740 1 -- 172.21.15.60:0/1945431586 shutdown_connections 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.586+0000 7f918e846740 1 -- 172.21.15.60:0/1945431586 wait complete. 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.586+0000 7f918e846740 1 -- start start 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.586+0000 7f918e846740 1 Processor -- start 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918e846740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18d790 0x55d70e1962c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918e846740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e18e0b0 0x55d70e193650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918e846740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e196c90 0x55d70e193b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918e846740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d70e199a80 con 0x55d70e196c90 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918e846740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d70e0ad0a0 con 0x55d70e18d790 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918e846740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d70e199c00 con 0x55d70e18e0b0 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918e846740 10 client.0 _ll_register_callbacks cb 0x55d70e0aca70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918a205640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e196c90 0x55d70e193b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918a205640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e196c90 0x55d70e193b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53450/0 (socket says 172.21.15.60:53450) 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918a205640 1 -- 172.21.15.60:0/2225364184 learned_addr learned my addr 172.21.15.60:0/2225364184 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f9189203640 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e18e0b0 0x55d70e193650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.602 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f9189a04640 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18d790 0x55d70e1962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918a205640 1 -- 172.21.15.60:0/2225364184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e18e0b0 msgr2=0x55d70e193650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918a205640 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e18e0b0 0x55d70e193650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918a205640 1 -- 172.21.15.60:0/2225364184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18d790 msgr2=0x55d70e1962c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918a205640 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18d790 0x55d70e1962c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918a205640 1 -- 172.21.15.60:0/2225364184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d70de053f0 con 0x55d70e196c90 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f9189a04640 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18d790 0x55d70e1962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f9189203640 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e18e0b0 0x55d70e193650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f918a205640 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e196c90 0x55d70e193b90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f917c00e940 tx=0x7f917c01c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4795eee1ea96035b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f916d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.587+0000 7f916d7fa640 1 -- 172.21.15.60:0/2225364184 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c09f110 con 0x55d70e196c90 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f916d7fa640 1 -- 172.21.15.60:0/2225364184 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f917c0a32d0 con 0x55d70e196c90 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f918e846740 1 -- 172.21.15.60:0/2225364184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d70e0fe340 con 0x55d70e196c90 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f916d7fa640 1 -- 172.21.15.60:0/2225364184 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f917c0a3760 con 0x55d70e196c90 2024-03-04T12:26:15.603 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f918e846740 10 client.15849 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f918e846740 1 -- 172.21.15.60:0/2225364184 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d70ddc50f0 con 0x55d70e196c90 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f916d7fa640 1 -- 172.21.15.60:0/2225364184 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d70e0fe340 con 0x55d70e196c90 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f916d7fa640 1 -- 172.21.15.60:0/2225364184 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f917c09d020 con 0x55d70e196c90 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f918e846740 10 client.15849 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f918e846740 10 client.15849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f918e846740 20 client.15849 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f918e846740 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55d70e1a6550 0x55d70e1a8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f918e846740 4 client.15849 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.588+0000 7f918e846740 1 -- 172.21.15.60:0/2225364184 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d70e0f8690 con 0x55d70e1a6550 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.589+0000 7f9189a04640 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55d70e1a6550 0x55d70e1a8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.589+0000 7f9189a04640 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55d70e1a6550 0x55d70e1a8940 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cf7e9d53c523795 server_cookie=bde88571e182b8f1 in_seq=0 out_seq=0 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.589+0000 7f916d7fa640 10 client.15849 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.589+0000 7f916d7fa640 1 -- 172.21.15.60:0/2225364184 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d70e0f8690 con 0x55d70e1a6550 2024-03-04T12:26:15.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f916d7fa640 10 client.15849 handle_command_reply: tid=0 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f918e846740 1 client.15849 shutdown 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f918e846740 1 -- 172.21.15.60:0/2225364184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e196c90 msgr2=0x55d70e193b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f918e846740 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e196c90 0x55d70e193b90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f917c00e940 tx=0x7f917c01c280 comp rx=0 tx=0).stop 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f918e846740 1 -- 172.21.15.60:0/2225364184 shutdown_connections 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f918e846740 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55d70e1a6550 0x55d70e1a8940 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f918e846740 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e196c90 0x55d70e193b90 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f918e846740 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e18e0b0 0x55d70e193650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f918e846740 1 --2- 172.21.15.60:0/2225364184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18d790 0x55d70e1962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f916d7fa640 0 client.15849 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f918e846740 1 -- 172.21.15.60:0/2225364184 >> 172.21.15.60:0/2225364184 conn(0x55d70e187ec0 msgr2=0x55d70e189c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f918e846740 1 -- 172.21.15.60:0/2225364184 shutdown_connections 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f918e846740 1 -- 172.21.15.60:0/2225364184 wait complete. 2024-03-04T12:26:15.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.590+0000 7f918e846740 20 client.15849 trim_cache size 0 max 16384 2024-03-04T12:26:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.605+0000 7f918e846740 1 Processor -- start 2024-03-04T12:26:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.605+0000 7f918e846740 1 -- start start 2024-03-04T12:26:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.605+0000 7f918e846740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18d790 0x55d70e195d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.605+0000 7f918e846740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18e0b0 0x55d70e1962c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.605+0000 7f918e846740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 0x55d70e191f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.605+0000 7f918e846740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91841b06e0 con 0x55d70e18e0b0 2024-03-04T12:26:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.605+0000 7f918e846740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91841b2060 con 0x55d70e18d790 2024-03-04T12:26:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.605+0000 7f918e846740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91841b0d40 con 0x55d70e196c90 2024-03-04T12:26:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f9189203640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18e0b0 0x55d70e1962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f9189203640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18e0b0 0x55d70e1962c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53454/0 (socket says 172.21.15.60:53454) 2024-03-04T12:26:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f9189203640 1 -- 172.21.15.60:0/749606671 learned_addr learned my addr 172.21.15.60:0/749606671 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:15.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f918a205640 1 --2- 172.21.15.60:0/749606671 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 0x55d70e191f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f9189a04640 1 --2- 172.21.15.60:0/749606671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18d790 0x55d70e195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f9189203640 1 -- 172.21.15.60:0/749606671 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 msgr2=0x55d70e191f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f9189203640 1 --2- 172.21.15.60:0/749606671 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 0x55d70e191f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f9189203640 1 -- 172.21.15.60:0/749606671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18d790 msgr2=0x55d70e195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f9189203640 1 --2- 172.21.15.60:0/749606671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18d790 0x55d70e195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f9189203640 1 -- 172.21.15.60:0/749606671 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d70e1adae0 con 0x55d70e18e0b0 2024-03-04T12:26:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f9189a04640 1 --2- 172.21.15.60:0/749606671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18d790 0x55d70e195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f9189203640 1 --2- 172.21.15.60:0/749606671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18e0b0 0x55d70e1962c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9174009900 tx=0x7f9174046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430424267db3306 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.606+0000 7f916f7fe640 1 -- 172.21.15.60:0/749606671 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9174054070 con 0x55d70e18e0b0 2024-03-04T12:26:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.607+0000 7f916f7fe640 1 -- 172.21.15.60:0/749606671 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9174004440 con 0x55d70e18e0b0 2024-03-04T12:26:15.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.607+0000 7f916f7fe640 1 -- 172.21.15.60:0/749606671 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91740035f0 con 0x55d70e18e0b0 2024-03-04T12:26:15.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.607+0000 7f918e846740 1 -- 172.21.15.60:0/749606671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18e0b0 msgr2=0x55d70e1962c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.607+0000 7f918e846740 1 --2- 172.21.15.60:0/749606671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18e0b0 0x55d70e1962c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9174009900 tx=0x7f9174046000 comp rx=0 tx=0).stop 2024-03-04T12:26:15.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.607+0000 7f918e846740 1 -- 172.21.15.60:0/749606671 shutdown_connections 2024-03-04T12:26:15.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.607+0000 7f918e846740 1 --2- 172.21.15.60:0/749606671 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 0x55d70e191f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.607+0000 7f918e846740 1 --2- 172.21.15.60:0/749606671 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18e0b0 0x55d70e1962c0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.607+0000 7f918e846740 1 --2- 172.21.15.60:0/749606671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18d790 0x55d70e195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.607+0000 7f918e846740 1 -- 172.21.15.60:0/749606671 >> 172.21.15.60:0/749606671 conn(0x55d70e187ec0 msgr2=0x55d70e193610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.607+0000 7f918e846740 1 -- 172.21.15.60:0/749606671 shutdown_connections 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.607+0000 7f918e846740 1 -- 172.21.15.60:0/749606671 wait complete. 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f918e846740 1 -- start start 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f918e846740 1 Processor -- start 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f918e846740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 0x55d70e19e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f918e846740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18e0b0 0x55d70e1900b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f918e846740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 0x55d70e190630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f918e846740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91841b1a00 con 0x55d70e18d790 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f918e846740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91841b13a0 con 0x55d70e18e0b0 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f918e846740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91841ae980 con 0x55d70e196c90 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f918e846740 10 client.0 _ll_register_callbacks cb 0x7f917c09a920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:15.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f9189a04640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 0x55d70e19e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f9189a04640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 0x55d70e19e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53466/0 (socket says 172.21.15.60:53466) 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f9189a04640 1 -- 172.21.15.60:0/1346191663 learned_addr learned my addr 172.21.15.60:0/1346191663 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f918a205640 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 0x55d70e190630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f9189203640 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18e0b0 0x55d70e1900b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f9189a04640 1 -- 172.21.15.60:0/1346191663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 msgr2=0x55d70e190630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f9189a04640 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 0x55d70e190630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f9189a04640 1 -- 172.21.15.60:0/1346191663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18e0b0 msgr2=0x55d70e1900b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f9189a04640 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18e0b0 0x55d70e1900b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.608+0000 7f9189a04640 1 -- 172.21.15.60:0/1346191663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d70e190e60 con 0x55d70e18d790 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f9189a04640 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 0x55d70e19e290 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9180011350 tx=0x7f918000fbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682ab375f62ef534 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f916d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f916d7fa640 1 -- 172.21.15.60:0/1346191663 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918001f040 con 0x55d70e18d790 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f916d7fa640 1 -- 172.21.15.60:0/1346191663 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f9180004dd0 con 0x55d70e18d790 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f918e846740 1 -- 172.21.15.60:0/1346191663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9174004440 con 0x55d70e18d790 2024-03-04T12:26:15.620 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f916d7fa640 1 -- 172.21.15.60:0/1346191663 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9180038050 con 0x55d70e18d790 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f918e846740 10 client.15855 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f918e846740 1 -- 172.21.15.60:0/1346191663 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d70e1fdae0 con 0x55d70e18d790 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f916d7fa640 1 -- 172.21.15.60:0/1346191663 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9174004440 con 0x55d70e18d790 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f916d7fa640 1 -- 172.21.15.60:0/1346191663 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9180018070 con 0x55d70e18d790 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f918e846740 10 client.15855 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f918e846740 10 client.15855 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f918e846740 20 client.15855 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f918e846740 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55d70e1fe0d0 0x55d70e2004f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f918e846740 4 client.15855 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.609+0000 7f918e846740 1 -- 172.21.15.60:0/1346191663 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d70e0764e0 con 0x55d70e1fe0d0 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.610+0000 7f9189203640 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55d70e1fe0d0 0x55d70e2004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.610+0000 7f9189203640 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55d70e1fe0d0 0x55d70e2004f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f98a7fa2815cde93 server_cookie=5e7534d247ea63d6 in_seq=0 out_seq=0 2024-03-04T12:26:15.621 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.610+0000 7f916d7fa640 10 client.15855 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.610+0000 7f916d7fa640 1 -- 172.21.15.60:0/1346191663 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d70e0764e0 con 0x55d70e1fe0d0 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.610+0000 7f916d7fa640 10 client.15855 handle_command_reply: tid=0 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.610+0000 7f918e846740 1 client.15855 shutdown 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918e846740 1 -- 172.21.15.60:0/1346191663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 msgr2=0x55d70e19e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918e846740 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 0x55d70e19e290 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9180011350 tx=0x7f918000fbd0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918e846740 1 -- 172.21.15.60:0/1346191663 shutdown_connections 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918e846740 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55d70e1fe0d0 0x55d70e2004f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918e846740 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55d70e196c90 0x55d70e190630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918e846740 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d70e18e0b0 0x55d70e1900b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918e846740 1 --2- 172.21.15.60:0/1346191663 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55d70e18d790 0x55d70e19e290 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f916d7fa640 0 client.15855 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918e846740 1 -- 172.21.15.60:0/1346191663 >> 172.21.15.60:0/1346191663 conn(0x55d70e187ec0 msgr2=0x55d70e18e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918e846740 1 -- 172.21.15.60:0/1346191663 shutdown_connections 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918e846740 1 -- 172.21.15.60:0/1346191663 wait complete. 2024-03-04T12:26:15.622 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918e846740 20 client.15855 trim_cache size 0 max 16384 2024-03-04T12:26:15.623 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:26:15.623 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "no active scrubs running", 2024-03-04T12:26:15.623 INFO:teuthology.orchestra.run.smithi060.stdout: "scrubs": {} 2024-03-04T12:26:15.623 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:26:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918bc68640 1 -- 172.21.15.60:0/3982681078 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f91700a2010 msgr2=0x7f91700a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918bc68640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f91700a2010 0x7f91700a44d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9184101c30 tx=0x7f917402c6e0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918bc68640 1 -- 172.21.15.60:0/3982681078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918410dbc0 msgr2=0x7f918419e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918bc68640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918410dbc0 0x7f918419e6c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f917c00e970 tx=0x7f917c00ee30 comp rx=0 tx=0).stop 2024-03-04T12:26:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918bc68640 1 -- 172.21.15.60:0/3982681078 shutdown_connections 2024-03-04T12:26:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918bc68640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f91700a2010 0x7f91700a44d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918bc68640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f918410dbc0 0x7f918419e6c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918bc68640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9184100a50 0x7f918419a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918bc68640 1 --2- 172.21.15.60:0/3982681078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91841000a0 0x7f9184199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.624 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918bc68640 1 -- 172.21.15.60:0/3982681078 >> 172.21.15.60:0/3982681078 conn(0x7f91840fbc70 msgr2=0x7f91840fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:15.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918bc68640 1 -- 172.21.15.60:0/3982681078 shutdown_connections 2024-03-04T12:26:15.625 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.611+0000 7f918bc68640 1 -- 172.21.15.60:0/3982681078 wait complete. 2024-03-04T12:26:15.628 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T12:26:15.629 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.726+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/3156985232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f88105500 msgr2=0x7f3f8810d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.726+0000 7f3f8fd7e640 1 --2- 172.21.15.60:0/3156985232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f88105500 0x7f3f8810d0d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f3f80009ca0 tx=0x7f3f8002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.726+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/3156985232 shutdown_connections 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.726+0000 7f3f8fd7e640 1 --2- 172.21.15.60:0/3156985232 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f88105500 0x7f3f8810d0d0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.726+0000 7f3f8fd7e640 1 --2- 172.21.15.60:0/3156985232 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f88100de0 0x7f3f88104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.726+0000 7f3f8fd7e640 1 --2- 172.21.15.60:0/3156985232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f88100430 0x7f3f88100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.726+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/3156985232 >> 172.21.15.60:0/3156985232 conn(0x7f3f880fbcb0 msgr2=0x7f3f880fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.726+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/3156985232 shutdown_connections 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.726+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/3156985232 wait complete. 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8fd7e640 1 Processor -- start 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8fd7e640 1 -- start start 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8fd7e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f88100430 0x7f3f88101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.735 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8fd7e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f88105500 0x7f3f88102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8fd7e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f88102940 0x7f3f881a5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8fd7e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f881105b0 con 0x7f3f88105500 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8fd7e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f88110430 con 0x7f3f88100430 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8fd7e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f88110730 con 0x7f3f88102940 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8d319640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f88105500 0x7f3f88102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8d319640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f88105500 0x7f3f88102400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53484/0 (socket says 172.21.15.60:53484) 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8d319640 1 -- 172.21.15.60:0/311202174 learned_addr learned my addr 172.21.15.60:0/311202174 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8db1a640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f88100430 0x7f3f88101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8e31b640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f88102940 0x7f3f881a5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8d319640 1 -- 172.21.15.60:0/311202174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f88102940 msgr2=0x7f3f881a5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8d319640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f88102940 0x7f3f881a5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8d319640 1 -- 172.21.15.60:0/311202174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f88100430 msgr2=0x7f3f88101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.736 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8d319640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f88100430 0x7f3f88101ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.727+0000 7f3f8d319640 1 -- 172.21.15.60:0/311202174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f881a6470 con 0x7f3f88105500 2024-03-04T12:26:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.728+0000 7f3f8d319640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f88105500 0x7f3f88102400 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800e9f0 tx=0x7f3f7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.728+0000 7f3f7effd640 1 -- 172.21.15.60:0/311202174 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7800ccc0 con 0x7f3f88105500 2024-03-04T12:26:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.728+0000 7f3f7effd640 1 -- 172.21.15.60:0/311202174 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f78005500 con 0x7f3f88105500 2024-03-04T12:26:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.728+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/311202174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f881a6760 con 0x7f3f88105500 2024-03-04T12:26:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.728+0000 7f3f7effd640 1 -- 172.21.15.60:0/311202174 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f78010690 con 0x7f3f88105500 2024-03-04T12:26:15.737 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.728+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/311202174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f881a6c60 con 0x7f3f88105500 2024-03-04T12:26:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.729+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/311202174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f88101900 con 0x7f3f88105500 2024-03-04T12:26:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.730+0000 7f3f7effd640 1 -- 172.21.15.60:0/311202174 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f3f780046c0 con 0x7f3f88105500 2024-03-04T12:26:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.731+0000 7f3f7effd640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f3f740a2010 0x7f3f740a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.731+0000 7f3f8db1a640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f3f740a2010 0x7f3f740a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.731+0000 7f3f7effd640 1 -- 172.21.15.60:0/311202174 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f3f780ceae0 con 0x7f3f88105500 2024-03-04T12:26:15.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.731+0000 7f3f8db1a640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f3f740a2010 0x7f3f740a44d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3f84009bd0 tx=0x7f3f84005f90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:15.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.736+0000 7f3f7effd640 1 -- 172.21.15.60:0/311202174 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3f7809b5e0 con 0x7f3f88105500 2024-03-04T12:26:15.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.954+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/311202174 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f88100de0 con 0x7f3f88105500 2024-03-04T12:26:15.961 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.955+0000 7f3f7effd640 1 -- 172.21.15.60:0/311202174 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11373 (secure 0 0 0) 0x7f3f7809aee0 con 0x7f3f88105500 2024-03-04T12:26:15.962 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:26:15.962 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:25:37.524434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:26:15.962 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 29 2024-03-04T12:26:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.957+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/311202174 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f3f740a2010 msgr2=0x7f3f740a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.957+0000 7f3f8fd7e640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f3f740a2010 0x7f3f740a44d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3f84009bd0 tx=0x7f3f84005f90 comp rx=0 tx=0).stop 2024-03-04T12:26:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.957+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/311202174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f88105500 msgr2=0x7f3f88102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.957+0000 7f3f8fd7e640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f88105500 0x7f3f88102400 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3f7800e9f0 tx=0x7f3f7800eeb0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.957+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/311202174 shutdown_connections 2024-03-04T12:26:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.957+0000 7f3f8fd7e640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f88102940 0x7f3f881a5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.957+0000 7f3f8fd7e640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f3f740a2010 0x7f3f740a44d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.957+0000 7f3f8fd7e640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f3f88105500 0x7f3f88102400 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.957+0000 7f3f8fd7e640 1 --2- 172.21.15.60:0/311202174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3f88100430 0x7f3f88101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:15.964 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.957+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/311202174 >> 172.21.15.60:0/311202174 conn(0x7f3f880fbcb0 msgr2=0x7f3f880fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:15.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.957+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/311202174 shutdown_connections 2024-03-04T12:26:15.965 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:15.957+0000 7f3f8fd7e640 1 -- 172.21.15.60:0/311202174 wait complete. 2024-03-04T12:26:15.975 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:26:15.975 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-04T12:26:16.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.074+0000 7f7fb2bda640 1 -- 172.21.15.60:0/4290723334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fac100430 msgr2=0x7f7fac100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.074+0000 7f7fb2bda640 1 --2- 172.21.15.60:0/4290723334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fac100430 0x7f7fac100810 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009900 tx=0x7f7f9c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:16.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.074+0000 7f7fb2bda640 1 -- 172.21.15.60:0/4290723334 shutdown_connections 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.074+0000 7f7fb2bda640 1 --2- 172.21.15.60:0/4290723334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fac105500 0x7f7fac10d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.074+0000 7f7fb2bda640 1 --2- 172.21.15.60:0/4290723334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fac100de0 0x7f7fac104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.074+0000 7f7fb2bda640 1 --2- 172.21.15.60:0/4290723334 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fac100430 0x7f7fac100810 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.074+0000 7f7fb2bda640 1 -- 172.21.15.60:0/4290723334 >> 172.21.15.60:0/4290723334 conn(0x7f7fac077350 msgr2=0x7f7fac077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.074+0000 7f7fb2bda640 1 -- 172.21.15.60:0/4290723334 shutdown_connections 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.074+0000 7f7fb2bda640 1 -- 172.21.15.60:0/4290723334 wait complete. 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.074+0000 7f7fb2bda640 1 Processor -- start 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.074+0000 7f7fb2bda640 1 -- start start 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fb2bda640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fac100430 0x7f7fac198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fb2bda640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fac105500 0x7f7fac198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fb2bda640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fac1994f0 0x7f7fac1a3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fb2bda640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1105c0 con 0x7f7fac105500 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fb2bda640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac110440 con 0x7f7fac1994f0 2024-03-04T12:26:16.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fb2bda640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac110740 con 0x7f7fac100430 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fabfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fac105500 0x7f7fac198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fabfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fac105500 0x7f7fac198f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53510/0 (socket says 172.21.15.60:53510) 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fabfff640 1 -- 172.21.15.60:0/412262304 learned_addr learned my addr 172.21.15.60:0/412262304 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fb1177640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fac1994f0 0x7f7fac1a3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fb0976640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fac100430 0x7f7fac198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fabfff640 1 -- 172.21.15.60:0/412262304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fac100430 msgr2=0x7f7fac198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fabfff640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fac100430 0x7f7fac198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fabfff640 1 -- 172.21.15.60:0/412262304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fac1994f0 msgr2=0x7f7fac1a3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fabfff640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fac1994f0 0x7f7fac1a3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fabfff640 1 -- 172.21.15.60:0/412262304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac1a4270 con 0x7f7fac105500 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fb1177640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fac1994f0 0x7f7fac1a3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fabfff640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fac105500 0x7f7fac198f60 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400c970 tx=0x7f7f9400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.084 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fa9ffb640 1 -- 172.21.15.60:0/412262304 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f94007bb0 con 0x7f7fac105500 2024-03-04T12:26:16.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fa9ffb640 1 -- 172.21.15.60:0/412262304 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f94007d50 con 0x7f7fac105500 2024-03-04T12:26:16.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fb2bda640 1 -- 172.21.15.60:0/412262304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fac1a4560 con 0x7f7fac105500 2024-03-04T12:26:16.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.075+0000 7f7fa9ffb640 1 -- 172.21.15.60:0/412262304 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9400f450 con 0x7f7fac105500 2024-03-04T12:26:16.085 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.076+0000 7f7fb2bda640 1 -- 172.21.15.60:0/412262304 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fac1a4ab0 con 0x7f7fac105500 2024-03-04T12:26:16.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fa9ffb640 1 -- 172.21.15.60:0/412262304 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f7f940046c0 con 0x7f7fac105500 2024-03-04T12:26:16.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fb57b0740 1 Processor -- start 2024-03-04T12:26:16.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fb57b0740 1 -- start start 2024-03-04T12:26:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fa9ffb640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f7f980a2010 0x7f7f980a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fb57b0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c369d0 0x558a44c36db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fb57b0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 0x558a44c3f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fb57b0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c3fed0 0x558a44c422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fb57b0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a44bfddc0 con 0x558a44c372f0 2024-03-04T12:26:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fb57b0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a44b99ec0 con 0x558a44c369d0 2024-03-04T12:26:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fb57b0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a44b42d30 con 0x558a44c3fed0 2024-03-04T12:26:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fa9ffb640 1 -- 172.21.15.60:0/412262304 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f7f940ce8f0 con 0x7f7fac105500 2024-03-04T12:26:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fb1177640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f7f980a2010 0x7f7f980a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fabfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 0x558a44c3f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fabfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 0x558a44c3f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53520/0 (socket says 172.21.15.60:53520) 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.078+0000 7f7fabfff640 1 -- 172.21.15.60:0/1758969962 learned_addr learned my addr 172.21.15.60:0/1758969962 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fb0976640 1 --2- 172.21.15.60:0/1758969962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c369d0 0x558a44c36db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fb1177640 1 --2- 172.21.15.60:0/1758969962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c3fed0 0x558a44c422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fabfff640 1 -- 172.21.15.60:0/1758969962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c3fed0 msgr2=0x558a44c422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fabfff640 1 --2- 172.21.15.60:0/1758969962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c3fed0 0x558a44c422c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fabfff640 1 -- 172.21.15.60:0/1758969962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c369d0 msgr2=0x558a44c36db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fabfff640 1 --2- 172.21.15.60:0/1758969962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c369d0 0x558a44c36db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fabfff640 1 -- 172.21.15.60:0/1758969962 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a448ae3f0 con 0x558a44c372f0 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fb1177640 1 --2- 172.21.15.60:0/1758969962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c3fed0 0x558a44c422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fb1177640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f7f980a2010 0x7f7f980a44d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4005fd0 tx=0x7f7fa4009900 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fabfff640 1 --2- 172.21.15.60:0/1758969962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 0x558a44c3f990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7f94007590 tx=0x7f7f9409abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d276d2e9a44fc25 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fa2ffd640 1 -- 172.21.15.60:0/1758969962 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9401d070 con 0x558a44c372f0 2024-03-04T12:26:16.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fa2ffd640 1 -- 172.21.15.60:0/1758969962 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f940a3490 con 0x558a44c372f0 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.079+0000 7f7fa2ffd640 1 -- 172.21.15.60:0/1758969962 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f940cbe00 con 0x558a44c372f0 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.080+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1758969962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 msgr2=0x558a44c3f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.080+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/1758969962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 0x558a44c3f990 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7f94007590 tx=0x7f7f9409abf0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.080+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1758969962 shutdown_connections 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.080+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/1758969962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c3fed0 0x558a44c422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.080+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/1758969962 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 0x558a44c3f990 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.080+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/1758969962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c369d0 0x558a44c36db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.080+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1758969962 >> 172.21.15.60:0/1758969962 conn(0x558a44c31100 msgr2=0x558a44c314e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.080+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1758969962 shutdown_connections 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.080+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1758969962 wait complete. 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.080+0000 7f7fb57b0740 1 -- start start 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.080+0000 7f7fb57b0740 1 Processor -- start 2024-03-04T12:26:16.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb57b0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c369d0 0x558a44c3cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb57b0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c372f0 0x558a44c3d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb57b0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c3fed0 0x558a44c3d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb57b0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a44b384a0 con 0x558a44c3fed0 2024-03-04T12:26:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb57b0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a44b95f60 con 0x558a44c372f0 2024-03-04T12:26:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb57b0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x558a44c42cc0 con 0x558a44c369d0 2024-03-04T12:26:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb57b0740 10 client.0 _ll_register_callbacks cb 0x558a44b941c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb1177640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c3fed0 0x558a44c3d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb1177640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c3fed0 0x558a44c3d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53528/0 (socket says 172.21.15.60:53528) 2024-03-04T12:26:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb1177640 1 -- 172.21.15.60:0/3147753477 learned_addr learned my addr 172.21.15.60:0/3147753477 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fabfff640 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c372f0 0x558a44c3d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb0976640 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c369d0 0x558a44c3cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb1177640 1 -- 172.21.15.60:0/3147753477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c369d0 msgr2=0x558a44c3cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb1177640 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c369d0 0x558a44c3cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb1177640 1 -- 172.21.15.60:0/3147753477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c372f0 msgr2=0x558a44c3d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb1177640 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c372f0 0x558a44c3d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb1177640 1 -- 172.21.15.60:0/3147753477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a449787d0 con 0x558a44c3fed0 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb1177640 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c3fed0 0x558a44c3d870 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40138f0 tx=0x7f7fa4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5abdffab74c20dc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7f8bfff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7f8bfff640 1 -- 172.21.15.60:0/3147753477 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa400fcf0 con 0x558a44c3fed0 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7f8bfff640 1 -- 172.21.15.60:0/3147753477 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f7fa4020420 con 0x558a44c3fed0 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.081+0000 7f7fb57b0740 1 -- 172.21.15.60:0/3147753477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7f940a3490 con 0x558a44c3fed0 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.082+0000 7f7f8bfff640 1 -- 172.21.15.60:0/3147753477 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fa4030050 con 0x558a44c3fed0 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.082+0000 7f7fb57b0740 10 client.15873 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.082+0000 7f7fb57b0740 1 -- 172.21.15.60:0/3147753477 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x558a44c4eca0 con 0x558a44c3fed0 2024-03-04T12:26:16.096 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.082+0000 7f7f8bfff640 1 -- 172.21.15.60:0/3147753477 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f940a3490 con 0x558a44c3fed0 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.082+0000 7f7f8bfff640 1 -- 172.21.15.60:0/3147753477 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7fa400f040 con 0x558a44c3fed0 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.082+0000 7f7fb57b0740 10 client.15873 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.082+0000 7f7fb57b0740 10 client.15873 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.082+0000 7f7fb57b0740 20 client.15873 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.082+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x558a44c50160 0x558a44c52580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.082+0000 7f7fb57b0740 4 client.15873 mds_command: new command op to 24394 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.082+0000 7f7fb57b0740 1 -- 172.21.15.60:0/3147753477 --> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558a44ba7400 con 0x558a44c50160 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.082+0000 7f7fb0976640 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x558a44c50160 0x558a44c52580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7fb0976640 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x558a44c50160 0x558a44c52580 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=50fc3c247bb169b8 server_cookie=35f8193c8ea1fba5 in_seq=0 out_seq=0 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7f8bfff640 10 client.15873 ms_handle_connect on v2:172.21.15.86:6832/777854669 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7f8bfff640 1 -- 172.21.15.60:0/3147753477 <== mds.4 v2:172.21.15.86:6832/777854669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558a44ba7400 con 0x558a44c50160 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7f8bfff640 10 client.15873 handle_command_reply: tid=0 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7fb57b0740 1 client.15873 shutdown 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7fb57b0740 1 -- 172.21.15.60:0/3147753477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c3fed0 msgr2=0x558a44c3d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c3fed0 0x558a44c3d870 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40138f0 tx=0x7f7fa4004ab0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7fb57b0740 1 -- 172.21.15.60:0/3147753477 shutdown_connections 2024-03-04T12:26:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c3fed0 0x558a44c3d870 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x558a44c50160 0x558a44c52580 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c372f0 0x558a44c3d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/3147753477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c369d0 0x558a44c3cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7f8bfff640 0 client.15873 ms_handle_reset on v2:172.21.15.86:6832/777854669 2024-03-04T12:26:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.083+0000 7f7fb57b0740 1 -- 172.21.15.60:0/3147753477 >> 172.21.15.60:0/3147753477 conn(0x558a44c31100 msgr2=0x558a44c32e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.084+0000 7f7fb57b0740 1 -- 172.21.15.60:0/3147753477 shutdown_connections 2024-03-04T12:26:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.084+0000 7f7fb57b0740 1 -- 172.21.15.60:0/3147753477 wait complete. 2024-03-04T12:26:16.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.084+0000 7f7fb57b0740 20 client.15873 trim_cache size 0 max 16384 2024-03-04T12:26:16.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.095+0000 7f7fb57b0740 1 Processor -- start 2024-03-04T12:26:16.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.095+0000 7f7fb57b0740 1 -- start start 2024-03-04T12:26:16.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.095+0000 7f7fb57b0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c369d0 0x558a44c3f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.095+0000 7f7fb57b0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 0x558a44c472e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.095+0000 7f7fb57b0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c3fed0 0x558a44c3a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.095+0000 7f7fb57b0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1ae530 con 0x558a44c372f0 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.095+0000 7f7fb57b0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1afeb0 con 0x558a44c369d0 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.095+0000 7f7fb57b0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1aeb90 con 0x558a44c3fed0 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fabfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 0x558a44c472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fabfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 0x558a44c472e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53538/0 (socket says 172.21.15.60:53538) 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fabfff640 1 -- 172.21.15.60:0/2387695261 learned_addr learned my addr 172.21.15.60:0/2387695261 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fb1177640 1 --2- 172.21.15.60:0/2387695261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c3fed0 0x558a44c3a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fb0976640 1 --2- 172.21.15.60:0/2387695261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c369d0 0x558a44c3f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fabfff640 1 -- 172.21.15.60:0/2387695261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c3fed0 msgr2=0x558a44c3a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fabfff640 1 --2- 172.21.15.60:0/2387695261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c3fed0 0x558a44c3a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fabfff640 1 -- 172.21.15.60:0/2387695261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c369d0 msgr2=0x558a44c3f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fabfff640 1 --2- 172.21.15.60:0/2387695261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c369d0 0x558a44c3f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fabfff640 1 -- 172.21.15.60:0/2387695261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558a44c38a00 con 0x558a44c372f0 2024-03-04T12:26:16.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fb0976640 1 --2- 172.21.15.60:0/2387695261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c369d0 0x558a44c3f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fabfff640 1 --2- 172.21.15.60:0/2387695261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 0x558a44c472e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f7f94098fa0 tx=0x7f7f9409ae30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41ace6dc767cf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fa27fc640 1 -- 172.21.15.60:0/2387695261 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9401d040 con 0x558a44c372f0 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fa27fc640 1 -- 172.21.15.60:0/2387695261 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f940990b0 con 0x558a44c372f0 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.096+0000 7f7fa27fc640 1 -- 172.21.15.60:0/2387695261 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9405fc10 con 0x558a44c372f0 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.097+0000 7f7fb57b0740 1 -- 172.21.15.60:0/2387695261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 msgr2=0x558a44c472e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.097+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/2387695261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 0x558a44c472e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f7f94098fa0 tx=0x7f7f9409ae30 comp rx=0 tx=0).stop 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.097+0000 7f7fb57b0740 1 -- 172.21.15.60:0/2387695261 shutdown_connections 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.097+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/2387695261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c3fed0 0x558a44c3a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.097+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/2387695261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c372f0 0x558a44c472e0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.097+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/2387695261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c369d0 0x558a44c3f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.097+0000 7f7fb57b0740 1 -- 172.21.15.60:0/2387695261 >> 172.21.15.60:0/2387695261 conn(0x558a44c31100 msgr2=0x558a44c56ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.097+0000 7f7fb57b0740 1 -- 172.21.15.60:0/2387695261 shutdown_connections 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.097+0000 7f7fb57b0740 1 -- 172.21.15.60:0/2387695261 wait complete. 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.097+0000 7f7fb57b0740 1 -- start start 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.097+0000 7f7fb57b0740 1 Processor -- start 2024-03-04T12:26:16.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb57b0740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c369d0 0x558a44ca4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb57b0740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c372f0 0x558a44ca4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb57b0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c3fed0 0x558a44c9e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb57b0740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1af850 con 0x558a44c369d0 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb57b0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1af1f0 con 0x558a44c3fed0 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb57b0740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fac1ac7d0 con 0x558a44c372f0 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb57b0740 10 client.0 _ll_register_callbacks cb 0x7f7fa4009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb0976640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c369d0 0x558a44ca4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb0976640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c369d0 0x558a44ca4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53552/0 (socket says 172.21.15.60:53552) 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb0976640 1 -- 172.21.15.60:0/1693279972 learned_addr learned my addr 172.21.15.60:0/1693279972 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fabfff640 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c372f0 0x558a44ca4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb1177640 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c3fed0 0x558a44c9e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb0976640 1 -- 172.21.15.60:0/1693279972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c372f0 msgr2=0x558a44ca4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb0976640 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c372f0 0x558a44ca4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb0976640 1 -- 172.21.15.60:0/1693279972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c3fed0 msgr2=0x558a44c9e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb0976640 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c3fed0 0x558a44c9e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb0976640 1 -- 172.21.15.60:0/1693279972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558a44c9edc0 con 0x558a44c369d0 2024-03-04T12:26:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fabfff640 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c372f0 0x558a44ca4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb0976640 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c369d0 0x558a44ca4220 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009900 tx=0x7f7f9c02ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ea4a7927d6bb29 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7f8bfff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7f8bfff640 1 -- 172.21.15.60:0/1693279972 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c049070 con 0x558a44c369d0 2024-03-04T12:26:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7f8bfff640 1 -- 172.21.15.60:0/1693279972 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f7f9c004040 con 0x558a44c369d0 2024-03-04T12:26:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1693279972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558a44ba1750 con 0x558a44c369d0 2024-03-04T12:26:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7f8bfff640 1 -- 172.21.15.60:0/1693279972 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f9c054050 con 0x558a44c369d0 2024-03-04T12:26:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.098+0000 7f7fb57b0740 10 client.15879 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.099+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1693279972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x558a44cab980 con 0x558a44c369d0 2024-03-04T12:26:16.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.099+0000 7f7f8bfff640 1 -- 172.21.15.60:0/1693279972 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558a44ba1750 con 0x558a44c369d0 2024-03-04T12:26:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.099+0000 7f7f8bfff640 1 -- 172.21.15.60:0/1693279972 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7f9c005160 con 0x558a44c369d0 2024-03-04T12:26:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.099+0000 7f7fb57b0740 10 client.15879 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.099+0000 7f7fb57b0740 10 client.15879 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-04T12:26:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.099+0000 7f7fb57b0740 20 client.15879 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.099+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x558a44cac030 0x558a44cae450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.099+0000 7f7fb57b0740 4 client.15879 mds_command: new command op to 24394 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:26:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.099+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1693279972 --> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558a44b1f640 con 0x558a44cac030 2024-03-04T12:26:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.099+0000 7f7fabfff640 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x558a44cac030 0x558a44cae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.099+0000 7f7fabfff640 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x558a44cac030 0x558a44cae450 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=86420d2d1bd82b44 server_cookie=898e0b7872f052f9 in_seq=0 out_seq=0 2024-03-04T12:26:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.099+0000 7f7f8bfff640 10 client.15879 ms_handle_connect on v2:172.21.15.86:6832/777854669 2024-03-04T12:26:16.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7f8bfff640 1 -- 172.21.15.60:0/1693279972 <== mds.4 v2:172.21.15.86:6832/777854669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558a44b1f640 con 0x558a44cac030 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7f8bfff640 10 client.15879 handle_command_reply: tid=0 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb57b0740 1 client.15879 shutdown 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1693279972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c369d0 msgr2=0x558a44ca4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c369d0 0x558a44ca4220 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c009900 tx=0x7f7f9c02ccd0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1693279972 shutdown_connections 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x558a44cac030 0x558a44cae450 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558a44c3fed0 0x558a44c9e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558a44c372f0 0x558a44ca4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7f8bfff640 0 client.15879 ms_handle_reset on v2:172.21.15.86:6832/777854669 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb57b0740 1 --2- 172.21.15.60:0/1693279972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x558a44c369d0 0x558a44ca4220 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1693279972 >> 172.21.15.60:0/1693279972 conn(0x558a44c31100 msgr2=0x558a44c3b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1693279972 shutdown_connections 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb57b0740 1 -- 172.21.15.60:0/1693279972 wait complete. 2024-03-04T12:26:16.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb57b0740 20 client.15879 trim_cache size 0 max 16384 2024-03-04T12:26:16.114 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:26:16.114 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb2bda640 1 -- 172.21.15.60:0/412262304 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f7f980a2010 msgr2=0x7f7f980a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.100+0000 7f7fb2bda640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f7f980a2010 0x7f7f980a44d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4005fd0 tx=0x7f7fa4009900 comp rx=0 tx=0).stop 2024-03-04T12:26:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.101+0000 7f7fb2bda640 1 -- 172.21.15.60:0/412262304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fac105500 msgr2=0x7f7fac198f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.101+0000 7f7fb2bda640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fac105500 0x7f7fac198f60 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f7f9400c970 tx=0x7f7f9400ce30 comp rx=0 tx=0).stop 2024-03-04T12:26:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.101+0000 7f7fb2bda640 1 -- 172.21.15.60:0/412262304 shutdown_connections 2024-03-04T12:26:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.101+0000 7f7fb2bda640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7fac1994f0 0x7f7fac1a3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.101+0000 7f7fb2bda640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f7f980a2010 0x7f7f980a44d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.101+0000 7f7fb2bda640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7fac105500 0x7f7fac198f60 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.101+0000 7f7fb2bda640 1 --2- 172.21.15.60:0/412262304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fac100430 0x7f7fac198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.101+0000 7f7fb2bda640 1 -- 172.21.15.60:0/412262304 >> 172.21.15.60:0/412262304 conn(0x7f7fac077350 msgr2=0x7f7fac0fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.101+0000 7f7fb2bda640 1 -- 172.21.15.60:0/412262304 shutdown_connections 2024-03-04T12:26:16.115 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.101+0000 7f7fb2bda640 1 -- 172.21.15.60:0/412262304 wait complete. 2024-03-04T12:26:16.117 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:26:16.118 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-04T12:26:16.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.216+0000 7f0f60a9a640 1 -- 172.21.15.60:0/2007025349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f5c102e70 msgr2=0x7f0f5c10b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.216+0000 7f0f60a9a640 1 --2- 172.21.15.60:0/2007025349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f5c102e70 0x7f0f5c10b350 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0f50009900 tx=0x7f0f5002bcd0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.216+0000 7f0f60a9a640 1 -- 172.21.15.60:0/2007025349 shutdown_connections 2024-03-04T12:26:16.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.216+0000 7f0f60a9a640 1 --2- 172.21.15.60:0/2007025349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5c10b920 0x7f0f5c10dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.216+0000 7f0f60a9a640 1 --2- 172.21.15.60:0/2007025349 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f5c102e70 0x7f0f5c10b350 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.216+0000 7f0f60a9a640 1 --2- 172.21.15.60:0/2007025349 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f5c102550 0x7f0f5c102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.216+0000 7f0f60a9a640 1 -- 172.21.15.60:0/2007025349 >> 172.21.15.60:0/2007025349 conn(0x7f0f5c0773e0 msgr2=0x7f0f5c0777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.217+0000 7f0f60a9a640 1 -- 172.21.15.60:0/2007025349 shutdown_connections 2024-03-04T12:26:16.224 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.217+0000 7f0f60a9a640 1 -- 172.21.15.60:0/2007025349 wait complete. 2024-03-04T12:26:16.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.217+0000 7f0f60a9a640 1 Processor -- start 2024-03-04T12:26:16.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.217+0000 7f0f60a9a640 1 -- start start 2024-03-04T12:26:16.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.217+0000 7f0f60a9a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f5c102550 0x7f0f5c1001a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.217+0000 7f0f60a9a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5c102e70 0x7f0f5c1006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.217+0000 7f0f60a9a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f5c10b920 0x7f0f5c100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.217+0000 7f0f60a9a640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1116a0 con 0x7f0f5c10b920 2024-03-04T12:26:16.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.217+0000 7f0f60a9a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c111520 con 0x7f0f5c102550 2024-03-04T12:26:16.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.217+0000 7f0f60a9a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c111820 con 0x7f0f5c102e70 2024-03-04T12:26:16.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f5ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f5c10b920 0x7f0f5c100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.226 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f5ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f5c10b920 0x7f0f5c100c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53570/0 (socket says 172.21.15.60:53570) 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/290589390 learned_addr learned my addr 172.21.15.60:0/290589390 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f59d9b640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5c102e70 0x7f0f5c1006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f5a59c640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f5c102550 0x7f0f5c1001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/290589390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5c102e70 msgr2=0x7f0f5c1006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5c102e70 0x7f0f5c1006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/290589390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f5c102550 msgr2=0x7f0f5c1001a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f5c102550 0x7f0f5c1001a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/290589390 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c1a60c0 con 0x7f0f5c10b920 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f5a59c640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f5c102550 0x7f0f5c1001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f5c10b920 0x7f0f5c100c20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c00e960 tx=0x7f0f4c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f4b7fe640 1 -- 172.21.15.60:0/290589390 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c004d20 con 0x7f0f5c10b920 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f60a9a640 1 -- 172.21.15.60:0/290589390 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c1a63b0 con 0x7f0f5c10b920 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f4b7fe640 1 -- 172.21.15.60:0/290589390 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f4c005020 con 0x7f0f5c10b920 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f4b7fe640 1 -- 172.21.15.60:0/290589390 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c0116d0 con 0x7f0f5c10b920 2024-03-04T12:26:16.227 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.218+0000 7f0f60a9a640 1 -- 172.21.15.60:0/290589390 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c1a6900 con 0x7f0f5c10b920 2024-03-04T12:26:16.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.220+0000 7f0f4b7fe640 1 -- 172.21.15.60:0/290589390 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f0f4c0054d0 con 0x7f0f5c10b920 2024-03-04T12:26:16.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f63683740 1 Processor -- start 2024-03-04T12:26:16.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f63683740 1 -- start start 2024-03-04T12:26:16.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f4b7fe640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0f440a2010 0x7f0f440a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f63683740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4509d0 0x555d1b450db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f63683740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4512f0 0x555d1b459990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f63683740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b45c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f63683740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1b417dc0 con 0x555d1b459ed0 2024-03-04T12:26:16.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f63683740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1b3b3ec0 con 0x555d1b4512f0 2024-03-04T12:26:16.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f63683740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1b35cd30 con 0x555d1b4509d0 2024-03-04T12:26:16.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f59d9b640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0f440a2010 0x7f0f440a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f5ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b45c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f4b7fe640 1 -- 172.21.15.60:0/290589390 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f0f4c0ce9e0 con 0x7f0f5c10b920 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f5ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b45c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53582/0 (socket says 172.21.15.60:53582) 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/3874412376 learned_addr learned my addr 172.21.15.60:0/3874412376 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f5a59c640 1 --2- 172.21.15.60:0/3874412376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4509d0 0x555d1b450db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.221+0000 7f0f59d9b640 1 --2- 172.21.15.60:0/3874412376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4512f0 0x555d1b459990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.222+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/3874412376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4509d0 msgr2=0x555d1b450db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.222+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/3874412376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4509d0 0x555d1b450db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.222+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/3874412376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4512f0 msgr2=0x555d1b459990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.222+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/3874412376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4512f0 0x555d1b459990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.222+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/3874412376 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d1b0c83f0 con 0x555d1b459ed0 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.222+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/3874412376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b45c2c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c0977b0 tx=0x7f0f4c09a920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d827139bfa57ed4d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.222+0000 7f0f59d9b640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0f440a2010 0x7f0f440a44d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c101c30 tx=0x7f0f5002cbe0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.222+0000 7f0f48ff9640 1 -- 172.21.15.60:0/3874412376 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c01f040 con 0x555d1b459ed0 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.222+0000 7f0f48ff9640 1 -- 172.21.15.60:0/3874412376 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f4c09b4e0 con 0x555d1b459ed0 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.222+0000 7f0f48ff9640 1 -- 172.21.15.60:0/3874412376 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c097450 con 0x555d1b459ed0 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 -- 172.21.15.60:0/3874412376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 msgr2=0x555d1b45c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 --2- 172.21.15.60:0/3874412376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b45c2c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c0977b0 tx=0x7f0f4c09a920 comp rx=0 tx=0).stop 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 -- 172.21.15.60:0/3874412376 shutdown_connections 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 --2- 172.21.15.60:0/3874412376 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b45c2c0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 --2- 172.21.15.60:0/3874412376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4512f0 0x555d1b459990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 --2- 172.21.15.60:0/3874412376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4509d0 0x555d1b450db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 -- 172.21.15.60:0/3874412376 >> 172.21.15.60:0/3874412376 conn(0x555d1b44b100 msgr2=0x555d1b44b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 -- 172.21.15.60:0/3874412376 shutdown_connections 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 -- 172.21.15.60:0/3874412376 wait complete. 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 -- start start 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 Processor -- start 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4509d0 0x555d1b456f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.237 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4512f0 0x555d1b457460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b4579a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1b3524a0 con 0x555d1b459ed0 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1b3aff60 con 0x555d1b4512f0 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d1b45cc60 con 0x555d1b4509d0 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f63683740 10 client.0 _ll_register_callbacks cb 0x555d1b3ae1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f5ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b4579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.223+0000 7f0f5ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b4579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53598/0 (socket says 172.21.15.60:53598) 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/847540864 learned_addr learned my addr 172.21.15.60:0/847540864 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f5a59c640 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4509d0 0x555d1b456f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f59d9b640 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4512f0 0x555d1b457460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/847540864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4509d0 msgr2=0x555d1b456f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4509d0 0x555d1b456f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/847540864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4512f0 msgr2=0x555d1b457460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4512f0 0x555d1b457460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/847540864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d1b1927d0 con 0x555d1b459ed0 2024-03-04T12:26:16.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f59d9b640 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4512f0 0x555d1b457460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b4579a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c099090 tx=0x7f0f4c06c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6588f5566f5eda1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f41ffb640 1 -- 172.21.15.60:0/847540864 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c01f040 con 0x555d1b459ed0 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f41ffb640 1 -- 172.21.15.60:0/847540864 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f0f4c01ac60 con 0x555d1b459ed0 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f63683740 1 -- 172.21.15.60:0/847540864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f4c09b4e0 con 0x555d1b459ed0 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f41ffb640 1 -- 172.21.15.60:0/847540864 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f4c01ae80 con 0x555d1b459ed0 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f63683740 10 client.15891 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f63683740 1 -- 172.21.15.60:0/847540864 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x555d1b468da0 con 0x555d1b459ed0 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.224+0000 7f0f41ffb640 1 -- 172.21.15.60:0/847540864 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f4c09b4e0 con 0x555d1b459ed0 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.225+0000 7f0f41ffb640 1 -- 172.21.15.60:0/847540864 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0f4c00f040 con 0x555d1b459ed0 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.225+0000 7f0f63683740 10 client.15891 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.225+0000 7f0f63683740 10 client.15891 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.225+0000 7f0f63683740 20 client.15891 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.225+0000 7f0f63683740 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x555d1b46a0f0 0x555d1b46c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.225+0000 7f0f63683740 4 client.15891 mds_command: new command op to 24407 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:16.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.225+0000 7f0f63683740 1 -- 172.21.15.60:0/847540864 --> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d1b3c1400 con 0x555d1b46a0f0 2024-03-04T12:26:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.225+0000 7f0f5a59c640 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x555d1b46a0f0 0x555d1b46c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.225+0000 7f0f41ffb640 10 client.15891 ms_handle_connect on v2:172.21.15.63:6832/2270964219 2024-03-04T12:26:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.225+0000 7f0f5a59c640 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x555d1b46a0f0 0x555d1b46c4e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b4815a925ebfec31 server_cookie=b22427417e8ca15a in_seq=0 out_seq=0 2024-03-04T12:26:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f41ffb640 1 -- 172.21.15.60:0/847540864 <== mds.3 v2:172.21.15.63:6832/2270964219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555d1b3c1400 con 0x555d1b46a0f0 2024-03-04T12:26:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f41ffb640 10 client.15891 handle_command_reply: tid=0 2024-03-04T12:26:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f63683740 1 client.15891 shutdown 2024-03-04T12:26:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f63683740 1 -- 172.21.15.60:0/847540864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 msgr2=0x555d1b4579a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f63683740 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b4579a0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c099090 tx=0x7f0f4c06c940 comp rx=0 tx=0).stop 2024-03-04T12:26:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f63683740 1 -- 172.21.15.60:0/847540864 shutdown_connections 2024-03-04T12:26:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f63683740 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x555d1b46a0f0 0x555d1b46c4e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f63683740 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b4579a0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f63683740 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4512f0 0x555d1b457460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f63683740 1 --2- 172.21.15.60:0/847540864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4509d0 0x555d1b456f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f41ffb640 0 client.15891 ms_handle_reset on v2:172.21.15.63:6832/2270964219 2024-03-04T12:26:16.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f63683740 1 -- 172.21.15.60:0/847540864 >> 172.21.15.60:0/847540864 conn(0x555d1b44b100 msgr2=0x555d1b44ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f63683740 1 -- 172.21.15.60:0/847540864 shutdown_connections 2024-03-04T12:26:16.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.226+0000 7f0f63683740 1 -- 172.21.15.60:0/847540864 wait complete. 2024-03-04T12:26:16.241 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.227+0000 7f0f63683740 20 client.15891 trim_cache size 0 max 16384 2024-03-04T12:26:16.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.238+0000 7f0f63683740 1 Processor -- start 2024-03-04T12:26:16.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.238+0000 7f0f63683740 1 -- start start 2024-03-04T12:26:16.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.238+0000 7f0f63683740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4509d0 0x555d1b4595e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.238+0000 7f0f63683740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b4512f0 0x555d1b455e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.238+0000 7f0f63683740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b459ed0 0x555d1b4541a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.238+0000 7f0f63683740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1b07a0 con 0x555d1b4512f0 2024-03-04T12:26:16.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.238+0000 7f0f63683740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1b2120 con 0x555d1b4509d0 2024-03-04T12:26:16.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.238+0000 7f0f63683740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1b0e00 con 0x555d1b459ed0 2024-03-04T12:26:16.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f59d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b4512f0 0x555d1b455e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f59d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b4512f0 0x555d1b455e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53608/0 (socket says 172.21.15.60:53608) 2024-03-04T12:26:16.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f59d9b640 1 -- 172.21.15.60:0/1751764570 learned_addr learned my addr 172.21.15.60:0/1751764570 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f5a59c640 1 --2- 172.21.15.60:0/1751764570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4509d0 0x555d1b4595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f59d9b640 1 -- 172.21.15.60:0/1751764570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b459ed0 msgr2=0x555d1b4541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/1751764570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b459ed0 0x555d1b4541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f59d9b640 1 --2- 172.21.15.60:0/1751764570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b459ed0 0x555d1b4541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f59d9b640 1 -- 172.21.15.60:0/1751764570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4509d0 msgr2=0x555d1b4595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f59d9b640 1 --2- 172.21.15.60:0/1751764570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4509d0 0x555d1b4595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f59d9b640 1 -- 172.21.15.60:0/1751764570 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d1b446b30 con 0x555d1b4512f0 2024-03-04T12:26:16.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f5a59c640 1 --2- 172.21.15.60:0/1751764570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4509d0 0x555d1b4595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:16.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/1751764570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b459ed0 0x555d1b4541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f59d9b640 1 --2- 172.21.15.60:0/1751764570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b4512f0 0x555d1b455e60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0f500096c0 tx=0x7f0f50044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5289bd07d78d1793 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f43fff640 1 -- 172.21.15.60:0/1751764570 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f50051070 con 0x555d1b4512f0 2024-03-04T12:26:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f43fff640 1 -- 172.21.15.60:0/1751764570 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f500046c0 con 0x555d1b4512f0 2024-03-04T12:26:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.239+0000 7f0f43fff640 1 -- 172.21.15.60:0/1751764570 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f50048070 con 0x555d1b4512f0 2024-03-04T12:26:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 -- 172.21.15.60:0/1751764570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b4512f0 msgr2=0x555d1b455e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 --2- 172.21.15.60:0/1751764570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b4512f0 0x555d1b455e60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0f500096c0 tx=0x7f0f50044000 comp rx=0 tx=0).stop 2024-03-04T12:26:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 -- 172.21.15.60:0/1751764570 shutdown_connections 2024-03-04T12:26:16.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 --2- 172.21.15.60:0/1751764570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b459ed0 0x555d1b4541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 --2- 172.21.15.60:0/1751764570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b4512f0 0x555d1b455e60 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 --2- 172.21.15.60:0/1751764570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4509d0 0x555d1b4595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 -- 172.21.15.60:0/1751764570 >> 172.21.15.60:0/1751764570 conn(0x555d1b44b100 msgr2=0x555d1b44a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 -- 172.21.15.60:0/1751764570 shutdown_connections 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 -- 172.21.15.60:0/1751764570 wait complete. 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 -- start start 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 Processor -- start 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4509d0 0x555d1b46d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4512f0 0x555d1b46da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b4c01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1b1ac0 con 0x555d1b459ed0 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1b1460 con 0x555d1b4509d0 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c1aea40 con 0x555d1b4512f0 2024-03-04T12:26:16.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.240+0000 7f0f63683740 10 client.0 _ll_register_callbacks cb 0x7f0f4c009270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f5ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b4c01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f5ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b4c01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53624/0 (socket says 172.21.15.60:53624) 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/4274811033 learned_addr learned my addr 172.21.15.60:0/4274811033 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f59d9b640 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4512f0 0x555d1b46da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f5a59c640 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4509d0 0x555d1b46d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/4274811033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4512f0 msgr2=0x555d1b46da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4512f0 0x555d1b46da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/4274811033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4509d0 msgr2=0x555d1b46d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4509d0 0x555d1b46d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f5ad9d640 1 -- 172.21.15.60:0/4274811033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d1b4c0710 con 0x555d1b459ed0 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f59d9b640 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4512f0 0x555d1b46da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f5ad9d640 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b4c01d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c09ad70 tx=0x7f0f4c0cc1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d5324ce06062cd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f41ffb640 1 -- 172.21.15.60:0/4274811033 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c01f040 con 0x555d1b459ed0 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f41ffb640 1 -- 172.21.15.60:0/4274811033 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f0f4c066120 con 0x555d1b459ed0 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f63683740 1 -- 172.21.15.60:0/4274811033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f500046c0 con 0x555d1b459ed0 2024-03-04T12:26:16.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f41ffb640 1 -- 172.21.15.60:0/4274811033 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f4c09a740 con 0x555d1b459ed0 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f63683740 10 client.15897 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f63683740 1 -- 172.21.15.60:0/4274811033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x555d1b4c0710 con 0x555d1b459ed0 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.241+0000 7f0f41ffb640 1 -- 172.21.15.60:0/4274811033 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f500046c0 con 0x555d1b459ed0 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f41ffb640 1 -- 172.21.15.60:0/4274811033 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0f4c0b3b10 con 0x555d1b459ed0 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f63683740 10 client.15897 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f63683740 10 client.15897 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f63683740 20 client.15897 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f63683740 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x555d1b4c1b20 0x555d1b4c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f63683740 4 client.15897 mds_command: new command op to 24407 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f63683740 1 -- 172.21.15.60:0/4274811033 --> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555d1b3bb750 con 0x555d1b4c1b20 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f5a59c640 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x555d1b4c1b20 0x555d1b4c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f5a59c640 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x555d1b4c1b20 0x555d1b4c3f10 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=926aea5608d42097 server_cookie=f4513995aff6341b in_seq=0 out_seq=0 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f41ffb640 10 client.15897 ms_handle_connect on v2:172.21.15.63:6832/2270964219 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f41ffb640 1 -- 172.21.15.60:0/4274811033 <== mds.3 v2:172.21.15.63:6832/2270964219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555d1b3bb750 con 0x555d1b4c1b20 2024-03-04T12:26:16.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f41ffb640 10 client.15897 handle_command_reply: tid=0 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f63683740 1 client.15897 shutdown 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.242+0000 7f0f63683740 1 -- 172.21.15.60:0/4274811033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 msgr2=0x555d1b4c01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f63683740 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b4c01d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c09ad70 tx=0x7f0f4c0cc1c0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f63683740 1 -- 172.21.15.60:0/4274811033 shutdown_connections 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f63683740 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x555d1b4c1b20 0x555d1b4c3f10 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f63683740 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x555d1b459ed0 0x555d1b4c01d0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f63683740 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555d1b4512f0 0x555d1b46da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f63683740 1 --2- 172.21.15.60:0/4274811033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d1b4509d0 0x555d1b46d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f41ffb640 0 client.15897 ms_handle_reset on v2:172.21.15.63:6832/2270964219 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f63683740 1 -- 172.21.15.60:0/4274811033 >> 172.21.15.60:0/4274811033 conn(0x555d1b44b100 msgr2=0x555d1b464eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f63683740 1 -- 172.21.15.60:0/4274811033 shutdown_connections 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f63683740 1 -- 172.21.15.60:0/4274811033 wait complete. 2024-03-04T12:26:16.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f63683740 20 client.15897 trim_cache size 0 max 16384 2024-03-04T12:26:16.257 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:26:16.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f60a9a640 1 -- 172.21.15.60:0/290589390 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0f440a2010 msgr2=0x7f0f440a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f60a9a640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0f440a2010 0x7f0f440a44d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c101c30 tx=0x7f0f5002cbe0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f60a9a640 1 -- 172.21.15.60:0/290589390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f5c10b920 msgr2=0x7f0f5c100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f60a9a640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f5c10b920 0x7f0f5c100c20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c00e960 tx=0x7f0f4c00ee20 comp rx=0 tx=0).stop 2024-03-04T12:26:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f60a9a640 1 -- 172.21.15.60:0/290589390 shutdown_connections 2024-03-04T12:26:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f60a9a640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0f440a2010 0x7f0f440a44d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f60a9a640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0f5c10b920 0x7f0f5c100c20 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f60a9a640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5c102e70 0x7f0f5c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f60a9a640 1 --2- 172.21.15.60:0/290589390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f5c102550 0x7f0f5c1001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f60a9a640 1 -- 172.21.15.60:0/290589390 >> 172.21.15.60:0/290589390 conn(0x7f0f5c0773e0 msgr2=0x7f0f5c0fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f60a9a640 1 -- 172.21.15.60:0/290589390 shutdown_connections 2024-03-04T12:26:16.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.243+0000 7f0f60a9a640 1 -- 172.21.15.60:0/290589390 wait complete. 2024-03-04T12:26:16.259 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:16 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/311202174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:26:16.260 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:26:16.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-04T12:26:16.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:16 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/311202174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:26:16.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:16 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/311202174' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:26:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.361+0000 7f40b399e640 1 -- 172.21.15.60:0/3149120499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40ac102ec0 msgr2=0x7f40ac10b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.361+0000 7f40b399e640 1 --2- 172.21.15.60:0/3149120499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40ac102ec0 0x7f40ac10b3a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f40a0009900 tx=0x7f40a002bcd0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.361+0000 7f40b399e640 1 -- 172.21.15.60:0/3149120499 shutdown_connections 2024-03-04T12:26:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.361+0000 7f40b399e640 1 --2- 172.21.15.60:0/3149120499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40ac10ba10 0x7f40ac10de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.361+0000 7f40b399e640 1 --2- 172.21.15.60:0/3149120499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40ac102ec0 0x7f40ac10b3a0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.361+0000 7f40b399e640 1 --2- 172.21.15.60:0/3149120499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f40ac1025a0 0x7f40ac102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.370 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.361+0000 7f40b399e640 1 -- 172.21.15.60:0/3149120499 >> 172.21.15.60:0/3149120499 conn(0x7f40ac0773a0 msgr2=0x7f40ac0777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.361+0000 7f40b399e640 1 -- 172.21.15.60:0/3149120499 shutdown_connections 2024-03-04T12:26:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.361+0000 7f40b399e640 1 -- 172.21.15.60:0/3149120499 wait complete. 2024-03-04T12:26:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b399e640 1 Processor -- start 2024-03-04T12:26:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b399e640 1 -- start start 2024-03-04T12:26:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b399e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40ac1025a0 0x7f40ac1001f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b399e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40ac102ec0 0x7f40ac100730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b399e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f40ac10ba10 0x7f40ac100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b399e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac1117c0 con 0x7f40ac1025a0 2024-03-04T12:26:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b399e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac111640 con 0x7f40ac10ba10 2024-03-04T12:26:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b399e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac111940 con 0x7f40ac102ec0 2024-03-04T12:26:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b173a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40ac1025a0 0x7f40ac1001f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.371 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b173a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40ac1025a0 0x7f40ac1001f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53636/0 (socket says 172.21.15.60:53636) 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b173a640 1 -- 172.21.15.60:0/404142813 learned_addr learned my addr 172.21.15.60:0/404142813 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b0f39640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40ac102ec0 0x7f40ac100730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f40ac10ba10 0x7f40ac100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b173a640 1 -- 172.21.15.60:0/404142813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40ac102ec0 msgr2=0x7f40ac100730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b173a640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40ac102ec0 0x7f40ac100730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b173a640 1 -- 172.21.15.60:0/404142813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f40ac10ba10 msgr2=0x7f40ac100c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b173a640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f40ac10ba10 0x7f40ac100c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b173a640 1 -- 172.21.15.60:0/404142813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac1a6140 con 0x7f40ac1025a0 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b0f39640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40ac102ec0 0x7f40ac100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.362+0000 7f40b173a640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40ac1025a0 0x7f40ac1001f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f409c00cce0 tx=0x7f409c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.363+0000 7f40aa7fc640 1 -- 172.21.15.60:0/404142813 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c00f650 con 0x7f40ac1025a0 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.363+0000 7f40aa7fc640 1 -- 172.21.15.60:0/404142813 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f409c00fcd0 con 0x7f40ac1025a0 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.363+0000 7f40b399e640 1 -- 172.21.15.60:0/404142813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ac1a6430 con 0x7f40ac1025a0 2024-03-04T12:26:16.372 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.363+0000 7f40aa7fc640 1 -- 172.21.15.60:0/404142813 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c018c70 con 0x7f40ac1025a0 2024-03-04T12:26:16.373 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.363+0000 7f40b399e640 1 -- 172.21.15.60:0/404142813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ac1a6980 con 0x7f40ac1025a0 2024-03-04T12:26:16.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.365+0000 7f40aa7fc640 1 -- 172.21.15.60:0/404142813 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f409c00f7f0 con 0x7f40ac1025a0 2024-03-04T12:26:16.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.365+0000 7f40b657f740 1 Processor -- start 2024-03-04T12:26:16.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.365+0000 7f40b657f740 1 -- start start 2024-03-04T12:26:16.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40aa7fc640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f40940a2040 0x7f40940a4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b657f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 0x55cdac2dcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b657f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b657f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b657f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdac2a3dc0 con 0x55cdac2dc9d0 2024-03-04T12:26:16.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b657f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdac23fec0 con 0x55cdac2e5ed0 2024-03-04T12:26:16.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b657f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdac1e8d30 con 0x55cdac2dd2f0 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f40940a2040 0x7f40940a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40aa7fc640 1 -- 172.21.15.60:0/404142813 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f409c0ced40 con 0x7f40ac1025a0 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b173a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 0x55cdac2dcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b173a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 0x55cdac2dcdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53638/0 (socket says 172.21.15.60:53638) 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b173a640 1 -- 172.21.15.60:0/2295196674 learned_addr learned my addr 172.21.15.60:0/2295196674 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b0f39640 1 --2- 172.21.15.60:0/2295196674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/2295196674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b173a640 1 -- 172.21.15.60:0/2295196674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dd2f0 msgr2=0x55cdac2e5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b173a640 1 --2- 172.21.15.60:0/2295196674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b173a640 1 -- 172.21.15.60:0/2295196674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2e5ed0 msgr2=0x55cdac2e82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b173a640 1 --2- 172.21.15.60:0/2295196674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b173a640 1 -- 172.21.15.60:0/2295196674 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cdabf543f0 con 0x55cdac2dc9d0 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/2295196674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:16.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f40940a2040 0x7f40940a4500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f40a4005d80 tx=0x7f40a40157f0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40b173a640 1 --2- 172.21.15.60:0/2295196674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 0x55cdac2dcdb0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f409c09ae40 tx=0x7f409c09ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=288b5b81081c232d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40937fe640 1 -- 172.21.15.60:0/2295196674 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c0a01b0 con 0x55cdac2dc9d0 2024-03-04T12:26:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40937fe640 1 -- 172.21.15.60:0/2295196674 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f409c09b840 con 0x55cdac2dc9d0 2024-03-04T12:26:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.366+0000 7f40937fe640 1 -- 172.21.15.60:0/2295196674 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c0a0990 con 0x55cdac2dc9d0 2024-03-04T12:26:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.367+0000 7f40b657f740 1 -- 172.21.15.60:0/2295196674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 msgr2=0x55cdac2dcdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.367+0000 7f40b657f740 1 --2- 172.21.15.60:0/2295196674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 0x55cdac2dcdb0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f409c09ae40 tx=0x7f409c09ac80 comp rx=0 tx=0).stop 2024-03-04T12:26:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.367+0000 7f40b657f740 1 -- 172.21.15.60:0/2295196674 shutdown_connections 2024-03-04T12:26:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.367+0000 7f40b657f740 1 --2- 172.21.15.60:0/2295196674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.367+0000 7f40b657f740 1 --2- 172.21.15.60:0/2295196674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.367+0000 7f40b657f740 1 --2- 172.21.15.60:0/2295196674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 0x55cdac2dcdb0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.367+0000 7f40b657f740 1 -- 172.21.15.60:0/2295196674 >> 172.21.15.60:0/2295196674 conn(0x55cdac2d7100 msgr2=0x55cdac2d74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.367+0000 7f40b657f740 1 -- 172.21.15.60:0/2295196674 shutdown_connections 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.367+0000 7f40b657f740 1 -- 172.21.15.60:0/2295196674 wait complete. 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b657f740 1 -- start start 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b657f740 1 Processor -- start 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b657f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dc9d0 0x55cdac2e2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b657f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b657f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b657f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdac1de4a0 con 0x55cdac2e5ed0 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b657f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdac23bf60 con 0x55cdac2dd2f0 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b657f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cdac2e8cc0 con 0x55cdac2dc9d0 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b657f740 10 client.0 _ll_register_callbacks cb 0x55cdac23a1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b1f3b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b1f3b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53652/0 (socket says 172.21.15.60:53652) 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b1f3b640 1 -- 172.21.15.60:0/1859372613 learned_addr learned my addr 172.21.15.60:0/1859372613 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.383 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b173a640 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dc9d0 0x55cdac2e2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b0f39640 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b1f3b640 1 -- 172.21.15.60:0/1859372613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dc9d0 msgr2=0x55cdac2e2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dc9d0 0x55cdac2e2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b1f3b640 1 -- 172.21.15.60:0/1859372613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2dd2f0 msgr2=0x55cdac2e3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b1f3b640 1 -- 172.21.15.60:0/1859372613 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cdac01e7d0 con 0x55cdac2e5ed0 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.368+0000 7f40b0f39640 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e3970 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f40a40099b0 tx=0x7f40a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6feb19e128355915 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f4090ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f4090ff9640 1 -- 172.21.15.60:0/1859372613 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a4025020 con 0x55cdac2e5ed0 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f4090ff9640 1 -- 172.21.15.60:0/1859372613 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f40a4004220 con 0x55cdac2e5ed0 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f40b657f740 1 -- 172.21.15.60:0/1859372613 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f409c09b840 con 0x55cdac2e5ed0 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f4090ff9640 1 -- 172.21.15.60:0/1859372613 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40a402d050 con 0x55cdac2e5ed0 2024-03-04T12:26:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f40b657f740 10 client.15909 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f40b657f740 1 -- 172.21.15.60:0/1859372613 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55cdac2f4dc0 con 0x55cdac2e5ed0 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f4090ff9640 1 -- 172.21.15.60:0/1859372613 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f409c09b840 con 0x55cdac2e5ed0 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f4090ff9640 1 -- 172.21.15.60:0/1859372613 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f40a4004b60 con 0x55cdac2e5ed0 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f40b657f740 10 client.15909 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f40b657f740 10 client.15909 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f40b657f740 20 client.15909 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f40b657f740 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55cdac2f62d0 0x55cdac2f86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f40b657f740 4 client.15909 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.369+0000 7f40b657f740 1 -- 172.21.15.60:0/1859372613 --> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cdac24d400 con 0x55cdac2f62d0 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.370+0000 7f40b173a640 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55cdac2f62d0 0x55cdac2f86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.370+0000 7f4090ff9640 10 client.15909 ms_handle_connect on v2:172.21.15.63:6836/3974574804 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.370+0000 7f40b173a640 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55cdac2f62d0 0x55cdac2f86c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7eb8d965b8afd3b7 server_cookie=be1f091356a06a58 in_seq=0 out_seq=0 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f4090ff9640 1 -- 172.21.15.60:0/1859372613 <== mds.2 v2:172.21.15.63:6836/3974574804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cdac24d400 con 0x55cdac2f62d0 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f4090ff9640 10 client.15909 handle_command_reply: tid=0 2024-03-04T12:26:16.385 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f40b657f740 1 client.15909 shutdown 2024-03-04T12:26:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f40b657f740 1 -- 172.21.15.60:0/1859372613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 msgr2=0x55cdac2e3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f40b657f740 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e3970 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f40a40099b0 tx=0x7f40a4004730 comp rx=0 tx=0).stop 2024-03-04T12:26:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f40b657f740 1 -- 172.21.15.60:0/1859372613 shutdown_connections 2024-03-04T12:26:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f40b657f740 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55cdac2f62d0 0x55cdac2f86c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f40b657f740 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e3970 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f40b657f740 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f4090ff9640 0 client.15909 ms_handle_reset on v2:172.21.15.63:6836/3974574804 2024-03-04T12:26:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f40b657f740 1 --2- 172.21.15.60:0/1859372613 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dc9d0 0x55cdac2e2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f40b657f740 1 -- 172.21.15.60:0/1859372613 >> 172.21.15.60:0/1859372613 conn(0x55cdac2d7100 msgr2=0x55cdac2d8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f40b657f740 1 -- 172.21.15.60:0/1859372613 shutdown_connections 2024-03-04T12:26:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f40b657f740 1 -- 172.21.15.60:0/1859372613 wait complete. 2024-03-04T12:26:16.386 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.371+0000 7f40b657f740 20 client.15909 trim_cache size 0 max 16384 2024-03-04T12:26:16.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b657f740 1 Processor -- start 2024-03-04T12:26:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b657f740 1 -- start start 2024-03-04T12:26:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b657f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 0x55cdac2e5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b657f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b657f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b657f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac1b06b0 con 0x55cdac2dc9d0 2024-03-04T12:26:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b657f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac1b2030 con 0x55cdac2e5ed0 2024-03-04T12:26:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b657f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac1b0d10 con 0x55cdac2dd2f0 2024-03-04T12:26:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b173a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 0x55cdac2e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b173a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 0x55cdac2e5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53654/0 (socket says 172.21.15.60:53654) 2024-03-04T12:26:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b173a640 1 -- 172.21.15.60:0/1606342079 learned_addr learned my addr 172.21.15.60:0/1606342079 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/1606342079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b0f39640 1 --2- 172.21.15.60:0/1606342079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b173a640 1 -- 172.21.15.60:0/1606342079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dd2f0 msgr2=0x55cdac2e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b173a640 1 --2- 172.21.15.60:0/1606342079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b173a640 1 -- 172.21.15.60:0/1606342079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2e5ed0 msgr2=0x55cdac2e0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b173a640 1 --2- 172.21.15.60:0/1606342079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b173a640 1 -- 172.21.15.60:0/1606342079 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cdac2dd120 con 0x55cdac2dc9d0 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b0f39640 1 --2- 172.21.15.60:0/1606342079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.383+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/1606342079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f40b173a640 1 --2- 172.21.15.60:0/1606342079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 0x55cdac2e5620 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f409c00ab20 tx=0x7f409c00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bbf0f71aa23e632 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f4092ffd640 1 -- 172.21.15.60:0/1606342079 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c0b7070 con 0x55cdac2dc9d0 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f4092ffd640 1 -- 172.21.15.60:0/1606342079 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f409c0a38e0 con 0x55cdac2dc9d0 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f4092ffd640 1 -- 172.21.15.60:0/1606342079 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c0a0440 con 0x55cdac2dc9d0 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f40b657f740 1 -- 172.21.15.60:0/1606342079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 msgr2=0x55cdac2e5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f40b657f740 1 --2- 172.21.15.60:0/1606342079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 0x55cdac2e5620 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f409c00ab20 tx=0x7f409c00cf70 comp rx=0 tx=0).stop 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f40b657f740 1 -- 172.21.15.60:0/1606342079 shutdown_connections 2024-03-04T12:26:16.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f40b657f740 1 --2- 172.21.15.60:0/1606342079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2e5ed0 0x55cdac2e0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f40b657f740 1 --2- 172.21.15.60:0/1606342079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dd2f0 0x55cdac2e1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f40b657f740 1 --2- 172.21.15.60:0/1606342079 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2dc9d0 0x55cdac2e5620 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f40b657f740 1 -- 172.21.15.60:0/1606342079 >> 172.21.15.60:0/1606342079 conn(0x55cdac2d7100 msgr2=0x55cdac2e17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f40b657f740 1 -- 172.21.15.60:0/1606342079 shutdown_connections 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.384+0000 7f40b657f740 1 -- 172.21.15.60:0/1606342079 wait complete. 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b657f740 1 -- start start 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b657f740 1 Processor -- start 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b657f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dc9d0 0x55cdac2eba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b657f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2dd2f0 0x55cdac2ebf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b657f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 0x55cdac2ec490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b657f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac1b19d0 con 0x55cdac2e5ed0 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b657f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac1b1370 con 0x55cdac2dd2f0 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b657f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac1ae950 con 0x55cdac2dc9d0 2024-03-04T12:26:16.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b657f740 10 client.0 _ll_register_callbacks cb 0x7f409c018650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b1f3b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 0x55cdac2ec490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b1f3b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 0x55cdac2ec490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53660/0 (socket says 172.21.15.60:53660) 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b1f3b640 1 -- 172.21.15.60:0/768340922 learned_addr learned my addr 172.21.15.60:0/768340922 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b173a640 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dc9d0 0x55cdac2eba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b0f39640 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2dd2f0 0x55cdac2ebf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b1f3b640 1 -- 172.21.15.60:0/768340922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dc9d0 msgr2=0x55cdac2eba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dc9d0 0x55cdac2eba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b1f3b640 1 -- 172.21.15.60:0/768340922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2dd2f0 msgr2=0x55cdac2ebf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2dd2f0 0x55cdac2ebf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.385+0000 7f40b1f3b640 1 -- 172.21.15.60:0/768340922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cdac34c810 con 0x55cdac2e5ed0 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f40b1f3b640 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 0x55cdac2ec490 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f40a40151b0 tx=0x7f40a4005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398e60b4fea551a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f4090ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f4090ff9640 1 -- 172.21.15.60:0/768340922 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a4025040 con 0x55cdac2e5ed0 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f4090ff9640 1 -- 172.21.15.60:0/768340922 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f40a4004bb0 con 0x55cdac2e5ed0 2024-03-04T12:26:16.399 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f40b657f740 1 -- 172.21.15.60:0/768340922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f409c0a38e0 con 0x55cdac2e5ed0 2024-03-04T12:26:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f4090ff9640 1 -- 172.21.15.60:0/768340922 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40a4041050 con 0x55cdac2e5ed0 2024-03-04T12:26:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f40b657f740 10 client.15915 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f40b657f740 1 -- 172.21.15.60:0/768340922 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55cdac34da80 con 0x55cdac2e5ed0 2024-03-04T12:26:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f4090ff9640 1 -- 172.21.15.60:0/768340922 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f409c0a38e0 con 0x55cdac2e5ed0 2024-03-04T12:26:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f4090ff9640 1 -- 172.21.15.60:0/768340922 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f40a40054b0 con 0x55cdac2e5ed0 2024-03-04T12:26:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f40b657f740 10 client.15915 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f40b657f740 10 client.15915 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-04T12:26:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f40b657f740 20 client.15915 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:16.400 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f40b657f740 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55cdac34e110 0x55cdac350530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f40b657f740 4 client.15915 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.386+0000 7f40b657f740 1 -- 172.21.15.60:0/768340922 --> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cdac247750 con 0x55cdac34e110 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b173a640 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55cdac34e110 0x55cdac350530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f4090ff9640 10 client.15915 ms_handle_connect on v2:172.21.15.63:6836/3974574804 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b173a640 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55cdac34e110 0x55cdac350530 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a7c4fc954c33fb59 server_cookie=e1df16d2732dc886 in_seq=0 out_seq=0 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f4090ff9640 1 -- 172.21.15.60:0/768340922 <== mds.2 v2:172.21.15.63:6836/3974574804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cdac247750 con 0x55cdac34e110 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f4090ff9640 10 client.15915 handle_command_reply: tid=0 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b657f740 1 client.15915 shutdown 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b657f740 1 -- 172.21.15.60:0/768340922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 msgr2=0x55cdac2ec490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b657f740 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 0x55cdac2ec490 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f40a40151b0 tx=0x7f40a4005b20 comp rx=0 tx=0).stop 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b657f740 1 -- 172.21.15.60:0/768340922 shutdown_connections 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b657f740 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55cdac34e110 0x55cdac350530 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b657f740 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cdac2e5ed0 0x55cdac2ec490 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f4090ff9640 0 client.15915 ms_handle_reset on v2:172.21.15.63:6836/3974574804 2024-03-04T12:26:16.401 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b657f740 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cdac2dd2f0 0x55cdac2ebf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b657f740 1 --2- 172.21.15.60:0/768340922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cdac2dc9d0 0x55cdac2eba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b657f740 1 -- 172.21.15.60:0/768340922 >> 172.21.15.60:0/768340922 conn(0x55cdac2d7100 msgr2=0x55cdac2e2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b657f740 1 -- 172.21.15.60:0/768340922 shutdown_connections 2024-03-04T12:26:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.387+0000 7f40b657f740 1 -- 172.21.15.60:0/768340922 wait complete. 2024-03-04T12:26:16.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b657f740 20 client.15915 trim_cache size 0 max 16384 2024-03-04T12:26:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:26:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b399e640 1 -- 172.21.15.60:0/404142813 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f40940a2040 msgr2=0x7f40940a4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b399e640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f40940a2040 0x7f40940a4500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f40a4005d80 tx=0x7f40a40157f0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b399e640 1 -- 172.21.15.60:0/404142813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40ac1025a0 msgr2=0x7f40ac1001f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b399e640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40ac1025a0 0x7f40ac1001f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f409c00cce0 tx=0x7f409c00a620 comp rx=0 tx=0).stop 2024-03-04T12:26:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b399e640 1 -- 172.21.15.60:0/404142813 shutdown_connections 2024-03-04T12:26:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b399e640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f40ac10ba10 0x7f40ac100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b399e640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40ac102ec0 0x7f40ac100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b399e640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f40940a2040 0x7f40940a4500 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b399e640 1 --2- 172.21.15.60:0/404142813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f40ac1025a0 0x7f40ac1001f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b399e640 1 -- 172.21.15.60:0/404142813 >> 172.21.15.60:0/404142813 conn(0x7f40ac0773a0 msgr2=0x7f40ac0fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b399e640 1 -- 172.21.15.60:0/404142813 shutdown_connections 2024-03-04T12:26:16.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.388+0000 7f40b399e640 1 -- 172.21.15.60:0/404142813 wait complete. 2024-03-04T12:26:16.405 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:26:16.405 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-04T12:26:16.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.503+0000 7fad81989640 1 -- 172.21.15.60:0/1555462001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad7c100540 msgr2=0x7fad7c100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.503+0000 7fad81989640 1 --2- 172.21.15.60:0/1555462001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad7c100540 0x7fad7c100920 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009930 tx=0x7fad6c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:16.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.503+0000 7fad81989640 1 -- 172.21.15.60:0/1555462001 shutdown_connections 2024-03-04T12:26:16.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.503+0000 7fad81989640 1 --2- 172.21.15.60:0/1555462001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c1027c0 0x7fad7c10d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.503+0000 7fad81989640 1 --2- 172.21.15.60:0/1555462001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad7c101e20 0x7fad7c102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.512 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.503+0000 7fad81989640 1 --2- 172.21.15.60:0/1555462001 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad7c100540 0x7fad7c100920 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.503+0000 7fad81989640 1 -- 172.21.15.60:0/1555462001 >> 172.21.15.60:0/1555462001 conn(0x7fad7c077410 msgr2=0x7fad7c077820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.503+0000 7fad81989640 1 -- 172.21.15.60:0/1555462001 shutdown_connections 2024-03-04T12:26:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.503+0000 7fad81989640 1 -- 172.21.15.60:0/1555462001 wait complete. 2024-03-04T12:26:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad81989640 1 Processor -- start 2024-03-04T12:26:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad81989640 1 -- start start 2024-03-04T12:26:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad81989640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad7c100540 0x7fad7c19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad81989640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad7c101e20 0x7fad7c19ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad81989640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c1027c0 0x7fad7c1969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad81989640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1106b0 con 0x7fad7c100540 2024-03-04T12:26:16.513 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad81989640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c110530 con 0x7fad7c1027c0 2024-03-04T12:26:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad81989640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c110830 con 0x7fad7c101e20 2024-03-04T12:26:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad7affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad7c100540 0x7fad7c19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad7affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad7c100540 0x7fad7c19c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53678/0 (socket says 172.21.15.60:53678) 2024-03-04T12:26:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad7affd640 1 -- 172.21.15.60:0/1813984648 learned_addr learned my addr 172.21.15.60:0/1813984648 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad727fc640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad7c101e20 0x7fad7c19ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad7b7fe640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c1027c0 0x7fad7c1969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad7affd640 1 -- 172.21.15.60:0/1813984648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad7c101e20 msgr2=0x7fad7c19ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad7affd640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad7c101e20 0x7fad7c19ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad7affd640 1 -- 172.21.15.60:0/1813984648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c1027c0 msgr2=0x7fad7c1969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad7affd640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c1027c0 0x7fad7c1969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad7affd640 1 -- 172.21.15.60:0/1813984648 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7c197130 con 0x7fad7c100540 2024-03-04T12:26:16.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.504+0000 7fad7b7fe640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c1027c0 0x7fad7c1969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.505+0000 7fad7affd640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad7c100540 0x7fad7c19c7b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009230 tx=0x7fad6c02cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.505+0000 7fad80987640 1 -- 172.21.15.60:0/1813984648 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c041070 con 0x7fad7c100540 2024-03-04T12:26:16.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.505+0000 7fad81989640 1 -- 172.21.15.60:0/1813984648 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7c1973c0 con 0x7fad7c100540 2024-03-04T12:26:16.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.505+0000 7fad80987640 1 -- 172.21.15.60:0/1813984648 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad6c004450 con 0x7fad7c100540 2024-03-04T12:26:16.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.505+0000 7fad80987640 1 -- 172.21.15.60:0/1813984648 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c034440 con 0x7fad7c100540 2024-03-04T12:26:16.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.505+0000 7fad81989640 1 -- 172.21.15.60:0/1813984648 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7c1a3670 con 0x7fad7c100540 2024-03-04T12:26:16.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.507+0000 7fad80987640 1 -- 172.21.15.60:0/1813984648 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7fad6c044070 con 0x7fad7c100540 2024-03-04T12:26:16.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.507+0000 7fad8456b740 1 Processor -- start 2024-03-04T12:26:16.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.507+0000 7fad8456b740 1 -- start start 2024-03-04T12:26:16.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.507+0000 7fad8456b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 0x5557d6c6e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.507+0000 7fad8456b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c79ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.507+0000 7fad8456b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c7c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.507+0000 7fad80987640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fad680a2010 0x7fad680a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.507+0000 7fad8456b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d6c35dc0 con 0x5557d6c7a0e0 2024-03-04T12:26:16.526 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.507+0000 7fad8456b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d6bd1ec0 con 0x5557d6c6ed80 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.507+0000 7fad8456b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d6b7ad30 con 0x5557d6c6e3d0 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad727fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c7c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad7b7fe640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fad680a2010 0x7fad680a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad727fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c7c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53682/0 (socket says 172.21.15.60:53682) 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad727fc640 1 -- 172.21.15.60:0/3814734040 learned_addr learned my addr 172.21.15.60:0/3814734040 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad7b7fe640 1 --2- 172.21.15.60:0/3814734040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 0x5557d6c6e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad80987640 1 -- 172.21.15.60:0/1813984648 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fad6c0f0200 con 0x7fad7c100540 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad7affd640 1 --2- 172.21.15.60:0/3814734040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c79ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad727fc640 1 -- 172.21.15.60:0/3814734040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 msgr2=0x5557d6c6e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad727fc640 1 --2- 172.21.15.60:0/3814734040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 0x5557d6c6e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad727fc640 1 -- 172.21.15.60:0/3814734040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 msgr2=0x5557d6c79ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad727fc640 1 --2- 172.21.15.60:0/3814734040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c79ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad727fc640 1 -- 172.21.15.60:0/3814734040 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557d68e63f0 con 0x5557d6c7a0e0 2024-03-04T12:26:16.527 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad7b7fe640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fad680a2010 0x7fad680a44d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fad74005fd0 tx=0x7fad74009e80 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad727fc640 1 --2- 172.21.15.60:0/3814734040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c7c4d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fad6400b6c0 tx=0x7fad6400bb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a192e92761c90d32 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad71ffb640 1 -- 172.21.15.60:0/3814734040 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64003bd0 con 0x5557d6c7a0e0 2024-03-04T12:26:16.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad71ffb640 1 -- 172.21.15.60:0/3814734040 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad64015070 con 0x5557d6c7a0e0 2024-03-04T12:26:16.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.508+0000 7fad71ffb640 1 -- 172.21.15.60:0/3814734040 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64004d70 con 0x5557d6c7a0e0 2024-03-04T12:26:16.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.509+0000 7fad8456b740 1 -- 172.21.15.60:0/3814734040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 msgr2=0x5557d6c7c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.509+0000 7fad8456b740 1 --2- 172.21.15.60:0/3814734040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c7c4d0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fad6400b6c0 tx=0x7fad6400bb80 comp rx=0 tx=0).stop 2024-03-04T12:26:16.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.509+0000 7fad8456b740 1 -- 172.21.15.60:0/3814734040 shutdown_connections 2024-03-04T12:26:16.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.509+0000 7fad8456b740 1 --2- 172.21.15.60:0/3814734040 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c7c4d0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.509+0000 7fad8456b740 1 --2- 172.21.15.60:0/3814734040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c79ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.509+0000 7fad8456b740 1 --2- 172.21.15.60:0/3814734040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 0x5557d6c6e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.509+0000 7fad8456b740 1 -- 172.21.15.60:0/3814734040 >> 172.21.15.60:0/3814734040 conn(0x5557d6c69100 msgr2=0x5557d6c694e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.509+0000 7fad8456b740 1 -- 172.21.15.60:0/3814734040 shutdown_connections 2024-03-04T12:26:16.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.509+0000 7fad8456b740 1 -- 172.21.15.60:0/3814734040 wait complete. 2024-03-04T12:26:16.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.509+0000 7fad8456b740 1 -- start start 2024-03-04T12:26:16.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad8456b740 1 Processor -- start 2024-03-04T12:26:16.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad8456b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 0x5557d6c771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad8456b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c776f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad8456b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c77c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad8456b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d6b704a0 con 0x5557d6c7a0e0 2024-03-04T12:26:16.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad8456b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d6bcdf60 con 0x5557d6c6ed80 2024-03-04T12:26:16.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad8456b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557d6c7ced0 con 0x5557d6c6e3d0 2024-03-04T12:26:16.530 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad8456b740 10 client.0 _ll_register_callbacks cb 0x5557d6bcc1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:16.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad7b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c77c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad7b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c77c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53684/0 (socket says 172.21.15.60:53684) 2024-03-04T12:26:16.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad7b7fe640 1 -- 172.21.15.60:0/2299745025 learned_addr learned my addr 172.21.15.60:0/2299745025 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad7affd640 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 0x5557d6c771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad727fc640 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c776f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad7b7fe640 1 -- 172.21.15.60:0/2299745025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 msgr2=0x5557d6c771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.531 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad7b7fe640 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 0x5557d6c771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad7b7fe640 1 -- 172.21.15.60:0/2299745025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 msgr2=0x5557d6c776f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad7b7fe640 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c776f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad7b7fe640 1 -- 172.21.15.60:0/2299745025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557d69b07d0 con 0x5557d6c7a0e0 2024-03-04T12:26:16.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad727fc640 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c776f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.532 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad7b7fe640 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c77c30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fad74011d10 tx=0x7fad74009410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e55c8cb5a8c54a5e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad5affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:16.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.510+0000 7fad5affd640 1 -- 172.21.15.60:0/2299745025 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74026070 con 0x5557d6c7a0e0 2024-03-04T12:26:16.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad5affd640 1 -- 172.21.15.60:0/2299745025 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fad740097e0 con 0x5557d6c7a0e0 2024-03-04T12:26:16.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad8456b740 1 -- 172.21.15.60:0/2299745025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad64015070 con 0x5557d6c7a0e0 2024-03-04T12:26:16.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad5affd640 1 -- 172.21.15.60:0/2299745025 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad74031050 con 0x5557d6c7a0e0 2024-03-04T12:26:16.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad8456b740 10 client.15927 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:16.533 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad8456b740 1 -- 172.21.15.60:0/2299745025 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5557d6c890c0 con 0x5557d6c7a0e0 2024-03-04T12:26:16.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad5affd640 1 -- 172.21.15.60:0/2299745025 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad64015070 con 0x5557d6c7a0e0 2024-03-04T12:26:16.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad5affd640 1 -- 172.21.15.60:0/2299745025 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fad7400f050 con 0x5557d6c7a0e0 2024-03-04T12:26:16.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad8456b740 10 client.15927 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:16.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad8456b740 10 client.15927 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-04T12:26:16.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad8456b740 20 client.15927 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:16.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad8456b740 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5557d6c8a570 0x5557d6c8c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.534 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad8456b740 4 client.15927 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:16.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad8456b740 1 -- 172.21.15.60:0/2299745025 --> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5557d6bdf400 con 0x5557d6c8a570 2024-03-04T12:26:16.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.511+0000 7fad7affd640 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5557d6c8a570 0x5557d6c8c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.512+0000 7fad7affd640 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5557d6c8a570 0x5557d6c8c9f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f1b6bc366128229 server_cookie=628b12f926ea944d in_seq=0 out_seq=0 2024-03-04T12:26:16.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.512+0000 7fad5affd640 10 client.15927 ms_handle_connect on v2:172.21.15.86:6838/3347800322 2024-03-04T12:26:16.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad5affd640 1 -- 172.21.15.60:0/2299745025 <== mds.1 v2:172.21.15.86:6838/3347800322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5557d6bdf400 con 0x5557d6c8a570 2024-03-04T12:26:16.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad5affd640 10 client.15927 handle_command_reply: tid=0 2024-03-04T12:26:16.535 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad8456b740 1 client.15927 shutdown 2024-03-04T12:26:16.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad8456b740 1 -- 172.21.15.60:0/2299745025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 msgr2=0x5557d6c77c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad8456b740 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c77c30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fad74011d10 tx=0x7fad74009410 comp rx=0 tx=0).stop 2024-03-04T12:26:16.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad8456b740 1 -- 172.21.15.60:0/2299745025 shutdown_connections 2024-03-04T12:26:16.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad8456b740 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5557d6c8a570 0x5557d6c8c9f0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad8456b740 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c77c30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad8456b740 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c776f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad5affd640 0 client.15927 ms_handle_reset on v2:172.21.15.86:6838/3347800322 2024-03-04T12:26:16.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad8456b740 1 --2- 172.21.15.60:0/2299745025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 0x5557d6c771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad8456b740 1 -- 172.21.15.60:0/2299745025 >> 172.21.15.60:0/2299745025 conn(0x5557d6c69100 msgr2=0x5557d6c6ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad8456b740 1 -- 172.21.15.60:0/2299745025 shutdown_connections 2024-03-04T12:26:16.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad8456b740 1 -- 172.21.15.60:0/2299745025 wait complete. 2024-03-04T12:26:16.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.513+0000 7fad8456b740 20 client.15927 trim_cache size 0 max 16384 2024-03-04T12:26:16.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.525+0000 7fad8456b740 1 Processor -- start 2024-03-04T12:26:16.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.525+0000 7fad8456b740 1 -- start start 2024-03-04T12:26:16.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.525+0000 7fad8456b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 0x5557d6c798e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.525+0000 7fad8456b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c75a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.525+0000 7fad8456b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c75fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.525+0000 7fad8456b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1ac100 con 0x5557d6c7a0e0 2024-03-04T12:26:16.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.525+0000 7fad8456b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1ada80 con 0x5557d6c6ed80 2024-03-04T12:26:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.525+0000 7fad8456b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1ac760 con 0x5557d6c6e3d0 2024-03-04T12:26:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.525+0000 7fad7b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c75fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.525+0000 7fad7b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c75fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53690/0 (socket says 172.21.15.60:53690) 2024-03-04T12:26:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.525+0000 7fad7b7fe640 1 -- 172.21.15.60:0/102902721 learned_addr learned my addr 172.21.15.60:0/102902721 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad7b7fe640 1 -- 172.21.15.60:0/102902721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 msgr2=0x5557d6c798e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad727fc640 1 --2- 172.21.15.60:0/102902721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c75a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad7b7fe640 1 --2- 172.21.15.60:0/102902721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 0x5557d6c798e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad7b7fe640 1 -- 172.21.15.60:0/102902721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 msgr2=0x5557d6c75a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad7b7fe640 1 --2- 172.21.15.60:0/102902721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c75a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad7b7fe640 1 -- 172.21.15.60:0/102902721 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557d6c6a2c0 con 0x5557d6c7a0e0 2024-03-04T12:26:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad727fc640 1 --2- 172.21.15.60:0/102902721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c75a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad7b7fe640 1 --2- 172.21.15.60:0/102902721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c75fc0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fad740143e0 tx=0x7fad74004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c1cc9301b50378 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad717fa640 1 -- 172.21.15.60:0/102902721 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7402e070 con 0x5557d6c7a0e0 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad717fa640 1 -- 172.21.15.60:0/102902721 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad74020450 con 0x5557d6c7a0e0 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad717fa640 1 -- 172.21.15.60:0/102902721 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad74026070 con 0x5557d6c7a0e0 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad8456b740 1 -- 172.21.15.60:0/102902721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 msgr2=0x5557d6c75fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad8456b740 1 --2- 172.21.15.60:0/102902721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c75fc0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fad740143e0 tx=0x7fad74004a20 comp rx=0 tx=0).stop 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad8456b740 1 -- 172.21.15.60:0/102902721 shutdown_connections 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad8456b740 1 --2- 172.21.15.60:0/102902721 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c7a0e0 0x5557d6c75fc0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad8456b740 1 --2- 172.21.15.60:0/102902721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c6ed80 0x5557d6c75a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.526+0000 7fad8456b740 1 --2- 172.21.15.60:0/102902721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6e3d0 0x5557d6c798e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad8456b740 1 -- 172.21.15.60:0/102902721 >> 172.21.15.60:0/102902721 conn(0x5557d6c69100 msgr2=0x5557d6c91350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad8456b740 1 -- 172.21.15.60:0/102902721 shutdown_connections 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad8456b740 1 -- 172.21.15.60:0/102902721 wait complete. 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad8456b740 1 -- start start 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad8456b740 1 Processor -- start 2024-03-04T12:26:16.547 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad8456b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c6e3d0 0x5557d6c7fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad8456b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6ed80 0x5557d6c80040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad8456b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c7a0e0 0x5557d6c80580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad8456b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1ad420 con 0x5557d6c6e3d0 2024-03-04T12:26:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad8456b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1acdc0 con 0x5557d6c7a0e0 2024-03-04T12:26:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad8456b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c1aa3a0 con 0x5557d6c6ed80 2024-03-04T12:26:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad8456b740 10 client.0 _ll_register_callbacks cb 0x7fad64004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad7affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c6e3d0 0x5557d6c7fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad7affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c6e3d0 0x5557d6c7fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53706/0 (socket says 172.21.15.60:53706) 2024-03-04T12:26:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.527+0000 7fad7affd640 1 -- 172.21.15.60:0/262677371 learned_addr learned my addr 172.21.15.60:0/262677371 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad7b7fe640 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c7a0e0 0x5557d6c80580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad727fc640 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6ed80 0x5557d6c80040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad7affd640 1 -- 172.21.15.60:0/262677371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6ed80 msgr2=0x5557d6c80040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad7affd640 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6ed80 0x5557d6c80040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad7affd640 1 -- 172.21.15.60:0/262677371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c7a0e0 msgr2=0x5557d6c80580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad7affd640 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c7a0e0 0x5557d6c80580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad7affd640 1 -- 172.21.15.60:0/262677371 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557d6ce0ae0 con 0x5557d6c6e3d0 2024-03-04T12:26:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad727fc640 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6ed80 0x5557d6c80040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad7affd640 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c6e3d0 0x5557d6c7fb00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fad6c02cec0 tx=0x7fad6c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc944514e2398c4a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad5affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad5affd640 1 -- 172.21.15.60:0/262677371 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c0d0070 con 0x5557d6c6e3d0 2024-03-04T12:26:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad5affd640 1 -- 172.21.15.60:0/262677371 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fad6c0b8d80 con 0x5557d6c6e3d0 2024-03-04T12:26:16.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad8456b740 1 -- 172.21.15.60:0/262677371 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad74020450 con 0x5557d6c6e3d0 2024-03-04T12:26:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad5affd640 1 -- 172.21.15.60:0/262677371 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad6c04b040 con 0x5557d6c6e3d0 2024-03-04T12:26:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad8456b740 10 client.15933 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad8456b740 1 -- 172.21.15.60:0/262677371 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5557d6ce1d30 con 0x5557d6c6e3d0 2024-03-04T12:26:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad5affd640 1 -- 172.21.15.60:0/262677371 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad74020450 con 0x5557d6c6e3d0 2024-03-04T12:26:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.528+0000 7fad5affd640 1 -- 172.21.15.60:0/262677371 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fad6c0036f0 con 0x5557d6c6e3d0 2024-03-04T12:26:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad8456b740 10 client.15933 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad8456b740 10 client.15933 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-04T12:26:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad8456b740 20 client.15933 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad8456b740 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5557d6ce2360 0x5557d6ce4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad8456b740 4 client.15933 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:26:16.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad8456b740 1 -- 172.21.15.60:0/262677371 --> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5557d6bd9750 con 0x5557d6ce2360 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad727fc640 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5557d6ce2360 0x5557d6ce4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad727fc640 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5557d6ce2360 0x5557d6ce4750 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7203928949067571 server_cookie=e4e2474c73e87e87 in_seq=0 out_seq=0 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad5affd640 10 client.15933 ms_handle_connect on v2:172.21.15.86:6838/3347800322 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad5affd640 1 -- 172.21.15.60:0/262677371 <== mds.1 v2:172.21.15.86:6838/3347800322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5557d6bd9750 con 0x5557d6ce2360 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad5affd640 10 client.15933 handle_command_reply: tid=0 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad8456b740 1 client.15933 shutdown 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad8456b740 1 -- 172.21.15.60:0/262677371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c6e3d0 msgr2=0x5557d6c7fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.529+0000 7fad8456b740 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c6e3d0 0x5557d6c7fb00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fad6c02cec0 tx=0x7fad6c000bc0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad8456b740 1 -- 172.21.15.60:0/262677371 shutdown_connections 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad8456b740 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5557d6c7a0e0 0x5557d6c80580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad8456b740 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5557d6c6ed80 0x5557d6c80040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad8456b740 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5557d6ce2360 0x5557d6ce4750 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad8456b740 1 --2- 172.21.15.60:0/262677371 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5557d6c6e3d0 0x5557d6c7fb00 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad5affd640 0 client.15933 ms_handle_reset on v2:172.21.15.86:6838/3347800322 2024-03-04T12:26:16.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad8456b740 1 -- 172.21.15.60:0/262677371 >> 172.21.15.60:0/262677371 conn(0x5557d6c69100 msgr2=0x5557d6c759c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad8456b740 1 -- 172.21.15.60:0/262677371 shutdown_connections 2024-03-04T12:26:16.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad8456b740 1 -- 172.21.15.60:0/262677371 wait complete. 2024-03-04T12:26:16.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad8456b740 20 client.15933 trim_cache size 0 max 16384 2024-03-04T12:26:16.552 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:26:16.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad81989640 1 -- 172.21.15.60:0/1813984648 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fad680a2010 msgr2=0x7fad680a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad81989640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fad680a2010 0x7fad680a44d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fad74005fd0 tx=0x7fad74009e80 comp rx=0 tx=0).stop 2024-03-04T12:26:16.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad81989640 1 -- 172.21.15.60:0/1813984648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad7c100540 msgr2=0x7fad7c19c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad81989640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad7c100540 0x7fad7c19c7b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009230 tx=0x7fad6c02cc70 comp rx=0 tx=0).stop 2024-03-04T12:26:16.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad81989640 1 -- 172.21.15.60:0/1813984648 shutdown_connections 2024-03-04T12:26:16.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad81989640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c1027c0 0x7fad7c1969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad81989640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad7c101e20 0x7fad7c19ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad81989640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fad680a2010 0x7fad680a44d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad81989640 1 --2- 172.21.15.60:0/1813984648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fad7c100540 0x7fad7c19c7b0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad81989640 1 -- 172.21.15.60:0/1813984648 >> 172.21.15.60:0/1813984648 conn(0x7fad7c077410 msgr2=0x7fad7c104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad81989640 1 -- 172.21.15.60:0/1813984648 shutdown_connections 2024-03-04T12:26:16.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.530+0000 7fad81989640 1 -- 172.21.15.60:0/1813984648 wait complete. 2024-03-04T12:26:16.555 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:26:16.555 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T12:26:16.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.654+0000 7f0ac563d640 1 -- 172.21.15.60:0/73745889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ac0100430 msgr2=0x7f0ac0100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.654+0000 7f0ac563d640 1 --2- 172.21.15.60:0/73745889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ac0100430 0x7f0ac0100810 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009900 tx=0x7f0ab802bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:16.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.654+0000 7f0ac563d640 1 -- 172.21.15.60:0/73745889 shutdown_connections 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.654+0000 7f0ac563d640 1 --2- 172.21.15.60:0/73745889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ac0105500 0x7f0ac010d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.654+0000 7f0ac563d640 1 --2- 172.21.15.60:0/73745889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac0100de0 0x7f0ac0104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.654+0000 7f0ac563d640 1 --2- 172.21.15.60:0/73745889 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ac0100430 0x7f0ac0100810 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.654+0000 7f0ac563d640 1 -- 172.21.15.60:0/73745889 >> 172.21.15.60:0/73745889 conn(0x7f0ac00773b0 msgr2=0x7f0ac00777c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.654+0000 7f0ac563d640 1 -- 172.21.15.60:0/73745889 shutdown_connections 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.654+0000 7f0ac563d640 1 -- 172.21.15.60:0/73745889 wait complete. 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.654+0000 7f0ac563d640 1 Processor -- start 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.654+0000 7f0ac563d640 1 -- start start 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0ac563d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac0100430 0x7f0ac019d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0ac563d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ac0105500 0x7f0ac019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0ac563d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ac0197a10 0x7f0ac0197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0ac563d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01105b0 con 0x7f0ac0197a10 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0ac563d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0110430 con 0x7f0ac0105500 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0ac563d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0110730 con 0x7f0ac0100430 2024-03-04T12:26:16.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0abf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ac0197a10 0x7f0ac0197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0abf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ac0197a10 0x7f0ac0197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53726/0 (socket says 172.21.15.60:53726) 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0abf7fe640 1 -- 172.21.15.60:0/2349976658 learned_addr learned my addr 172.21.15.60:0/2349976658 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0abeffd640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac0100430 0x7f0ac019d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ac0105500 0x7f0ac019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0abf7fe640 1 -- 172.21.15.60:0/2349976658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac0100430 msgr2=0x7f0ac019d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac0100430 0x7f0ac019d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0abf7fe640 1 -- 172.21.15.60:0/2349976658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ac0105500 msgr2=0x7f0ac019dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ac0105500 0x7f0ac019dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0abf7fe640 1 -- 172.21.15.60:0/2349976658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac0198530 con 0x7f0ac0197a10 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0abeffd640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac0100430 0x7f0ac019d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ac0197a10 0x7f0ac0197ec0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000eaa0 tx=0x7f0ab000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0ab7fff640 1 -- 172.21.15.60:0/2349976658 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab000cdb0 con 0x7f0ac0197a10 2024-03-04T12:26:16.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0ab7fff640 1 -- 172.21.15.60:0/2349976658 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ab0005020 con 0x7f0ac0197a10 2024-03-04T12:26:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0ac563d640 1 -- 172.21.15.60:0/2349976658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac0198820 con 0x7f0ac0197a10 2024-03-04T12:26:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0ab7fff640 1 -- 172.21.15.60:0/2349976658 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab00116b0 con 0x7f0ac0197a10 2024-03-04T12:26:16.665 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.655+0000 7f0ac563d640 1 -- 172.21.15.60:0/2349976658 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac01a4a20 con 0x7f0ac0197a10 2024-03-04T12:26:16.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.658+0000 7f0ab7fff640 1 -- 172.21.15.60:0/2349976658 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f0ab0005560 con 0x7f0ac0197a10 2024-03-04T12:26:16.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.658+0000 7f0ac821d740 1 Processor -- start 2024-03-04T12:26:16.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.658+0000 7f0ac821d740 1 -- start start 2024-03-04T12:26:16.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.658+0000 7f0ab7fff640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0aac0a2010 0x7f0aac0a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.658+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0aac0a2010 0x7f0aac0a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.658+0000 7f0ab7fff640 1 -- 172.21.15.60:0/2349976658 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f0ab00cead0 con 0x7f0ac0197a10 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.658+0000 7f0ac821d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee09d0 0x560ac4ee0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.658+0000 7f0ac821d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee12f0 0x560ac4ee9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.658+0000 7f0ac821d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 0x560ac4eec2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.658+0000 7f0ac821d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac4ea7dc0 con 0x560ac4ee9ed0 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.658+0000 7f0ac821d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac4e43ec0 con 0x560ac4ee09d0 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.658+0000 7f0ac821d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac4decd30 con 0x560ac4ee12f0 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0abf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 0x560ac4eec2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0abf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 0x560ac4eec2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53740/0 (socket says 172.21.15.60:53740) 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0abeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee09d0 0x560ac4ee0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0abf7fe640 1 -- 172.21.15.60:0/1042252196 learned_addr learned my addr 172.21.15.60:0/1042252196 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/1042252196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee12f0 0x560ac4ee9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0aac0a2010 0x7f0aac0a44d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0ac006aac0 tx=0x7f0aa8005f90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0abf7fe640 1 -- 172.21.15.60:0/1042252196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee12f0 msgr2=0x560ac4ee9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/1042252196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee12f0 0x560ac4ee9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0abf7fe640 1 -- 172.21.15.60:0/1042252196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee09d0 msgr2=0x560ac4ee0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/1042252196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee09d0 0x560ac4ee0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0abf7fe640 1 -- 172.21.15.60:0/1042252196 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ac4b583f0 con 0x560ac4ee9ed0 2024-03-04T12:26:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/1042252196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 0x560ac4eec2c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f0ab00cbe10 tx=0x7f0ab009b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8a8b258f42220 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0ab57fa640 1 -- 172.21.15.60:0/1042252196 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab001f070 con 0x560ac4ee9ed0 2024-03-04T12:26:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0ab57fa640 1 -- 172.21.15.60:0/1042252196 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ab00051d0 con 0x560ac4ee9ed0 2024-03-04T12:26:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.659+0000 7f0ab57fa640 1 -- 172.21.15.60:0/1042252196 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab009a580 con 0x560ac4ee9ed0 2024-03-04T12:26:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.660+0000 7f0ac821d740 1 -- 172.21.15.60:0/1042252196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 msgr2=0x560ac4eec2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.660+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1042252196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 0x560ac4eec2c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f0ab00cbe10 tx=0x7f0ab009b730 comp rx=0 tx=0).stop 2024-03-04T12:26:16.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.660+0000 7f0ac821d740 1 -- 172.21.15.60:0/1042252196 shutdown_connections 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.660+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1042252196 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 0x560ac4eec2c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.660+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1042252196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee12f0 0x560ac4ee9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.660+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1042252196 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee09d0 0x560ac4ee0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.660+0000 7f0ac821d740 1 -- 172.21.15.60:0/1042252196 >> 172.21.15.60:0/1042252196 conn(0x560ac4edb100 msgr2=0x560ac4edb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.660+0000 7f0ac821d740 1 -- 172.21.15.60:0/1042252196 shutdown_connections 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.660+0000 7f0ac821d740 1 -- 172.21.15.60:0/1042252196 wait complete. 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.660+0000 7f0ac821d740 1 -- start start 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.660+0000 7f0ac821d740 1 Processor -- start 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0ac821d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee09d0 0x560ac4ee6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0ac821d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee12f0 0x560ac4ee7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0ac821d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee9ed0 0x560ac4ee7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0ac821d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac4de24a0 con 0x560ac4ee12f0 2024-03-04T12:26:16.675 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0ac821d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac4e3ff60 con 0x560ac4ee9ed0 2024-03-04T12:26:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0ac821d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac4eecc00 con 0x560ac4ee09d0 2024-03-04T12:26:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0ac821d740 10 client.0 _ll_register_callbacks cb 0x560ac4e3e1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0abe7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee12f0 0x560ac4ee7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0abe7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee12f0 0x560ac4ee7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53750/0 (socket says 172.21.15.60:53750) 2024-03-04T12:26:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0abe7fc640 1 -- 172.21.15.60:0/1789775126 learned_addr learned my addr 172.21.15.60:0/1789775126 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0abeffd640 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee09d0 0x560ac4ee6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee9ed0 0x560ac4ee7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0abe7fc640 1 -- 172.21.15.60:0/1789775126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee09d0 msgr2=0x560ac4ee6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee09d0 0x560ac4ee6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0abe7fc640 1 -- 172.21.15.60:0/1789775126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee9ed0 msgr2=0x560ac4ee7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee9ed0 0x560ac4ee7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0abe7fc640 1 -- 172.21.15.60:0/1789775126 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ac4c227d0 con 0x560ac4ee12f0 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee9ed0 0x560ac4ee7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee12f0 0x560ac4ee7420 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8012bf0 tx=0x7f0aa8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd30652233c156d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0aa67fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0aa67fc640 1 -- 172.21.15.60:0/1789775126 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa801eb40 con 0x560ac4ee12f0 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0aa67fc640 1 -- 172.21.15.60:0/1789775126 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f0aa801ece0 con 0x560ac4ee12f0 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.661+0000 7f0ac821d740 1 -- 172.21.15.60:0/1789775126 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ab00051d0 con 0x560ac4ee12f0 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.662+0000 7f0aa67fc640 1 -- 172.21.15.60:0/1789775126 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0aa8028420 con 0x560ac4ee12f0 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.662+0000 7f0ac821d740 10 client.15945 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.662+0000 7f0ac821d740 1 -- 172.21.15.60:0/1789775126 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x560ac4ef8d60 con 0x560ac4ee12f0 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.662+0000 7f0aa67fc640 1 -- 172.21.15.60:0/1789775126 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ab00051d0 con 0x560ac4ee12f0 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.662+0000 7f0aa67fc640 1 -- 172.21.15.60:0/1789775126 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0aa801d6c0 con 0x560ac4ee12f0 2024-03-04T12:26:16.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.662+0000 7f0ac821d740 10 client.15945 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.662+0000 7f0ac821d740 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.662+0000 7f0ac821d740 20 client.15945 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.662+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560ac4efa270 0x560ac4efc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.662+0000 7f0ac821d740 4 client.15945 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.662+0000 7f0ac821d740 1 -- 172.21.15.60:0/1789775126 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ac4e51400 con 0x560ac4efa270 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.662+0000 7f0abeffd640 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560ac4efa270 0x560ac4efc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.663+0000 7f0aa67fc640 10 client.15945 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.663+0000 7f0abeffd640 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560ac4efa270 0x560ac4efc680 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=579e442750a49fe server_cookie=c1482fbdcf4f1c06 in_seq=0 out_seq=0 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.663+0000 7f0aa67fc640 1 -- 172.21.15.60:0/1789775126 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560ac4e51400 con 0x560ac4efa270 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.663+0000 7f0aa67fc640 10 client.15945 handle_command_reply: tid=0 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.663+0000 7f0ac821d740 1 client.15945 shutdown 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.663+0000 7f0ac821d740 1 -- 172.21.15.60:0/1789775126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee12f0 msgr2=0x560ac4ee7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.663+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee12f0 0x560ac4ee7420 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8012bf0 tx=0x7f0aa8004730 comp rx=0 tx=0).stop 2024-03-04T12:26:16.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.664+0000 7f0ac821d740 1 -- 172.21.15.60:0/1789775126 shutdown_connections 2024-03-04T12:26:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.664+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560ac4efa270 0x560ac4efc680 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.664+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee9ed0 0x560ac4ee7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.664+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee12f0 0x560ac4ee7420 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.664+0000 7f0aa67fc640 0 client.15945 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.664+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1789775126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee09d0 0x560ac4ee6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.664+0000 7f0ac821d740 1 -- 172.21.15.60:0/1789775126 >> 172.21.15.60:0/1789775126 conn(0x560ac4edb100 msgr2=0x560ac4edce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.664+0000 7f0ac821d740 1 -- 172.21.15.60:0/1789775126 shutdown_connections 2024-03-04T12:26:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.664+0000 7f0ac821d740 1 -- 172.21.15.60:0/1789775126 wait complete. 2024-03-04T12:26:16.679 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.664+0000 7f0ac821d740 20 client.15945 trim_cache size 0 max 16384 2024-03-04T12:26:16.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.675+0000 7f0ac821d740 1 Processor -- start 2024-03-04T12:26:16.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.675+0000 7f0ac821d740 1 -- start start 2024-03-04T12:26:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.675+0000 7f0ac821d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee09d0 0x560ac4ee9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.675+0000 7f0ac821d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee12f0 0x560ac4ee2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0ac821d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 0x560ac4ee2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0ac821d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01ae5a0 con 0x560ac4ee9ed0 2024-03-04T12:26:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0ac821d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01aff20 con 0x560ac4ee12f0 2024-03-04T12:26:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0ac821d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01aec00 con 0x560ac4ee09d0 2024-03-04T12:26:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 0x560ac4ee2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 0x560ac4ee2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53760/0 (socket says 172.21.15.60:53760) 2024-03-04T12:26:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abf7fe640 1 -- 172.21.15.60:0/4015226136 learned_addr learned my addr 172.21.15.60:0/4015226136 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abeffd640 1 --2- 172.21.15.60:0/4015226136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee09d0 0x560ac4ee9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abf7fe640 1 -- 172.21.15.60:0/4015226136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee09d0 msgr2=0x560ac4ee9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/4015226136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee09d0 0x560ac4ee9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/4015226136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee12f0 0x560ac4ee2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abf7fe640 1 -- 172.21.15.60:0/4015226136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee12f0 msgr2=0x560ac4ee2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/4015226136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee12f0 0x560ac4ee2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abf7fe640 1 -- 172.21.15.60:0/4015226136 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ac4ee1120 con 0x560ac4ee9ed0 2024-03-04T12:26:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abeffd640 1 --2- 172.21.15.60:0/4015226136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee09d0 0x560ac4ee9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/4015226136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee12f0 0x560ac4ee2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/4015226136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 0x560ac4ee2750 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0ab00a3140 tx=0x7f0ab009b050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc37f9132ca61a75 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0ab4ff9640 1 -- 172.21.15.60:0/4015226136 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab001f040 con 0x560ac4ee9ed0 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0ab4ff9640 1 -- 172.21.15.60:0/4015226136 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ab00051d0 con 0x560ac4ee9ed0 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.676+0000 7f0ab4ff9640 1 -- 172.21.15.60:0/4015226136 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab00cc240 con 0x560ac4ee9ed0 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.677+0000 7f0ac821d740 1 -- 172.21.15.60:0/4015226136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 msgr2=0x560ac4ee2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.677+0000 7f0ac821d740 1 --2- 172.21.15.60:0/4015226136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 0x560ac4ee2750 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0ab00a3140 tx=0x7f0ab009b050 comp rx=0 tx=0).stop 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.677+0000 7f0ac821d740 1 -- 172.21.15.60:0/4015226136 shutdown_connections 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.677+0000 7f0ac821d740 1 --2- 172.21.15.60:0/4015226136 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee9ed0 0x560ac4ee2750 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.677+0000 7f0ac821d740 1 --2- 172.21.15.60:0/4015226136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee12f0 0x560ac4ee2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.677+0000 7f0ac821d740 1 --2- 172.21.15.60:0/4015226136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee09d0 0x560ac4ee9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.677+0000 7f0ac821d740 1 -- 172.21.15.60:0/4015226136 >> 172.21.15.60:0/4015226136 conn(0x560ac4edb100 msgr2=0x560ac4ee16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.677+0000 7f0ac821d740 1 -- 172.21.15.60:0/4015226136 shutdown_connections 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.677+0000 7f0ac821d740 1 -- 172.21.15.60:0/4015226136 wait complete. 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.677+0000 7f0ac821d740 1 -- start start 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.677+0000 7f0ac821d740 1 Processor -- start 2024-03-04T12:26:16.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0ac821d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee09d0 0x560ac4eefa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0ac821d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee12f0 0x560ac4eeffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0ac821d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee9ed0 0x560ac4ef04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0ac821d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01af8c0 con 0x560ac4ee09d0 2024-03-04T12:26:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0ac821d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01af260 con 0x560ac4ee9ed0 2024-03-04T12:26:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0ac821d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01ac840 con 0x560ac4ee12f0 2024-03-04T12:26:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0ac821d740 10 client.0 _ll_register_callbacks cb 0x7f0ab0097210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee09d0 0x560ac4eefa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abeffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee09d0 0x560ac4eefa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53764/0 (socket says 172.21.15.60:53764) 2024-03-04T12:26:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abeffd640 1 -- 172.21.15.60:0/1717387693 learned_addr learned my addr 172.21.15.60:0/1717387693 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:16.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee9ed0 0x560ac4ef04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee12f0 0x560ac4eeffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abeffd640 1 -- 172.21.15.60:0/1717387693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee12f0 msgr2=0x560ac4eeffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abeffd640 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee12f0 0x560ac4eeffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abeffd640 1 -- 172.21.15.60:0/1717387693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee9ed0 msgr2=0x560ac4ef04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abeffd640 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee9ed0 0x560ac4ef04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abeffd640 1 -- 172.21.15.60:0/1717387693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ac4f509d0 con 0x560ac4ee09d0 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee12f0 0x560ac4eeffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abf7fe640 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee9ed0 0x560ac4ef04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0abeffd640 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee09d0 0x560ac4eefa70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009900 tx=0x7f0ab802cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2605752fc8c0edd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0aa67fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0aa67fc640 1 -- 172.21.15.60:0/1717387693 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8045df0 con 0x560ac4ee09d0 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0aa67fc640 1 -- 172.21.15.60:0/1717387693 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f0ab8004030 con 0x560ac4ee09d0 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0ac821d740 1 -- 172.21.15.60:0/1717387693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ab00051d0 con 0x560ac4ee09d0 2024-03-04T12:26:16.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0aa67fc640 1 -- 172.21.15.60:0/1717387693 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ab8054050 con 0x560ac4ee09d0 2024-03-04T12:26:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0ac821d740 10 client.15951 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.678+0000 7f0ac821d740 1 -- 172.21.15.60:0/1717387693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x560ac4f51c80 con 0x560ac4ee09d0 2024-03-04T12:26:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.679+0000 7f0aa67fc640 1 -- 172.21.15.60:0/1717387693 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ab00051d0 con 0x560ac4ee09d0 2024-03-04T12:26:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.679+0000 7f0aa67fc640 1 -- 172.21.15.60:0/1717387693 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0ab8005200 con 0x560ac4ee09d0 2024-03-04T12:26:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.679+0000 7f0ac821d740 10 client.15951 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.679+0000 7f0ac821d740 10 client.15951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.679+0000 7f0ac821d740 20 client.15951 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.679+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560ac4f52380 0x560ac4f547a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.679+0000 7f0ac821d740 4 client.15951 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:26:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.679+0000 7f0ac821d740 1 -- 172.21.15.60:0/1717387693 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560ac4e4b750 con 0x560ac4f52380 2024-03-04T12:26:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.679+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560ac4f52380 0x560ac4f547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:16.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0abe7fc640 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560ac4f52380 0x560ac4f547a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=485a7a38b6b38ea5 server_cookie=3600365f37e350cc in_seq=0 out_seq=0 2024-03-04T12:26:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0aa67fc640 10 client.15951 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0aa67fc640 1 -- 172.21.15.60:0/1717387693 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560ac4e4b750 con 0x560ac4f52380 2024-03-04T12:26:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0aa67fc640 10 client.15951 handle_command_reply: tid=0 2024-03-04T12:26:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0ac821d740 1 client.15951 shutdown 2024-03-04T12:26:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0ac821d740 1 -- 172.21.15.60:0/1717387693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee09d0 msgr2=0x560ac4eefa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee09d0 0x560ac4eefa70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009900 tx=0x7f0ab802cb80 comp rx=0 tx=0).stop 2024-03-04T12:26:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0ac821d740 1 -- 172.21.15.60:0/1717387693 shutdown_connections 2024-03-04T12:26:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560ac4f52380 0x560ac4f547a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ac4ee9ed0 0x560ac4ef04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0aa67fc640 0 client.15951 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560ac4ee12f0 0x560ac4eeffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0ac821d740 1 --2- 172.21.15.60:0/1717387693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560ac4ee09d0 0x560ac4eefa70 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0ac821d740 1 -- 172.21.15.60:0/1717387693 >> 172.21.15.60:0/1717387693 conn(0x560ac4edb100 msgr2=0x560ac4ee2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0ac821d740 1 -- 172.21.15.60:0/1717387693 shutdown_connections 2024-03-04T12:26:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0ac821d740 1 -- 172.21.15.60:0/1717387693 wait complete. 2024-03-04T12:26:16.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.680+0000 7f0ac821d740 20 client.15951 trim_cache size 0 max 16384 2024-03-04T12:26:16.700 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:26:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.681+0000 7f0ac563d640 1 -- 172.21.15.60:0/2349976658 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0aac0a2010 msgr2=0x7f0aac0a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.681+0000 7f0ac563d640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0aac0a2010 0x7f0aac0a44d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0ac006aac0 tx=0x7f0aa8005f90 comp rx=0 tx=0).stop 2024-03-04T12:26:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.681+0000 7f0ac563d640 1 -- 172.21.15.60:0/2349976658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ac0197a10 msgr2=0x7f0ac0197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.681+0000 7f0ac563d640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ac0197a10 0x7f0ac0197ec0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000eaa0 tx=0x7f0ab000ef60 comp rx=0 tx=0).stop 2024-03-04T12:26:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.681+0000 7f0ac563d640 1 -- 172.21.15.60:0/2349976658 shutdown_connections 2024-03-04T12:26:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.681+0000 7f0ac563d640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0aac0a2010 0x7f0aac0a44d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.681+0000 7f0ac563d640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0ac0197a10 0x7f0ac0197ec0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.681+0000 7f0ac563d640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0ac0105500 0x7f0ac019dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.681+0000 7f0ac563d640 1 --2- 172.21.15.60:0/2349976658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac0100430 0x7f0ac019d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.681+0000 7f0ac563d640 1 -- 172.21.15.60:0/2349976658 >> 172.21.15.60:0/2349976658 conn(0x7f0ac00773b0 msgr2=0x7f0ac0104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.681+0000 7f0ac563d640 1 -- 172.21.15.60:0/2349976658 shutdown_connections 2024-03-04T12:26:16.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:16.681+0000 7f0ac563d640 1 -- 172.21.15.60:0/2349976658 wait complete. 2024-03-04T12:26:17.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:17 smithi086 ceph-mon[36851]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-03-04T12:26:17.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:17 smithi063 ceph-mon[36717]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-03-04T12:26:17.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:17 smithi060 ceph-mon[32745]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 25 KiB/s wr, 32 op/s 2024-03-04T12:26:17.704 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T12:26:17.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:26:17.704 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T12:26:17.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.816+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/4085244775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c107a30 msgr2=0x7fbf9c10e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:17.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.816+0000 7fbfa1ad1640 1 --2- 172.21.15.60:0/4085244775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c107a30 0x7fbf9c10e3d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009930 tx=0x7fbf9402bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:17.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.817+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/4085244775 shutdown_connections 2024-03-04T12:26:17.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.817+0000 7fbfa1ad1640 1 --2- 172.21.15.60:0/4085244775 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c107a30 0x7fbf9c10e3d0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.817+0000 7fbfa1ad1640 1 --2- 172.21.15.60:0/4085244775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf9c102ef0 0x7fbf9c107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.817+0000 7fbfa1ad1640 1 --2- 172.21.15.60:0/4085244775 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c102540 0x7fbf9c102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.817+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/4085244775 >> 172.21.15.60:0/4085244775 conn(0x7fbf9c077410 msgr2=0x7fbf9c077820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.817+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/4085244775 shutdown_connections 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.817+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/4085244775 wait complete. 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.817+0000 7fbfa1ad1640 1 Processor -- start 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.817+0000 7fbfa1ad1640 1 -- start start 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbfa1ad1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c102540 0x7fbf9c19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbfa1ad1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c107a30 0x7fbf9c19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbfa1ad1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf9c197b70 0x7fbf9c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbfa1ad1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c111810 con 0x7fbf9c102540 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbfa1ad1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c111690 con 0x7fbf9c107a30 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbfa1ad1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c111990 con 0x7fbf9c197b70 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf9b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c102540 0x7fbf9c19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf9b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c102540 0x7fbf9c19da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53784/0 (socket says 172.21.15.60:53784) 2024-03-04T12:26:17.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf9b7fe640 1 -- 172.21.15.60:0/1714078247 learned_addr learned my addr 172.21.15.60:0/1714078247 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf9bfff640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf9c197b70 0x7fbf9c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf9affd640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c107a30 0x7fbf9c19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf9b7fe640 1 -- 172.21.15.60:0/1714078247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf9c197b70 msgr2=0x7fbf9c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf9c197b70 0x7fbf9c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf9b7fe640 1 -- 172.21.15.60:0/1714078247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c107a30 msgr2=0x7fbf9c19dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c107a30 0x7fbf9c19dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf9b7fe640 1 -- 172.21.15.60:0/1714078247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c198770 con 0x7fbf9c102540 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf9affd640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c107a30 0x7fbf9c19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c102540 0x7fbf9c19da60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00e9f0 tx=0x7fbf8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf98ff9640 1 -- 172.21.15.60:0/1714078247 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c00cd80 con 0x7fbf9c102540 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/1714078247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c1a4800 con 0x7fbf9c102540 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf98ff9640 1 -- 172.21.15.60:0/1714078247 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf8c010910 con 0x7fbf9c102540 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbf98ff9640 1 -- 172.21.15.60:0/1714078247 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0188f0 con 0x7fbf9c102540 2024-03-04T12:26:17.827 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.818+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/1714078247 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c1a4c60 con 0x7fbf9c102540 2024-03-04T12:26:17.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbf98ff9640 1 -- 172.21.15.60:0/1714078247 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7fbf8c010ac0 con 0x7fbf9c102540 2024-03-04T12:26:17.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbfa4794740 1 Processor -- start 2024-03-04T12:26:17.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbf98ff9640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbf840a2010 0x7fbf840a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbfa4794740 1 -- start start 2024-03-04T12:26:17.834 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbf9bfff640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbf840a2010 0x7fbf840a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbf98ff9640 1 -- 172.21.15.60:0/1714078247 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fbf8c0cec60 con 0x7fbf9c102540 2024-03-04T12:26:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbfa4794740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 0x560cfe7de300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbfa4794740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7de840 0x560cfe7e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbfa4794740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7e7420 0x560cfe7e9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbfa4794740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cfe741b40 con 0x560cfe7ddf20 2024-03-04T12:26:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbfa4794740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cfe6ead80 con 0x560cfe7de840 2024-03-04T12:26:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbfa4794740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cfe73dbe0 con 0x560cfe7e7420 2024-03-04T12:26:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.821+0000 7fbf9b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 0x560cfe7de300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf9b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 0x560cfe7de300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53796/0 (socket says 172.21.15.60:53796) 2024-03-04T12:26:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf9b7fe640 1 -- 172.21.15.60:0/4099803693 learned_addr learned my addr 172.21.15.60:0/4099803693 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:17.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf9bfff640 1 --2- 172.21.15.60:0/4099803693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7e7420 0x560cfe7e9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf9affd640 1 --2- 172.21.15.60:0/4099803693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7de840 0x560cfe7e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf9b7fe640 1 -- 172.21.15.60:0/4099803693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7e7420 msgr2=0x560cfe7e9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/4099803693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7e7420 0x560cfe7e9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf9b7fe640 1 -- 172.21.15.60:0/4099803693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7de840 msgr2=0x560cfe7e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/4099803693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7de840 0x560cfe7e6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf9b7fe640 1 -- 172.21.15.60:0/4099803693 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cfe51e7e0 con 0x560cfe7ddf20 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf9bfff640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbf840a2010 0x7fbf840a44d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbf94005940 tx=0x7fbf940058d0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/4099803693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 0x560cfe7de300 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00e9c0 tx=0x7fbf8c09b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d3fe333e5ef134 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf81ffb640 1 -- 172.21.15.60:0/4099803693 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0976d0 con 0x560cfe7ddf20 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf81ffb640 1 -- 172.21.15.60:0/4099803693 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf8c0cc170 con 0x560cfe7ddf20 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.822+0000 7fbf81ffb640 1 -- 172.21.15.60:0/4099803693 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c096db0 con 0x560cfe7ddf20 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.823+0000 7fbfa4794740 1 -- 172.21.15.60:0/4099803693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 msgr2=0x560cfe7de300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:17.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.823+0000 7fbfa4794740 1 --2- 172.21.15.60:0/4099803693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 0x560cfe7de300 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00e9c0 tx=0x7fbf8c09b8a0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.823+0000 7fbfa4794740 1 -- 172.21.15.60:0/4099803693 shutdown_connections 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.823+0000 7fbfa4794740 1 --2- 172.21.15.60:0/4099803693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7e7420 0x560cfe7e9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.823+0000 7fbfa4794740 1 --2- 172.21.15.60:0/4099803693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7de840 0x560cfe7e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.823+0000 7fbfa4794740 1 --2- 172.21.15.60:0/4099803693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 0x560cfe7de300 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.823+0000 7fbfa4794740 1 -- 172.21.15.60:0/4099803693 >> 172.21.15.60:0/4099803693 conn(0x560cfe7d8650 msgr2=0x560cfe7d8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.823+0000 7fbfa4794740 1 -- 172.21.15.60:0/4099803693 shutdown_connections 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.823+0000 7fbfa4794740 1 -- 172.21.15.60:0/4099803693 wait complete. 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.823+0000 7fbfa4794740 1 -- start start 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.823+0000 7fbfa4794740 1 Processor -- start 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbfa4794740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 0x560cfe7e6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbfa4794740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7de840 0x560cfe7e41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbfa4794740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7e7420 0x560cfe7e47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbfa4794740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cfe7ea210 con 0x560cfe7ddf20 2024-03-04T12:26:17.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbfa4794740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cfe6fe1c0 con 0x560cfe7e7420 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbfa4794740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cfe7ea390 con 0x560cfe7de840 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbfa4794740 10 client.0 _ll_register_callbacks cb 0x560cfe6e3f80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf9b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 0x560cfe7e6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf9b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 0x560cfe7e6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53798/0 (socket says 172.21.15.60:53798) 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf9b7fe640 1 -- 172.21.15.60:0/635584160 learned_addr learned my addr 172.21.15.60:0/635584160 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf9affd640 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7de840 0x560cfe7e41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf9bfff640 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7e7420 0x560cfe7e47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf9b7fe640 1 -- 172.21.15.60:0/635584160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7de840 msgr2=0x560cfe7e41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7de840 0x560cfe7e41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf9b7fe640 1 -- 172.21.15.60:0/635584160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7e7420 msgr2=0x560cfe7e47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7e7420 0x560cfe7e47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf9b7fe640 1 -- 172.21.15.60:0/635584160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560cfe57ef70 con 0x560cfe7ddf20 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf9bfff640 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7e7420 0x560cfe7e47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 0x560cfe7e6e10 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c09aff0 tx=0x7fbf8c0cc550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3806333b5e02fe77 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:17.838 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf7affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf7affd640 1 -- 172.21.15.60:0/635584160 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0b7020 con 0x560cfe7ddf20 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.824+0000 7fbf7affd640 1 -- 172.21.15.60:0/635584160 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fbf8c095bd0 con 0x560cfe7ddf20 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbfa4794740 1 -- 172.21.15.60:0/635584160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf8c0cc170 con 0x560cfe7ddf20 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbf7affd640 1 -- 172.21.15.60:0/635584160 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf8c095e50 con 0x560cfe7ddf20 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbfa4794740 10 client.15963 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbfa4794740 1 -- 172.21.15.60:0/635584160 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x560cfe416170 con 0x560cfe7ddf20 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbf7affd640 1 -- 172.21.15.60:0/635584160 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf8c0cc170 con 0x560cfe7ddf20 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbf7affd640 1 -- 172.21.15.60:0/635584160 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbf8c099f20 con 0x560cfe7ddf20 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbfa4794740 10 client.15963 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbfa4794740 10 client.15963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbfa4794740 20 client.15963 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbfa4794740 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560cfe7f7400 0x560cfe7f97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbfa4794740 4 client.15963 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbfa4794740 1 -- 172.21.15.60:0/635584160 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560cfe717960 con 0x560cfe7f7400 2024-03-04T12:26:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.825+0000 7fbf9affd640 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560cfe7f7400 0x560cfe7f97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.826+0000 7fbf7affd640 10 client.15963 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.826+0000 7fbf9affd640 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560cfe7f7400 0x560cfe7f97f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebbd102963efac71 server_cookie=a28013c98328889f in_seq=0 out_seq=0 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.826+0000 7fbf7affd640 1 -- 172.21.15.60:0/635584160 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560cfe717960 con 0x560cfe7f7400 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.826+0000 7fbf7affd640 10 client.15963 handle_command_reply: tid=0 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.826+0000 7fbfa4794740 1 client.15963 shutdown 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.827+0000 7fbfa4794740 1 -- 172.21.15.60:0/635584160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 msgr2=0x560cfe7e6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.827+0000 7fbfa4794740 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 0x560cfe7e6e10 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c09aff0 tx=0x7fbf8c0cc550 comp rx=0 tx=0).stop 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.827+0000 7fbfa4794740 1 -- 172.21.15.60:0/635584160 shutdown_connections 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.827+0000 7fbfa4794740 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560cfe7f7400 0x560cfe7f97f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.827+0000 7fbfa4794740 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7e7420 0x560cfe7e47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.827+0000 7fbfa4794740 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7de840 0x560cfe7e41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.827+0000 7fbf7affd640 0 client.15963 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.827+0000 7fbfa4794740 1 --2- 172.21.15.60:0/635584160 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7ddf20 0x560cfe7e6e10 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.827+0000 7fbfa4794740 1 -- 172.21.15.60:0/635584160 >> 172.21.15.60:0/635584160 conn(0x560cfe7d8650 msgr2=0x560cfe7da3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.827+0000 7fbfa4794740 1 -- 172.21.15.60:0/635584160 shutdown_connections 2024-03-04T12:26:17.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.827+0000 7fbfa4794740 1 -- 172.21.15.60:0/635584160 wait complete. 2024-03-04T12:26:17.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.827+0000 7fbfa4794740 20 client.15963 trim_cache size 0 max 16384 2024-03-04T12:26:17.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.842+0000 7fbfa4794740 1 Processor -- start 2024-03-04T12:26:17.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.842+0000 7fbfa4794740 1 -- start start 2024-03-04T12:26:17.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.842+0000 7fbfa4794740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7ddf20 0x560cfe7fde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.860 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.842+0000 7fbfa4794740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 0x560cfe7fe3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.842+0000 7fbfa4794740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7e7420 0x560cfe7e2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.842+0000 7fbfa4794740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1ae470 con 0x560cfe7de840 2024-03-04T12:26:17.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.842+0000 7fbfa4794740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1afdf0 con 0x560cfe7e7420 2024-03-04T12:26:17.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.842+0000 7fbfa4794740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1aead0 con 0x560cfe7ddf20 2024-03-04T12:26:17.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.842+0000 7fbf9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 0x560cfe7fe3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.861 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.842+0000 7fbf9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 0x560cfe7fe3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53814/0 (socket says 172.21.15.60:53814) 2024-03-04T12:26:17.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.842+0000 7fbf9affd640 1 -- 172.21.15.60:0/2621718782 learned_addr learned my addr 172.21.15.60:0/2621718782 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:17.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.842+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/2621718782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7ddf20 0x560cfe7fde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.843+0000 7fbf9bfff640 1 --2- 172.21.15.60:0/2621718782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7e7420 0x560cfe7e2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.843+0000 7fbf9affd640 1 -- 172.21.15.60:0/2621718782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7ddf20 msgr2=0x560cfe7fde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:17.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.843+0000 7fbf9affd640 1 --2- 172.21.15.60:0/2621718782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7ddf20 0x560cfe7fde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.862 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.843+0000 7fbf9affd640 1 -- 172.21.15.60:0/2621718782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7e7420 msgr2=0x560cfe7e2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:17.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.843+0000 7fbf9affd640 1 --2- 172.21.15.60:0/2621718782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7e7420 0x560cfe7e2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.843+0000 7fbf9affd640 1 -- 172.21.15.60:0/2621718782 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cfe7e4b20 con 0x560cfe7de840 2024-03-04T12:26:17.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.843+0000 7fbf9affd640 1 --2- 172.21.15.60:0/2621718782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 0x560cfe7fe3b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fbf90000f80 tx=0x7fbf90014a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e434a606e8310d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:17.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.843+0000 7fbf817fa640 1 -- 172.21.15.60:0/2621718782 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9001d4d0 con 0x560cfe7de840 2024-03-04T12:26:17.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.843+0000 7fbf817fa640 1 -- 172.21.15.60:0/2621718782 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf90021070 con 0x560cfe7de840 2024-03-04T12:26:17.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.843+0000 7fbf817fa640 1 -- 172.21.15.60:0/2621718782 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf90012450 con 0x560cfe7de840 2024-03-04T12:26:17.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 -- 172.21.15.60:0/2621718782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 msgr2=0x560cfe7fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:17.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 --2- 172.21.15.60:0/2621718782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 0x560cfe7fe3b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fbf90000f80 tx=0x7fbf90014a40 comp rx=0 tx=0).stop 2024-03-04T12:26:17.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 -- 172.21.15.60:0/2621718782 shutdown_connections 2024-03-04T12:26:17.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 --2- 172.21.15.60:0/2621718782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7e7420 0x560cfe7e2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 --2- 172.21.15.60:0/2621718782 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 0x560cfe7fe3b0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.864 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 --2- 172.21.15.60:0/2621718782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7ddf20 0x560cfe7fde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 -- 172.21.15.60:0/2621718782 >> 172.21.15.60:0/2621718782 conn(0x560cfe7d8650 msgr2=0x560cfe7df480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:17.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 -- 172.21.15.60:0/2621718782 shutdown_connections 2024-03-04T12:26:17.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 -- 172.21.15.60:0/2621718782 wait complete. 2024-03-04T12:26:17.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 -- start start 2024-03-04T12:26:17.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 Processor -- start 2024-03-04T12:26:17.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7ddf20 0x560cfe7fe270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 0x560cfe7fa640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7e7420 0x560cfe7fabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1af790 con 0x560cfe7de840 2024-03-04T12:26:17.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1af130 con 0x560cfe7ddf20 2024-03-04T12:26:17.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c1ac710 con 0x560cfe7e7420 2024-03-04T12:26:17.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.844+0000 7fbfa4794740 10 client.0 _ll_register_callbacks cb 0x7fbf8c096cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:17.866 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 0x560cfe7fa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf9affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 0x560cfe7fa640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53830/0 (socket says 172.21.15.60:53830) 2024-03-04T12:26:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf9affd640 1 -- 172.21.15.60:0/905866282 learned_addr learned my addr 172.21.15.60:0/905866282 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf9bfff640 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7e7420 0x560cfe7fabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7ddf20 0x560cfe7fe270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf9affd640 1 -- 172.21.15.60:0/905866282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7e7420 msgr2=0x560cfe7fabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:17.867 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf9affd640 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7e7420 0x560cfe7fabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf9affd640 1 -- 172.21.15.60:0/905866282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7ddf20 msgr2=0x560cfe7fe270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:17.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf9affd640 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7ddf20 0x560cfe7fe270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf9affd640 1 -- 172.21.15.60:0/905866282 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560cfe7f9b60 con 0x560cfe7de840 2024-03-04T12:26:17.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf9affd640 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 0x560cfe7fa640 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fbf90009370 tx=0x7fbf900142f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71bae377826da3f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:17.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf7affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:17.868 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf7affd640 1 -- 172.21.15.60:0/905866282 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf90033060 con 0x560cfe7de840 2024-03-04T12:26:17.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbfa4794740 1 -- 172.21.15.60:0/905866282 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf90021070 con 0x560cfe7de840 2024-03-04T12:26:17.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf7affd640 1 -- 172.21.15.60:0/905866282 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fbf90004c80 con 0x560cfe7de840 2024-03-04T12:26:17.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbf7affd640 1 -- 172.21.15.60:0/905866282 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf9003a050 con 0x560cfe7de840 2024-03-04T12:26:17.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbfa4794740 10 client.15969 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:17.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.845+0000 7fbfa4794740 1 -- 172.21.15.60:0/905866282 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x560cfe84e6a0 con 0x560cfe7de840 2024-03-04T12:26:17.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.846+0000 7fbf7affd640 1 -- 172.21.15.60:0/905866282 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf90021070 con 0x560cfe7de840 2024-03-04T12:26:17.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.846+0000 7fbf7affd640 1 -- 172.21.15.60:0/905866282 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbf90005090 con 0x560cfe7de840 2024-03-04T12:26:17.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.846+0000 7fbfa4794740 10 client.15969 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:17.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.846+0000 7fbfa4794740 10 client.15969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:17.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.846+0000 7fbfa4794740 20 client.15969 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:17.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.846+0000 7fbfa4794740 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560cfe84ed20 0x560cfe851140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:17.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.846+0000 7fbfa4794740 4 client.15969 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T12:26:17.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.846+0000 7fbfa4794740 1 -- 172.21.15.60:0/905866282 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x560cfe703080 con 0x560cfe84ed20 2024-03-04T12:26:17.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.846+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560cfe84ed20 0x560cfe851140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:17.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.846+0000 7fbf9b7fe640 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560cfe84ed20 0x560cfe851140 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8839f321085547f server_cookie=a1a9d600a39e9c4 in_seq=0 out_seq=0 2024-03-04T12:26:17.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.846+0000 7fbf7affd640 10 client.15969 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:17.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.847+0000 7fbf7affd640 1 -- 172.21.15.60:0/905866282 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x560cfe703080 con 0x560cfe84ed20 2024-03-04T12:26:17.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.847+0000 7fbf7affd640 10 client.15969 handle_command_reply: tid=0 2024-03-04T12:26:17.871 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.847+0000 7fbfa4794740 1 client.15969 shutdown 2024-03-04T12:26:17.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa4794740 1 -- 172.21.15.60:0/905866282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 msgr2=0x560cfe7fa640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:17.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa4794740 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 0x560cfe7fa640 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fbf90009370 tx=0x7fbf900142f0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa4794740 1 -- 172.21.15.60:0/905866282 shutdown_connections 2024-03-04T12:26:17.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa4794740 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x560cfe84ed20 0x560cfe851140 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.872 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa4794740 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560cfe7e7420 0x560cfe7fabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa4794740 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x560cfe7de840 0x560cfe7fa640 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa4794740 1 --2- 172.21.15.60:0/905866282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cfe7ddf20 0x560cfe7fe270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa4794740 1 -- 172.21.15.60:0/905866282 >> 172.21.15.60:0/905866282 conn(0x560cfe7d8650 msgr2=0x560cfe7e2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:17.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbf7affd640 0 client.15969 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:17.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa4794740 1 -- 172.21.15.60:0/905866282 shutdown_connections 2024-03-04T12:26:17.873 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa4794740 1 -- 172.21.15.60:0/905866282 wait complete. 2024-03-04T12:26:17.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa4794740 20 client.15969 trim_cache size 0 max 16384 2024-03-04T12:26:17.874 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:26:17.874 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": 0, 2024-03-04T12:26:17.874 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_tag": "231e9210-94ef-403f-9cd9-fd6882a93c83", 2024-03-04T12:26:17.875 INFO:teuthology.orchestra.run.smithi060.stdout: "mode": "asynchronous" 2024-03-04T12:26:17.875 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:26:17.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/1714078247 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbf840a2010 msgr2=0x7fbf840a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:17.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa1ad1640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbf840a2010 0x7fbf840a44d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbf94005940 tx=0x7fbf940058d0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/1714078247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c102540 msgr2=0x7fbf9c19da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:17.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa1ad1640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c102540 0x7fbf9c19da60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00e9f0 tx=0x7fbf8c00eeb0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/1714078247 shutdown_connections 2024-03-04T12:26:17.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa1ad1640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf9c197b70 0x7fbf9c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa1ad1640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf9c107a30 0x7fbf9c19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa1ad1640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbf840a2010 0x7fbf840a44d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa1ad1640 1 --2- 172.21.15.60:0/1714078247 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbf9c102540 0x7fbf9c19da60 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:17.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.848+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/1714078247 >> 172.21.15.60:0/1714078247 conn(0x7fbf9c077410 msgr2=0x7fbf9c0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:17.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.849+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/1714078247 shutdown_connections 2024-03-04T12:26:17.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.849+0000 7fbfa1ad1640 1 -- 172.21.15.60:0/1714078247 wait complete. 2024-03-04T12:26:17.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T12:26:17.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T12:26:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.978+0000 7fd30299d640 1 -- 172.21.15.60:0/4137960946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2fc1014f0 msgr2=0x7fd2fc1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.978+0000 7fd30299d640 1 --2- 172.21.15.60:0/4137960946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2fc1014f0 0x7fd2fc1018d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009900 tx=0x7fd2ec02bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.978+0000 7fd30299d640 1 -- 172.21.15.60:0/4137960946 shutdown_connections 2024-03-04T12:26:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.978+0000 7fd30299d640 1 --2- 172.21.15.60:0/4137960946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2fc1065c0 0x7fd2fc10e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.978+0000 7fd30299d640 1 --2- 172.21.15.60:0/4137960946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2fc101ea0 0x7fd2fc105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.978+0000 7fd30299d640 1 --2- 172.21.15.60:0/4137960946 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2fc1014f0 0x7fd2fc1018d0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.978+0000 7fd30299d640 1 -- 172.21.15.60:0/4137960946 >> 172.21.15.60:0/4137960946 conn(0x7fd2fc0fcd50 msgr2=0x7fd2fc0ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:18.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.978+0000 7fd30299d640 1 -- 172.21.15.60:0/4137960946 shutdown_connections 2024-03-04T12:26:18.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.978+0000 7fd30299d640 1 -- 172.21.15.60:0/4137960946 wait complete. 2024-03-04T12:26:18.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd30299d640 1 Processor -- start 2024-03-04T12:26:18.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd30299d640 1 -- start start 2024-03-04T12:26:18.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd30299d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2fc1014f0 0x7fd2fc19d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd30299d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2fc1065c0 0x7fd2fc19dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd30299d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2fc197a50 0x7fd2fc197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd30299d640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc111670 con 0x7fd2fc1014f0 2024-03-04T12:26:18.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd30299d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc1114f0 con 0x7fd2fc1065c0 2024-03-04T12:26:18.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd30299d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc1117f0 con 0x7fd2fc197a50 2024-03-04T12:26:18.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd2fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2fc1014f0 0x7fd2fc19d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd2fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2fc1014f0 0x7fd2fc19d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53846/0 (socket says 172.21.15.60:53846) 2024-03-04T12:26:18.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd2fbfff640 1 -- 172.21.15.60:0/1264432780 learned_addr learned my addr 172.21.15.60:0/1264432780 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:18.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd300f3a640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2fc197a50 0x7fd2fc197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd2fb7fe640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2fc1065c0 0x7fd2fc19dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd2fbfff640 1 -- 172.21.15.60:0/1264432780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2fc197a50 msgr2=0x7fd2fc197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:18.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.979+0000 7fd2fbfff640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2fc197a50 0x7fd2fc197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.980+0000 7fd2fbfff640 1 -- 172.21.15.60:0/1264432780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2fc1065c0 msgr2=0x7fd2fc19dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:18.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.980+0000 7fd2fbfff640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2fc1065c0 0x7fd2fc19dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.980+0000 7fd2fbfff640 1 -- 172.21.15.60:0/1264432780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc198570 con 0x7fd2fc1014f0 2024-03-04T12:26:18.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.980+0000 7fd300f3a640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2fc197a50 0x7fd2fc197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:18.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.980+0000 7fd2fbfff640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2fc1014f0 0x7fd2fc19d850 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec006010 tx=0x7fd2ec02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.980+0000 7fd2f97fa640 1 -- 172.21.15.60:0/1264432780 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec041070 con 0x7fd2fc1014f0 2024-03-04T12:26:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.980+0000 7fd2f97fa640 1 -- 172.21.15.60:0/1264432780 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2ec004450 con 0x7fd2fc1014f0 2024-03-04T12:26:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.980+0000 7fd30299d640 1 -- 172.21.15.60:0/1264432780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2fc198800 con 0x7fd2fc1014f0 2024-03-04T12:26:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.980+0000 7fd2f97fa640 1 -- 172.21.15.60:0/1264432780 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec003e00 con 0x7fd2fc1014f0 2024-03-04T12:26:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.980+0000 7fd30299d640 1 -- 172.21.15.60:0/1264432780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2fc1a49a0 con 0x7fd2fc1014f0 2024-03-04T12:26:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.982+0000 7fd2f97fa640 1 -- 172.21.15.60:0/1264432780 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7fd2ec044070 con 0x7fd2fc1014f0 2024-03-04T12:26:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.982+0000 7fd305576740 1 Processor -- start 2024-03-04T12:26:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.982+0000 7fd305576740 1 -- start start 2024-03-04T12:26:18.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.982+0000 7fd305576740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 0x5597be22a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.982+0000 7fd305576740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22ade0 0x5597be237b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.982+0000 7fd305576740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be2380a0 0x5597be23a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.982+0000 7fd305576740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597be18de00 con 0x5597be22ade0 2024-03-04T12:26:18.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.982+0000 7fd305576740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597be136b10 con 0x5597be2380a0 2024-03-04T12:26:18.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.982+0000 7fd305576740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597be189ea0 con 0x5597be22a4c0 2024-03-04T12:26:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.982+0000 7fd2f97fa640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd2e80a2010 0x7fd2e80a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22ade0 0x5597be237b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd300f3a640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd2e80a2010 0x7fd2e80a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22ade0 0x5597be237b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53856/0 (socket says 172.21.15.60:53856) 2024-03-04T12:26:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2fbfff640 1 -- 172.21.15.60:0/1347857641 learned_addr learned my addr 172.21.15.60:0/1347857641 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:18.009 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd300f3a640 1 --2- 172.21.15.60:0/1347857641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 0x5597be22a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2fb7fe640 1 --2- 172.21.15.60:0/1347857641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be2380a0 0x5597be23a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2f97fa640 1 -- 172.21.15.60:0/1264432780 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fd2ec0f01a0 con 0x7fd2fc1014f0 2024-03-04T12:26:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2fbfff640 1 -- 172.21.15.60:0/1347857641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 msgr2=0x5597be22a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2fbfff640 1 --2- 172.21.15.60:0/1347857641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 0x5597be22a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2fbfff640 1 -- 172.21.15.60:0/1347857641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be2380a0 msgr2=0x5597be23a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2fbfff640 1 --2- 172.21.15.60:0/1347857641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be2380a0 0x5597be23a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2fbfff640 1 -- 172.21.15.60:0/1347857641 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597bdf6c7d0 con 0x5597be22ade0 2024-03-04T12:26:18.010 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd300f3a640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd2e80a2010 0x7fd2e80a44d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4005fd0 tx=0x7fd2f4011690 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:18.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2fbfff640 1 --2- 172.21.15.60:0/1347857641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22ade0 0x5597be237b60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec000c00 tx=0x7fd2ec08f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca33d4eaf8f32cd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:18.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2f27fc640 1 -- 172.21.15.60:0/1347857641 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec0c16c0 con 0x5597be22ade0 2024-03-04T12:26:18.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2f27fc640 1 -- 172.21.15.60:0/1347857641 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2ec02cd20 con 0x5597be22ade0 2024-03-04T12:26:18.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.983+0000 7fd2f27fc640 1 -- 172.21.15.60:0/1347857641 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec0c4df0 con 0x5597be22ade0 2024-03-04T12:26:18.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.984+0000 7fd305576740 1 -- 172.21.15.60:0/1347857641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22ade0 msgr2=0x5597be237b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:18.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.984+0000 7fd305576740 1 --2- 172.21.15.60:0/1347857641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22ade0 0x5597be237b60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec000c00 tx=0x7fd2ec08f770 comp rx=0 tx=0).stop 2024-03-04T12:26:18.011 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.984+0000 7fd305576740 1 -- 172.21.15.60:0/1347857641 shutdown_connections 2024-03-04T12:26:18.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.984+0000 7fd305576740 1 --2- 172.21.15.60:0/1347857641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be2380a0 0x5597be23a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.984+0000 7fd305576740 1 --2- 172.21.15.60:0/1347857641 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22ade0 0x5597be237b60 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.984+0000 7fd305576740 1 --2- 172.21.15.60:0/1347857641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 0x5597be22a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.984+0000 7fd305576740 1 -- 172.21.15.60:0/1347857641 >> 172.21.15.60:0/1347857641 conn(0x5597be224ec0 msgr2=0x5597be2252a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:18.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.984+0000 7fd305576740 1 -- 172.21.15.60:0/1347857641 shutdown_connections 2024-03-04T12:26:18.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.984+0000 7fd305576740 1 -- 172.21.15.60:0/1347857641 wait complete. 2024-03-04T12:26:18.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.984+0000 7fd305576740 1 -- start start 2024-03-04T12:26:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd305576740 1 Processor -- start 2024-03-04T12:26:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd305576740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 0x5597be237810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd305576740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 0x5597be234ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd305576740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 0x5597be2350e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd305576740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597be23ac20 con 0x5597be2380a0 2024-03-04T12:26:18.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd305576740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597be14a0a0 con 0x5597be22ade0 2024-03-04T12:26:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd305576740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597be23ada0 con 0x5597be22a4c0 2024-03-04T12:26:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd305576740 10 client.0 _ll_register_callbacks cb 0x5597be149a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd300f3a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 0x5597be2350e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd2fbfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 0x5597be237810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd300f3a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 0x5597be2350e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53868/0 (socket says 172.21.15.60:53868) 2024-03-04T12:26:18.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd300f3a640 1 -- 172.21.15.60:0/645092342 learned_addr learned my addr 172.21.15.60:0/645092342 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:18.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd2fb7fe640 1 --2- 172.21.15.60:0/645092342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 0x5597be234ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd300f3a640 1 -- 172.21.15.60:0/645092342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 msgr2=0x5597be237810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:18.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd300f3a640 1 --2- 172.21.15.60:0/645092342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 0x5597be237810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd300f3a640 1 -- 172.21.15.60:0/645092342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 msgr2=0x5597be234ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:18.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd300f3a640 1 --2- 172.21.15.60:0/645092342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 0x5597be234ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd300f3a640 1 -- 172.21.15.60:0/645092342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5597bdea23f0 con 0x5597be2380a0 2024-03-04T12:26:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd300f3a640 1 --2- 172.21.15.60:0/645092342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 0x5597be2350e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4003e20 tx=0x7fd2f401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530bc28b05f4b8f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.985+0000 7fd2db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd2db7fe640 1 -- 172.21.15.60:0/645092342 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4024070 con 0x5597be2380a0 2024-03-04T12:26:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd2db7fe640 1 -- 172.21.15.60:0/645092342 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fd2f401f660 con 0x5597be2380a0 2024-03-04T12:26:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd305576740 1 -- 172.21.15.60:0/645092342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd2ec02cd20 con 0x5597be2380a0 2024-03-04T12:26:18.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd2db7fe640 1 -- 172.21.15.60:0/645092342 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2f4033050 con 0x5597be2380a0 2024-03-04T12:26:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd305576740 10 client.15981 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd305576740 1 -- 172.21.15.60:0/645092342 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5597bde620f0 con 0x5597be2380a0 2024-03-04T12:26:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd2db7fe640 1 -- 172.21.15.60:0/645092342 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2ec02cd20 con 0x5597be2380a0 2024-03-04T12:26:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd2db7fe640 1 -- 172.21.15.60:0/645092342 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd2f4028850 con 0x5597be2380a0 2024-03-04T12:26:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd305576740 10 client.15981 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd305576740 10 client.15981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd305576740 20 client.15981 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd305576740 1 --2- 172.21.15.60:0/645092342 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5597be2478c0 0x5597be249cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd305576740 4 client.15981 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:18.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd305576740 1 -- 172.21.15.60:0/645092342 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5597be19b340 con 0x5597be2478c0 2024-03-04T12:26:18.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.986+0000 7fd2fbfff640 1 --2- 172.21.15.60:0/645092342 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5597be2478c0 0x5597be249cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.987+0000 7fd2db7fe640 10 client.15981 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:18.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.987+0000 7fd2fbfff640 1 --2- 172.21.15.60:0/645092342 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5597be2478c0 0x5597be249cb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5de59af9a2e2718 server_cookie=a9ea3281758d996a in_seq=0 out_seq=0 2024-03-04T12:26:18.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.987+0000 7fd2db7fe640 1 -- 172.21.15.60:0/645092342 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5597be19b340 con 0x5597be2478c0 2024-03-04T12:26:18.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.987+0000 7fd2db7fe640 10 client.15981 handle_command_reply: tid=0 2024-03-04T12:26:18.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.987+0000 7fd305576740 1 client.15981 shutdown 2024-03-04T12:26:18.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.988+0000 7fd305576740 1 -- 172.21.15.60:0/645092342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 msgr2=0x5597be2350e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:18.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.988+0000 7fd305576740 1 --2- 172.21.15.60:0/645092342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 0x5597be2350e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4003e20 tx=0x7fd2f401d000 comp rx=0 tx=0).stop 2024-03-04T12:26:18.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.988+0000 7fd305576740 1 -- 172.21.15.60:0/645092342 shutdown_connections 2024-03-04T12:26:18.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.988+0000 7fd305576740 1 --2- 172.21.15.60:0/645092342 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5597be2478c0 0x5597be249cb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.988+0000 7fd305576740 1 --2- 172.21.15.60:0/645092342 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 0x5597be2350e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.988+0000 7fd305576740 1 --2- 172.21.15.60:0/645092342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 0x5597be234ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.988+0000 7fd2db7fe640 0 client.15981 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:18.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.988+0000 7fd305576740 1 --2- 172.21.15.60:0/645092342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 0x5597be237810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.988+0000 7fd305576740 1 -- 172.21.15.60:0/645092342 >> 172.21.15.60:0/645092342 conn(0x5597be224ec0 msgr2=0x5597be226c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:18.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.988+0000 7fd305576740 1 -- 172.21.15.60:0/645092342 shutdown_connections 2024-03-04T12:26:18.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.988+0000 7fd305576740 1 -- 172.21.15.60:0/645092342 wait complete. 2024-03-04T12:26:18.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:17.988+0000 7fd305576740 20 client.15981 trim_cache size 0 max 16384 2024-03-04T12:26:18.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd305576740 1 Processor -- start 2024-03-04T12:26:18.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd305576740 1 -- start start 2024-03-04T12:26:18.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd305576740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 0x5597be2372d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd305576740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 0x5597be237810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.026 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd305576740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 0x5597be233470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd305576740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc1ac2a0 con 0x5597be2380a0 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd305576740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc1adc20 con 0x5597be22ade0 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd305576740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc1ac900 con 0x5597be22a4c0 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd300f3a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 0x5597be233470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd300f3a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 0x5597be233470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53874/0 (socket says 172.21.15.60:53874) 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd300f3a640 1 -- 172.21.15.60:0/3707593667 learned_addr learned my addr 172.21.15.60:0/3707593667 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd2fbfff640 1 --2- 172.21.15.60:0/3707593667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 0x5597be2372d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd2fb7fe640 1 --2- 172.21.15.60:0/3707593667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 0x5597be237810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd300f3a640 1 -- 172.21.15.60:0/3707593667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 msgr2=0x5597be2372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd300f3a640 1 --2- 172.21.15.60:0/3707593667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 0x5597be2372d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd300f3a640 1 -- 172.21.15.60:0/3707593667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 msgr2=0x5597be237810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd300f3a640 1 --2- 172.21.15.60:0/3707593667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 0x5597be237810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd300f3a640 1 -- 172.21.15.60:0/3707593667 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597be24eec0 con 0x5597be2380a0 2024-03-04T12:26:18.027 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd300f3a640 1 --2- 172.21.15.60:0/3707593667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 0x5597be233470 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400ffc0 tx=0x7fd2f400fcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f259152ea39ab5aa server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd2f1ffb640 1 -- 172.21.15.60:0/3707593667 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4024050 con 0x5597be2380a0 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd2f1ffb640 1 -- 172.21.15.60:0/3707593667 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2f4003e50 con 0x5597be2380a0 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.003+0000 7fd2f1ffb640 1 -- 172.21.15.60:0/3707593667 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40289c0 con 0x5597be2380a0 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.004+0000 7fd305576740 1 -- 172.21.15.60:0/3707593667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 msgr2=0x5597be233470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.004+0000 7fd305576740 1 --2- 172.21.15.60:0/3707593667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 0x5597be233470 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400ffc0 tx=0x7fd2f400fcf0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.004+0000 7fd305576740 1 -- 172.21.15.60:0/3707593667 shutdown_connections 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.004+0000 7fd305576740 1 --2- 172.21.15.60:0/3707593667 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be2380a0 0x5597be233470 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.004+0000 7fd305576740 1 --2- 172.21.15.60:0/3707593667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 0x5597be237810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.004+0000 7fd305576740 1 --2- 172.21.15.60:0/3707593667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be22a4c0 0x5597be2372d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.004+0000 7fd305576740 1 -- 172.21.15.60:0/3707593667 >> 172.21.15.60:0/3707593667 conn(0x5597be224ec0 msgr2=0x5597be234b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.004+0000 7fd305576740 1 -- 172.21.15.60:0/3707593667 shutdown_connections 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.004+0000 7fd305576740 1 -- 172.21.15.60:0/3707593667 wait complete. 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.004+0000 7fd305576740 1 -- start start 2024-03-04T12:26:18.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd305576740 1 Processor -- start 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd305576740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22a4c0 0x5597be23b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd305576740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 0x5597be23b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd305576740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be2380a0 0x5597be23e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd305576740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc1ad5c0 con 0x5597be22a4c0 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd305576740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc1acf60 con 0x5597be22ade0 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd305576740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2fc1aa540 con 0x5597be2380a0 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd305576740 10 client.0 _ll_register_callbacks cb 0x7fd2f4003c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22a4c0 0x5597be23b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2fbfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22a4c0 0x5597be23b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53878/0 (socket says 172.21.15.60:53878) 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2fbfff640 1 -- 172.21.15.60:0/2217411039 learned_addr learned my addr 172.21.15.60:0/2217411039 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd300f3a640 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be2380a0 0x5597be23e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2fb7fe640 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 0x5597be23b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2fbfff640 1 -- 172.21.15.60:0/2217411039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be2380a0 msgr2=0x5597be23e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:18.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2fbfff640 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be2380a0 0x5597be23e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2fbfff640 1 -- 172.21.15.60:0/2217411039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 msgr2=0x5597be23b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2fbfff640 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 0x5597be23b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2fbfff640 1 -- 172.21.15.60:0/2217411039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5597be23ee20 con 0x5597be22a4c0 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd300f3a640 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be2380a0 0x5597be23e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2fbfff640 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22a4c0 0x5597be23b3a0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0b8b60 tx=0x7fd2ec0f0420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d31e48a785f4f22 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2db7fe640 1 -- 172.21.15.60:0/2217411039 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec04b040 con 0x5597be22a4c0 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd2db7fe640 1 -- 172.21.15.60:0/2217411039 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fd2ec0ba7a0 con 0x5597be22a4c0 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.005+0000 7fd305576740 1 -- 172.21.15.60:0/2217411039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd2f4003e50 con 0x5597be22a4c0 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.006+0000 7fd2db7fe640 1 -- 172.21.15.60:0/2217411039 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2ec0d5180 con 0x5597be22a4c0 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.006+0000 7fd305576740 10 client.15987 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.006+0000 7fd305576740 1 -- 172.21.15.60:0/2217411039 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5597be29e810 con 0x5597be22a4c0 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.006+0000 7fd2db7fe640 1 -- 172.21.15.60:0/2217411039 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2f4003e50 con 0x5597be22a4c0 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.006+0000 7fd2db7fe640 1 -- 172.21.15.60:0/2217411039 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd2ec0b8d70 con 0x5597be22a4c0 2024-03-04T12:26:18.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.006+0000 7fd305576740 10 client.15987 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.006+0000 7fd305576740 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.006+0000 7fd305576740 20 client.15987 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.006+0000 7fd305576740 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5597be29edb0 0x5597be2a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.006+0000 7fd305576740 4 client.15987 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.006+0000 7fd305576740 1 -- 172.21.15.60:0/2217411039 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5597be195690 con 0x5597be29edb0 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.006+0000 7fd2fb7fe640 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5597be29edb0 0x5597be2a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd2fb7fe640 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5597be29edb0 0x5597be2a11d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=512ec61366e29602 server_cookie=cf487d4f03170af2 in_seq=0 out_seq=0 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd2db7fe640 10 client.15987 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd2db7fe640 1 -- 172.21.15.60:0/2217411039 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5597be195690 con 0x5597be29edb0 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd2db7fe640 10 client.15987 handle_command_reply: tid=0 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd305576740 1 client.15987 shutdown 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd305576740 1 -- 172.21.15.60:0/2217411039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22a4c0 msgr2=0x5597be23b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd305576740 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22a4c0 0x5597be23b3a0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0b8b60 tx=0x7fd2ec0f0420 comp rx=0 tx=0).stop 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd305576740 1 -- 172.21.15.60:0/2217411039 shutdown_connections 2024-03-04T12:26:18.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd305576740 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5597be29edb0 0x5597be2a11d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd305576740 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5597be2380a0 0x5597be23e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd305576740 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597be22ade0 0x5597be23b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd2db7fe640 0 client.15987 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd305576740 1 --2- 172.21.15.60:0/2217411039 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5597be22a4c0 0x5597be23b3a0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd305576740 1 -- 172.21.15.60:0/2217411039 >> 172.21.15.60:0/2217411039 conn(0x5597be224ec0 msgr2=0x5597be2333e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd305576740 1 -- 172.21.15.60:0/2217411039 shutdown_connections 2024-03-04T12:26:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd305576740 1 -- 172.21.15.60:0/2217411039 wait complete. 2024-03-04T12:26:18.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.007+0000 7fd305576740 20 client.15987 trim_cache size 0 max 16384 2024-03-04T12:26:18.032 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:26:18.032 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T12:26:18.033 INFO:teuthology.orchestra.run.smithi060.stdout: "scrubs": { 2024-03-04T12:26:18.033 INFO:teuthology.orchestra.run.smithi060.stdout: "231e9210-94ef-403f-9cd9-fd6882a93c83": { 2024-03-04T12:26:18.033 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/", 2024-03-04T12:26:18.033 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "231e9210-94ef-403f-9cd9-fd6882a93c83", 2024-03-04T12:26:18.033 INFO:teuthology.orchestra.run.smithi060.stdout: "options": "recursive,force" 2024-03-04T12:26:18.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:26:18.033 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:26:18.033 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:26:18.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.008+0000 7fd30299d640 1 -- 172.21.15.60:0/1264432780 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd2e80a2010 msgr2=0x7fd2e80a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:18.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.008+0000 7fd30299d640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd2e80a2010 0x7fd2e80a44d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2f4005fd0 tx=0x7fd2f4011690 comp rx=0 tx=0).stop 2024-03-04T12:26:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.008+0000 7fd30299d640 1 -- 172.21.15.60:0/1264432780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2fc1014f0 msgr2=0x7fd2fc19d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.008+0000 7fd30299d640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2fc1014f0 0x7fd2fc19d850 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec006010 tx=0x7fd2ec02cb30 comp rx=0 tx=0).stop 2024-03-04T12:26:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.008+0000 7fd30299d640 1 -- 172.21.15.60:0/1264432780 shutdown_connections 2024-03-04T12:26:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.008+0000 7fd30299d640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2fc197a50 0x7fd2fc197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.008+0000 7fd30299d640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd2fc1065c0 0x7fd2fc19dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.008+0000 7fd30299d640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd2e80a2010 0x7fd2e80a44d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.008+0000 7fd30299d640 1 --2- 172.21.15.60:0/1264432780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd2fc1014f0 0x7fd2fc19d850 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.008+0000 7fd30299d640 1 -- 172.21.15.60:0/1264432780 >> 172.21.15.60:0/1264432780 conn(0x7fd2fc0fcd50 msgr2=0x7fd2fc0fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.008+0000 7fd30299d640 1 -- 172.21.15.60:0/1264432780 shutdown_connections 2024-03-04T12:26:18.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:18.008+0000 7fd30299d640 1 -- 172.21.15.60:0/1264432780 wait complete. 2024-03-04T12:26:18.035 INFO:tasks.cephfs.filesystem:scrub status for tag:231e9210-94ef-403f-9cd9-fd6882a93c83 - {'path': '/', 'tag': '231e9210-94ef-403f-9cd9-fd6882a93c83', 'options': 'recursive,force'} 2024-03-04T12:26:19.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:19 smithi086 ceph-mon[36851]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-03-04T12:26:19.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:19 smithi086 ceph-mon[36851]: scrub queued for path: / 2024-03-04T12:26:19.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:19 smithi086 ceph-mon[36851]: scrub summary: idle+waiting paths [/] 2024-03-04T12:26:19.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:19 smithi086 ceph-mon[36851]: scrub summary: active paths [/] 2024-03-04T12:26:19.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:19 smithi086 ceph-mon[36851]: scrub summary: idle+waiting paths [/] 2024-03-04T12:26:19.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:19 smithi063 ceph-mon[36717]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-03-04T12:26:19.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:19 smithi063 ceph-mon[36717]: scrub queued for path: / 2024-03-04T12:26:19.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:19 smithi063 ceph-mon[36717]: scrub summary: idle+waiting paths [/] 2024-03-04T12:26:19.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:19 smithi063 ceph-mon[36717]: scrub summary: active paths [/] 2024-03-04T12:26:19.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:19 smithi063 ceph-mon[36717]: scrub summary: idle+waiting paths [/] 2024-03-04T12:26:19.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:19 smithi060 ceph-mon[32745]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-03-04T12:26:19.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:19 smithi060 ceph-mon[32745]: scrub queued for path: / 2024-03-04T12:26:19.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:19 smithi060 ceph-mon[32745]: scrub summary: idle+waiting paths [/] 2024-03-04T12:26:19.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:19 smithi060 ceph-mon[32745]: scrub summary: active paths [/] 2024-03-04T12:26:19.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:19 smithi060 ceph-mon[32745]: scrub summary: idle+waiting paths [/] 2024-03-04T12:26:21.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:21 smithi086 ceph-mon[36851]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.2 KiB/s wr, 15 op/s 2024-03-04T12:26:21.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:21 smithi063 ceph-mon[36717]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.2 KiB/s wr, 15 op/s 2024-03-04T12:26:21.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:21 smithi060 ceph-mon[32745]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.2 KiB/s wr, 15 op/s 2024-03-04T12:26:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.693+0000 7f51c6929640 1 -- 172.21.15.60:0/1725530391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c0100e00 msgr2=0x7f51c0104e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.693+0000 7f51c6929640 1 --2- 172.21.15.60:0/1725530391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c0100e00 0x7f51c0104e40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f51a8009900 tx=0x7f51a802bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.693+0000 7f51c6929640 1 -- 172.21.15.60:0/1725530391 shutdown_connections 2024-03-04T12:26:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.693+0000 7f51c6929640 1 --2- 172.21.15.60:0/1725530391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51c0105520 0x7f51c010d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.693+0000 7f51c6929640 1 --2- 172.21.15.60:0/1725530391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c0100e00 0x7f51c0104e40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.693+0000 7f51c6929640 1 --2- 172.21.15.60:0/1725530391 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51c0100450 0x7f51c0100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.693+0000 7f51c6929640 1 -- 172.21.15.60:0/1725530391 >> 172.21.15.60:0/1725530391 conn(0x7f51c00fbcd0 msgr2=0x7f51c00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.693+0000 7f51c6929640 1 -- 172.21.15.60:0/1725530391 shutdown_connections 2024-03-04T12:26:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.693+0000 7f51c6929640 1 -- 172.21.15.60:0/1725530391 wait complete. 2024-03-04T12:26:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.693+0000 7f51c6929640 1 Processor -- start 2024-03-04T12:26:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.693+0000 7f51c6929640 1 -- start start 2024-03-04T12:26:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51c6929640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51c0100450 0x7f51c0101ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51c6929640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51c0105520 0x7f51c0102420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51c6929640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c0102960 0x7f51c01a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51c6929640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c0110590 con 0x7f51c0100450 2024-03-04T12:26:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51c6929640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c0110410 con 0x7f51c0102960 2024-03-04T12:26:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51c6929640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c0110710 con 0x7f51c0105520 2024-03-04T12:26:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51c0100450 0x7f51c0101ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51c0100450 0x7f51c0101ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53898/0 (socket says 172.21.15.60:53898) 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51c4ec6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c0102960 0x7f51c01a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51bffff640 1 -- 172.21.15.60:0/1058330295 learned_addr learned my addr 172.21.15.60:0/1058330295 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51c0105520 0x7f51c0102420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51c4ec6640 1 -- 172.21.15.60:0/1058330295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51c0105520 msgr2=0x7f51c0102420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51c4ec6640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51c0105520 0x7f51c0102420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51c4ec6640 1 -- 172.21.15.60:0/1058330295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51c0100450 msgr2=0x7f51c0101ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51c4ec6640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51c0100450 0x7f51c0101ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51c4ec6640 1 -- 172.21.15.60:0/1058330295 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c01a64b0 con 0x7f51c0102960 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.694+0000 7f51bffff640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51c0100450 0x7f51c0101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.695+0000 7f51c4ec6640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c0102960 0x7f51c01a5db0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009fd0 tx=0x7f51b800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.695+0000 7f51bd7fa640 1 -- 172.21.15.60:0/1058330295 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b800ce20 con 0x7f51c0102960 2024-03-04T12:26:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.695+0000 7f51c6929640 1 -- 172.21.15.60:0/1058330295 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c01a67a0 con 0x7f51c0102960 2024-03-04T12:26:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.695+0000 7f51c6929640 1 -- 172.21.15.60:0/1058330295 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c01a6cc0 con 0x7f51c0102960 2024-03-04T12:26:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.695+0000 7f51bd7fa640 1 -- 172.21.15.60:0/1058330295 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51b8010430 con 0x7f51c0102960 2024-03-04T12:26:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.695+0000 7f51bd7fa640 1 -- 172.21.15.60:0/1058330295 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b800f620 con 0x7f51c0102960 2024-03-04T12:26:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.697+0000 7f51bd7fa640 1 -- 172.21.15.60:0/1058330295 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f51b8010950 con 0x7f51c0102960 2024-03-04T12:26:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.697+0000 7f51c95e4740 1 Processor -- start 2024-03-04T12:26:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51c95e4740 1 -- start start 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51c95e4740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b5286e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51c95e4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 0x5601b52877e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51c95e4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 0x5601b5292cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bd7fa640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f51ac0a2030 0x7f51ac0a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51c95e4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b51eb5f0 con 0x5601b5287360 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51c95e4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b5194930 con 0x5601b52908e0 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51c95e4740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b51e7690 con 0x5601b52880d0 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 0x5601b52877e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51c4ec6640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f51ac0a2030 0x7f51ac0a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 0x5601b52877e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53912/0 (socket says 172.21.15.60:53912) 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bffff640 1 -- 172.21.15.60:0/4126294978 learned_addr learned my addr 172.21.15.60:0/4126294978 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/4126294978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 0x5601b5292cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/4126294978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b5286e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bd7fa640 1 -- 172.21.15.60:0/1058330295 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f51b80ceac0 con 0x7f51c0102960 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bffff640 1 -- 172.21.15.60:0/4126294978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 msgr2=0x5601b5286e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bffff640 1 --2- 172.21.15.60:0/4126294978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b5286e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bffff640 1 -- 172.21.15.60:0/4126294978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 msgr2=0x5601b5292cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bffff640 1 --2- 172.21.15.60:0/4126294978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 0x5601b5292cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bffff640 1 -- 172.21.15.60:0/4126294978 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601b4f00210 con 0x5601b5287360 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.698+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/4126294978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b5286e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.699+0000 7f51bffff640 1 --2- 172.21.15.60:0/4126294978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 0x5601b52877e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f51b0011650 tx=0x7f51b0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265db52b8acc4bba server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.699+0000 7f51c4ec6640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f51ac0a2030 0x7f51ac0a44f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f51b80cbcf0 tx=0x7f51b8010cf0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.699+0000 7f51b67fc640 1 -- 172.21.15.60:0/4126294978 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b001e070 con 0x5601b5287360 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.699+0000 7f51b67fc640 1 -- 172.21.15.60:0/4126294978 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51b000fba0 con 0x5601b5287360 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.699+0000 7f51b67fc640 1 -- 172.21.15.60:0/4126294978 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b0019d40 con 0x5601b5287360 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.699+0000 7f51c95e4740 1 -- 172.21.15.60:0/4126294978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 msgr2=0x5601b52877e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.699+0000 7f51c95e4740 1 --2- 172.21.15.60:0/4126294978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 0x5601b52877e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f51b0011650 tx=0x7f51b0009f90 comp rx=0 tx=0).stop 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 -- 172.21.15.60:0/4126294978 shutdown_connections 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 --2- 172.21.15.60:0/4126294978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 0x5601b5292cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 --2- 172.21.15.60:0/4126294978 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 0x5601b52877e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 --2- 172.21.15.60:0/4126294978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b5286e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 -- 172.21.15.60:0/4126294978 >> 172.21.15.60:0/4126294978 conn(0x5601b524eb90 msgr2=0x5601b5282050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 -- 172.21.15.60:0/4126294978 shutdown_connections 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 -- 172.21.15.60:0/4126294978 wait complete. 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 -- start start 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 Processor -- start 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b5287360 0x5601b528d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52880d0 0x5601b528de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 0x5601b528e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b5293610 con 0x5601b52880d0 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b51a7d90 con 0x5601b52908e0 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.700+0000 7f51c95e4740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601b5293790 con 0x5601b5287360 2024-03-04T12:26:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51c95e4740 10 client.0 _ll_register_callbacks cb 0x5601b4effa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52880d0 0x5601b528de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51bffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52880d0 0x5601b528de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53916/0 (socket says 172.21.15.60:53916) 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51bffff640 1 -- 172.21.15.60:0/2745761783 learned_addr learned my addr 172.21.15.60:0/2745761783 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 0x5601b528e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b5287360 0x5601b528d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51bffff640 1 -- 172.21.15.60:0/2745761783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b5287360 msgr2=0x5601b528d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51bffff640 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b5287360 0x5601b528d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51bffff640 1 -- 172.21.15.60:0/2745761783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 msgr2=0x5601b528e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51bffff640 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 0x5601b528e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51bffff640 1 -- 172.21.15.60:0/2745761783 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601b4fcc620 con 0x5601b52880d0 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b5287360 0x5601b528d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51bffff640 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52880d0 0x5601b528de80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f51b0000c00 tx=0x7f51b00056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aeefd5cb4bc4193 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f519f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f519f7fe640 1 -- 172.21.15.60:0/2745761783 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b001e040 con 0x5601b52880d0 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f519f7fe640 1 -- 172.21.15.60:0/2745761783 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f51b0009cc0 con 0x5601b52880d0 2024-03-04T12:26:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.701+0000 7f51c95e4740 1 -- 172.21.15.60:0/2745761783 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f51b000fba0 con 0x5601b52880d0 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.702+0000 7f519f7fe640 1 -- 172.21.15.60:0/2745761783 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51b0030050 con 0x5601b52880d0 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.702+0000 7f51c95e4740 10 client.15999 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.702+0000 7f51c95e4740 1 -- 172.21.15.60:0/2745761783 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5601b529f920 con 0x5601b52880d0 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.702+0000 7f519f7fe640 1 -- 172.21.15.60:0/2745761783 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51b000fba0 con 0x5601b52880d0 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.702+0000 7f519f7fe640 1 -- 172.21.15.60:0/2745761783 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f51b0021070 con 0x5601b52880d0 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.702+0000 7f51c95e4740 10 client.15999 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.702+0000 7f51c95e4740 10 client.15999 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.702+0000 7f51c95e4740 20 client.15999 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.702+0000 7f51c95e4740 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5601b52a0cc0 0x5601b52a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.702+0000 7f51c95e4740 4 client.15999 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.702+0000 7f51c95e4740 1 -- 172.21.15.60:0/2745761783 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601b51f8b30 con 0x5601b52a0cc0 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.702+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5601b52a0cc0 0x5601b52a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f519f7fe640 10 client.15999 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5601b52a0cc0 0x5601b52a30b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=deb62b5e98dd9b66 server_cookie=dc527bb9652b229b in_seq=0 out_seq=0 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f519f7fe640 1 -- 172.21.15.60:0/2745761783 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5601b51f8b30 con 0x5601b52a0cc0 2024-03-04T12:26:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f519f7fe640 10 client.15999 handle_command_reply: tid=0 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f51c95e4740 1 client.15999 shutdown 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f51c95e4740 1 -- 172.21.15.60:0/2745761783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52880d0 msgr2=0x5601b528de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f51c95e4740 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52880d0 0x5601b528de80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f51b0000c00 tx=0x7f51b00056d0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f51c95e4740 1 -- 172.21.15.60:0/2745761783 shutdown_connections 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f51c95e4740 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5601b52a0cc0 0x5601b52a30b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f51c95e4740 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 0x5601b528e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f519f7fe640 0 client.15999 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f51c95e4740 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52880d0 0x5601b528de80 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f51c95e4740 1 --2- 172.21.15.60:0/2745761783 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b5287360 0x5601b528d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f51c95e4740 1 -- 172.21.15.60:0/2745761783 >> 172.21.15.60:0/2745761783 conn(0x5601b524eb90 msgr2=0x5601b5283da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f51c95e4740 1 -- 172.21.15.60:0/2745761783 shutdown_connections 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.703+0000 7f51c95e4740 1 -- 172.21.15.60:0/2745761783 wait complete. 2024-03-04T12:26:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.704+0000 7f51c95e4740 20 client.15999 trim_cache size 0 max 16384 2024-03-04T12:26:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51c95e4740 1 Processor -- start 2024-03-04T12:26:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51c95e4740 1 -- start start 2024-03-04T12:26:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51c95e4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 0x5601b5290070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51c95e4740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b528c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51c95e4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 0x5601b528c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51c95e4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c01b0710 con 0x5601b5287360 2024-03-04T12:26:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51c95e4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c01b2090 con 0x5601b52908e0 2024-03-04T12:26:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51c95e4740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c01b0d70 con 0x5601b52880d0 2024-03-04T12:26:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51bf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 0x5601b5290070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51bf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 0x5601b5290070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53926/0 (socket says 172.21.15.60:53926) 2024-03-04T12:26:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51bf7fe640 1 -- 172.21.15.60:0/388462570 learned_addr learned my addr 172.21.15.60:0/388462570 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51bffff640 1 --2- 172.21.15.60:0/388462570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b528c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51bf7fe640 1 -- 172.21.15.60:0/388462570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 msgr2=0x5601b528c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/388462570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b528c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51bf7fe640 1 -- 172.21.15.60:0/388462570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 msgr2=0x5601b528c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/388462570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 0x5601b528c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.717+0000 7f51bf7fe640 1 -- 172.21.15.60:0/388462570 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601b5286630 con 0x5601b5287360 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/388462570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 0x5601b5290070 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f51a8009900 tx=0x7f51a8041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8460d4ce50e540e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51b5ffb640 1 -- 172.21.15.60:0/388462570 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a8053070 con 0x5601b5287360 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51b5ffb640 1 -- 172.21.15.60:0/388462570 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51a80046c0 con 0x5601b5287360 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51b5ffb640 1 -- 172.21.15.60:0/388462570 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a8005040 con 0x5601b5287360 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51c95e4740 1 -- 172.21.15.60:0/388462570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 msgr2=0x5601b5290070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51c95e4740 1 --2- 172.21.15.60:0/388462570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 0x5601b5290070 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f51a8009900 tx=0x7f51a8041040 comp rx=0 tx=0).stop 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51c95e4740 1 -- 172.21.15.60:0/388462570 shutdown_connections 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51c95e4740 1 --2- 172.21.15.60:0/388462570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b52908e0 0x5601b528c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51c95e4740 1 --2- 172.21.15.60:0/388462570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b528c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51c95e4740 1 --2- 172.21.15.60:0/388462570 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b5287360 0x5601b5290070 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.729 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51c95e4740 1 -- 172.21.15.60:0/388462570 >> 172.21.15.60:0/388462570 conn(0x5601b524eb90 msgr2=0x5601b5281b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51c95e4740 1 -- 172.21.15.60:0/388462570 shutdown_connections 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.718+0000 7f51c95e4740 1 -- 172.21.15.60:0/388462570 wait complete. 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51c95e4740 1 -- start start 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51c95e4740 1 Processor -- start 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51c95e4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b5287360 0x5601b5297d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51c95e4740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b52960f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51c95e4740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52908e0 0x5601b5296630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51c95e4740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c01b1a30 con 0x5601b52908e0 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51c95e4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c01b13d0 con 0x5601b5287360 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51c95e4740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c01ae9b0 con 0x5601b52880d0 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51c95e4740 10 client.0 _ll_register_callbacks cb 0x7f51b001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51bf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52908e0 0x5601b5296630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51bf7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52908e0 0x5601b5296630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53940/0 (socket says 172.21.15.60:53940) 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51bffff640 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b52960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51bf7fe640 1 -- 172.21.15.60:0/1165421038 learned_addr learned my addr 172.21.15.60:0/1165421038 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:21.730 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.719+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b5287360 0x5601b5297d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f51bf7fe640 1 -- 172.21.15.60:0/1165421038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 msgr2=0x5601b52960f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b52960f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f51bf7fe640 1 -- 172.21.15.60:0/1165421038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b5287360 msgr2=0x5601b5297d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b5287360 0x5601b5297d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f51bf7fe640 1 -- 172.21.15.60:0/1165421038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601b5296e30 con 0x5601b52908e0 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f51bffff640 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b52960f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f51bf7fe640 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52908e0 0x5601b5296630 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f51a8009660 tx=0x7f51a8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa33390168fcfab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f519f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f519f7fe640 1 -- 172.21.15.60:0/1165421038 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51a8060060 con 0x5601b52908e0 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f51c95e4740 1 -- 172.21.15.60:0/1165421038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f51a80046c0 con 0x5601b52908e0 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f519f7fe640 1 -- 172.21.15.60:0/1165421038 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f51a8004ec0 con 0x5601b52908e0 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f519f7fe640 1 -- 172.21.15.60:0/1165421038 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51a8053070 con 0x5601b52908e0 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f51c95e4740 10 client.16005 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f51c95e4740 1 -- 172.21.15.60:0/1165421038 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5601b52f7fb0 con 0x5601b52908e0 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.720+0000 7f519f7fe640 1 -- 172.21.15.60:0/1165421038 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51a80046c0 con 0x5601b52908e0 2024-03-04T12:26:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.721+0000 7f519f7fe640 1 -- 172.21.15.60:0/1165421038 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f51a8003aa0 con 0x5601b52908e0 2024-03-04T12:26:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.721+0000 7f51c95e4740 10 client.16005 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.721+0000 7f51c95e4740 10 client.16005 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.721+0000 7f51c95e4740 20 client.16005 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.721+0000 7f51c95e4740 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5601b52f85b0 0x5601b52fa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.721+0000 7f51c95e4740 4 client.16005 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T12:26:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.721+0000 7f51c95e4740 1 -- 172.21.15.60:0/1165421038 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5601b51f2e80 con 0x5601b52f85b0 2024-03-04T12:26:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.721+0000 7f51bffff640 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5601b52f85b0 0x5601b52fa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.721+0000 7f519f7fe640 10 client.16005 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.721+0000 7f51bffff640 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5601b52f85b0 0x5601b52fa9d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ead091641ab3f5e7 server_cookie=78b666b83b4bdee6 in_seq=0 out_seq=0 2024-03-04T12:26:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f519f7fe640 1 -- 172.21.15.60:0/1165421038 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5601b51f2e80 con 0x5601b52f85b0 2024-03-04T12:26:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f519f7fe640 10 client.16005 handle_command_reply: tid=0 2024-03-04T12:26:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f51c95e4740 1 client.16005 shutdown 2024-03-04T12:26:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f51c95e4740 1 -- 172.21.15.60:0/1165421038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52908e0 msgr2=0x5601b5296630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f51c95e4740 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52908e0 0x5601b5296630 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f51a8009660 tx=0x7f51a8005e40 comp rx=0 tx=0).stop 2024-03-04T12:26:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f51c95e4740 1 -- 172.21.15.60:0/1165421038 shutdown_connections 2024-03-04T12:26:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f51c95e4740 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5601b52f85b0 0x5601b52fa9d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f51c95e4740 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5601b52908e0 0x5601b5296630 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f51c95e4740 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5601b52880d0 0x5601b52960f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f519f7fe640 0 client.16005 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f51c95e4740 1 --2- 172.21.15.60:0/1165421038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5601b5287360 0x5601b5297d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f51c95e4740 1 -- 172.21.15.60:0/1165421038 >> 172.21.15.60:0/1165421038 conn(0x5601b524eb90 msgr2=0x5601b528c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f51c95e4740 1 -- 172.21.15.60:0/1165421038 shutdown_connections 2024-03-04T12:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.727+0000 7f51c95e4740 1 -- 172.21.15.60:0/1165421038 wait complete. 2024-03-04T12:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c95e4740 20 client.16005 trim_cache size 0 max 16384 2024-03-04T12:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:{ 2024-03-04T12:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "message": "", 2024-03-04T12:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "return_code": 0 2024-03-04T12:26:21.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:} 2024-03-04T12:26:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c6929640 1 -- 172.21.15.60:0/1058330295 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f51ac0a2030 msgr2=0x7f51ac0a44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c6929640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f51ac0a2030 0x7f51ac0a44f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f51b80cbcf0 tx=0x7f51b8010cf0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c6929640 1 -- 172.21.15.60:0/1058330295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c0102960 msgr2=0x7f51c01a5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c6929640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c0102960 0x7f51c01a5db0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009fd0 tx=0x7f51b800ef10 comp rx=0 tx=0).stop 2024-03-04T12:26:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c6929640 1 -- 172.21.15.60:0/1058330295 shutdown_connections 2024-03-04T12:26:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c6929640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c0102960 0x7f51c01a5db0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c6929640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51c0105520 0x7f51c0102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c6929640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f51ac0a2030 0x7f51ac0a44f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c6929640 1 --2- 172.21.15.60:0/1058330295 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f51c0100450 0x7f51c0101ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c6929640 1 -- 172.21.15.60:0/1058330295 >> 172.21.15.60:0/1058330295 conn(0x7f51c00fbcd0 msgr2=0x7f51c0069d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c6929640 1 -- 172.21.15.60:0/1058330295 shutdown_connections 2024-03-04T12:26:21.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:21.728+0000 7f51c6929640 1 -- 172.21.15.60:0/1058330295 wait complete. 2024-03-04T12:26:22.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:22 smithi086 ceph-mon[36851]: scrub summary: idle 2024-03-04T12:26:22.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:22 smithi063 ceph-mon[36717]: scrub summary: idle 2024-03-04T12:26:22.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:22 smithi060 ceph-mon[32745]: scrub summary: idle 2024-03-04T12:26:23.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:23 smithi086 ceph-mon[36851]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.9 KiB/s wr, 17 op/s 2024-03-04T12:26:23.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:23 smithi063 ceph-mon[36717]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.9 KiB/s wr, 17 op/s 2024-03-04T12:26:23.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:23 smithi060 ceph-mon[32745]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.9 KiB/s wr, 17 op/s 2024-03-04T12:26:25.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:25 smithi086 ceph-mon[36851]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-04T12:26:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:25 smithi063 ceph-mon[36717]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-04T12:26:25.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:25 smithi060 ceph-mon[32745]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-04T12:26:27.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:27 smithi086 ceph-mon[36851]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 511 B/s wr, 14 op/s 2024-03-04T12:26:27.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:27 smithi063 ceph-mon[36717]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 511 B/s wr, 14 op/s 2024-03-04T12:26:27.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:27 smithi060 ceph-mon[32745]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 511 B/s wr, 14 op/s 2024-03-04T12:26:29.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:29 smithi086 ceph-mon[36851]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-04T12:26:29.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:29 smithi063 ceph-mon[36717]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-04T12:26:29.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:29 smithi060 ceph-mon[32745]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-04T12:26:31.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:31 smithi086 ceph-mon[36851]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-04T12:26:31.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:31 smithi063 ceph-mon[36717]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-04T12:26:31.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:31 smithi060 ceph-mon[32745]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-04T12:26:33.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:33 smithi086 ceph-mon[36851]: pgmap v22: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-03-04T12:26:33.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:33 smithi063 ceph-mon[36717]: pgmap v22: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-03-04T12:26:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:33 smithi060 ceph-mon[32745]: pgmap v22: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-03-04T12:26:34.828 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.819+0000 7f47a6259640 1 -- 172.21.15.60:0/3417452118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47a0100440 msgr2=0x7f47a0100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:34.828 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.819+0000 7f47a6259640 1 --2- 172.21.15.60:0/3417452118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47a0100440 0x7f47a0100820 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f4790009900 tx=0x7f479002bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:34.828 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.819+0000 7f47a6259640 1 -- 172.21.15.60:0/3417452118 shutdown_connections 2024-03-04T12:26:34.828 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.819+0000 7f47a6259640 1 --2- 172.21.15.60:0/3417452118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47a0105510 0x7f47a010d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.828 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.819+0000 7f47a6259640 1 --2- 172.21.15.60:0/3417452118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a0100df0 0x7f47a0104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.819+0000 7f47a6259640 1 --2- 172.21.15.60:0/3417452118 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47a0100440 0x7f47a0100820 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.819+0000 7f47a6259640 1 -- 172.21.15.60:0/3417452118 >> 172.21.15.60:0/3417452118 conn(0x7f47a0077320 msgr2=0x7f47a0077730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.819+0000 7f47a6259640 1 -- 172.21.15.60:0/3417452118 shutdown_connections 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.819+0000 7f47a6259640 1 -- 172.21.15.60:0/3417452118 wait complete. 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f47a6259640 1 Processor -- start 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f47a6259640 1 -- start start 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f47a6259640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47a0100440 0x7f47a0197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f47a6259640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47a0105510 0x7f47a01981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f47a6259640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a019c550 0x7f47a019ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f47a6259640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01105d0 con 0x7f47a0100440 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f47a6259640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0110450 con 0x7f47a019c550 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f47a6259640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0110750 con 0x7f47a0105510 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f479f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47a0100440 0x7f47a0197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.829 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f479f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47a0100440 0x7f47a0197c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43986/0 (socket says 172.21.15.60:43986) 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f479f7fe640 1 -- 172.21.15.60:0/850038813 learned_addr learned my addr 172.21.15.60:0/850038813 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f479ffff640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a019c550 0x7f47a019ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.820+0000 7f479effd640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47a0105510 0x7f47a01981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.821+0000 7f479f7fe640 1 -- 172.21.15.60:0/850038813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47a0105510 msgr2=0x7f47a01981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.821+0000 7f479f7fe640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47a0105510 0x7f47a01981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.821+0000 7f479f7fe640 1 -- 172.21.15.60:0/850038813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a019c550 msgr2=0x7f47a019ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.821+0000 7f479f7fe640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a019c550 0x7f47a019ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.821+0000 7f479f7fe640 1 -- 172.21.15.60:0/850038813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a019d040 con 0x7f47a0100440 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.821+0000 7f479f7fe640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47a0100440 0x7f47a0197c60 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4790006010 tx=0x7f479002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.821+0000 7f479cff9640 1 -- 172.21.15.60:0/850038813 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790041070 con 0x7f47a0100440 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.821+0000 7f479cff9640 1 -- 172.21.15.60:0/850038813 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4790004450 con 0x7f47a0100440 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.821+0000 7f47a6259640 1 -- 172.21.15.60:0/850038813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a019d2d0 con 0x7f47a0100440 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.821+0000 7f479cff9640 1 -- 172.21.15.60:0/850038813 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790003e00 con 0x7f47a0100440 2024-03-04T12:26:34.830 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.821+0000 7f47a6259640 1 -- 172.21.15.60:0/850038813 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a01a4a90 con 0x7f47a0100440 2024-03-04T12:26:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f479cff9640 1 -- 172.21.15.60:0/850038813 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f4790044070 con 0x7f47a0100440 2024-03-04T12:26:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f47a8e38740 1 Processor -- start 2024-03-04T12:26:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f47a8e38740 1 -- start start 2024-03-04T12:26:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f47a8e38740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 0x56331f4e3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f47a8e38740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e4360 0x56331f4e47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f47a8e38740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4ed8e0 0x56331f4efcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f47a8e38740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x56331f4485f0 con 0x56331f4e50d0 2024-03-04T12:26:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f47a8e38740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x56331f3f1930 con 0x56331f4ed8e0 2024-03-04T12:26:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f47a8e38740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56331f444690 con 0x56331f4e4360 2024-03-04T12:26:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f479ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 0x56331f4e3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f479ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 0x56331f4e3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:43998/0 (socket says 172.21.15.60:43998) 2024-03-04T12:26:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f479ffff640 1 -- 172.21.15.60:0/1754016033 learned_addr learned my addr 172.21.15.60:0/1754016033 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f479effd640 1 --2- 172.21.15.60:0/1754016033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4ed8e0 0x56331f4efcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f479f7fe640 1 --2- 172.21.15.60:0/1754016033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e4360 0x56331f4e47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.824+0000 7f479cff9640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f47880a2000 0x7f47880a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.825+0000 7f479ffff640 1 -- 172.21.15.60:0/1754016033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e4360 msgr2=0x56331f4e47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.825+0000 7f479ffff640 1 --2- 172.21.15.60:0/1754016033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e4360 0x56331f4e47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.825+0000 7f479ffff640 1 -- 172.21.15.60:0/1754016033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4ed8e0 msgr2=0x56331f4efcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.825+0000 7f479ffff640 1 --2- 172.21.15.60:0/1754016033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4ed8e0 0x56331f4efcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.825+0000 7f479ffff640 1 -- 172.21.15.60:0/1754016033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56331f15d210 con 0x56331f4e50d0 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.825+0000 7f479f7fe640 1 --2- 172.21.15.60:0/1754016033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e4360 0x56331f4e47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.825+0000 7f479cff9640 1 -- 172.21.15.60:0/850038813 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f47900f01d0 con 0x7f47a0100440 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.825+0000 7f479ffff640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f47880a2000 0x7f47880a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.825+0000 7f479ffff640 1 --2- 172.21.15.60:0/1754016033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 0x56331f4e3e20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4798010440 tx=0x7f4798010900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b6aaaafe24279d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.825+0000 7f4785ffb640 1 -- 172.21.15.60:0/1754016033 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479801d400 con 0x56331f4e50d0 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.825+0000 7f4785ffb640 1 -- 172.21.15.60:0/1754016033 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f479801da80 con 0x56331f4e50d0 2024-03-04T12:26:34.839 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.825+0000 7f4785ffb640 1 -- 172.21.15.60:0/1754016033 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479801c840 con 0x56331f4e50d0 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.826+0000 7f47a8e38740 1 -- 172.21.15.60:0/1754016033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 msgr2=0x56331f4e3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.826+0000 7f479ffff640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f47880a2000 0x7f47880a44c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4798021000 tx=0x7f479801e000 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.826+0000 7f47a8e38740 1 --2- 172.21.15.60:0/1754016033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 0x56331f4e3e20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4798010440 tx=0x7f4798010900 comp rx=0 tx=0).stop 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.826+0000 7f47a8e38740 1 -- 172.21.15.60:0/1754016033 shutdown_connections 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.826+0000 7f47a8e38740 1 --2- 172.21.15.60:0/1754016033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4ed8e0 0x56331f4efcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.826+0000 7f47a8e38740 1 --2- 172.21.15.60:0/1754016033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e4360 0x56331f4e47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.826+0000 7f47a8e38740 1 --2- 172.21.15.60:0/1754016033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 0x56331f4e3e20 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.826+0000 7f47a8e38740 1 -- 172.21.15.60:0/1754016033 >> 172.21.15.60:0/1754016033 conn(0x56331f4abb90 msgr2=0x56331f4df050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.826+0000 7f47a8e38740 1 -- 172.21.15.60:0/1754016033 shutdown_connections 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.826+0000 7f47a8e38740 1 -- 172.21.15.60:0/1754016033 wait complete. 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.826+0000 7f47a8e38740 1 -- start start 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.826+0000 7f47a8e38740 1 Processor -- start 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f47a8e38740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e4360 0x56331f4ea8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.840 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f47a8e38740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e50d0 0x56331f4eadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f47a8e38740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4ed8e0 0x56331f4eb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f47a8e38740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x56331f4f0610 con 0x56331f4e4360 2024-03-04T12:26:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f47a8e38740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x56331f404d90 con 0x56331f4ed8e0 2024-03-04T12:26:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f47a8e38740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56331f4f0790 con 0x56331f4e50d0 2024-03-04T12:26:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f479f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e4360 0x56331f4ea8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f47a8e38740 10 client.0 _ll_register_callbacks cb 0x56331f15ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f479f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e4360 0x56331f4ea8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44004/0 (socket says 172.21.15.60:44004) 2024-03-04T12:26:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f479f7fe640 1 -- 172.21.15.60:0/4235250983 learned_addr learned my addr 172.21.15.60:0/4235250983 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f479effd640 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e50d0 0x56331f4eadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.841 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f479ffff640 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4ed8e0 0x56331f4eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f479f7fe640 1 -- 172.21.15.60:0/4235250983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e50d0 msgr2=0x56331f4eadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f479f7fe640 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e50d0 0x56331f4eadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f479f7fe640 1 -- 172.21.15.60:0/4235250983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4ed8e0 msgr2=0x56331f4eb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f479f7fe640 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4ed8e0 0x56331f4eb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.827+0000 7f479f7fe640 1 -- 172.21.15.60:0/4235250983 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56331f229620 con 0x56331f4e4360 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f479f7fe640 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e4360 0x56331f4ea8b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f47900ed620 tx=0x7f47900bc620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3243d1aa3be591a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f477effd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f477effd640 1 -- 172.21.15.60:0/4235250983 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479004b070 con 0x56331f4e4360 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f477effd640 1 -- 172.21.15.60:0/4235250983 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f47900dc0d0 con 0x56331f4e4360 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f47a8e38740 1 -- 172.21.15.60:0/4235250983 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f479801da80 con 0x56331f4e4360 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f477effd640 1 -- 172.21.15.60:0/4235250983 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47900d5050 con 0x56331f4e4360 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f47a8e38740 10 client.16017 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:34.842 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f47a8e38740 1 -- 172.21.15.60:0/4235250983 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56331f4fc8f0 con 0x56331f4e4360 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f477effd640 1 -- 172.21.15.60:0/4235250983 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f479801da80 con 0x56331f4e4360 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f477effd640 1 -- 172.21.15.60:0/4235250983 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f47900bc9f0 con 0x56331f4e4360 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f47a8e38740 10 client.16017 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f47a8e38740 10 client.16017 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f47a8e38740 20 client.16017 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f47a8e38740 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56331f4fdd00 0x56331f500150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f47a8e38740 4 client.16017 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.828+0000 7f47a8e38740 1 -- 172.21.15.60:0/4235250983 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56331f455b30 con 0x56331f4fdd00 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.829+0000 7f479effd640 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56331f4fdd00 0x56331f500150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.829+0000 7f477effd640 10 client.16017 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.829+0000 7f479effd640 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56331f4fdd00 0x56331f500150 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa2bd1c59d08c4d server_cookie=4a85bd55fa8d0fe6 in_seq=0 out_seq=0 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f477effd640 1 -- 172.21.15.60:0/4235250983 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56331f455b30 con 0x56331f4fdd00 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f477effd640 10 client.16017 handle_command_reply: tid=0 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f47a8e38740 1 client.16017 shutdown 2024-03-04T12:26:34.843 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f47a8e38740 1 -- 172.21.15.60:0/4235250983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e4360 msgr2=0x56331f4ea8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f47a8e38740 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e4360 0x56331f4ea8b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f47900ed620 tx=0x7f47900bc620 comp rx=0 tx=0).stop 2024-03-04T12:26:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f47a8e38740 1 -- 172.21.15.60:0/4235250983 shutdown_connections 2024-03-04T12:26:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f47a8e38740 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56331f4fdd00 0x56331f500150 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f47a8e38740 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4ed8e0 0x56331f4eb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f477effd640 0 client.16017 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f47a8e38740 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e50d0 0x56331f4eadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f47a8e38740 1 --2- 172.21.15.60:0/4235250983 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e4360 0x56331f4ea8b0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f47a8e38740 1 -- 172.21.15.60:0/4235250983 >> 172.21.15.60:0/4235250983 conn(0x56331f4abb90 msgr2=0x56331f4e0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f47a8e38740 1 -- 172.21.15.60:0/4235250983 shutdown_connections 2024-03-04T12:26:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f47a8e38740 1 -- 172.21.15.60:0/4235250983 wait complete. 2024-03-04T12:26:34.844 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.830+0000 7f47a8e38740 20 client.16017 trim_cache size 0 max 16384 2024-03-04T12:26:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.843+0000 7f47a8e38740 1 Processor -- start 2024-03-04T12:26:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f47a8e38740 1 -- start start 2024-03-04T12:26:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f47a8e38740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e50d0 0x56331f4ecfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f47a8e38740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4ed8e0 0x56331f4e9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f47a8e38740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4e96c0 0x56331f4e7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f47a8e38740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01ac1c0 con 0x56331f4ed8e0 2024-03-04T12:26:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f47a8e38740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01adb40 con 0x56331f4e96c0 2024-03-04T12:26:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f47a8e38740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01ac820 con 0x56331f4e50d0 2024-03-04T12:26:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f479effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4ed8e0 0x56331f4e9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.855 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f479effd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4ed8e0 0x56331f4e9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44020/0 (socket says 172.21.15.60:44020) 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f479effd640 1 -- 172.21.15.60:0/638466092 learned_addr learned my addr 172.21.15.60:0/638466092 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f479ffff640 1 --2- 172.21.15.60:0/638466092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4e96c0 0x56331f4e7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f479f7fe640 1 --2- 172.21.15.60:0/638466092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e50d0 0x56331f4ecfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f479effd640 1 -- 172.21.15.60:0/638466092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e50d0 msgr2=0x56331f4ecfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f479effd640 1 --2- 172.21.15.60:0/638466092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e50d0 0x56331f4ecfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f479effd640 1 -- 172.21.15.60:0/638466092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4e96c0 msgr2=0x56331f4e7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f479effd640 1 --2- 172.21.15.60:0/638466092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4e96c0 0x56331f4e7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f479effd640 1 -- 172.21.15.60:0/638466092 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56331f505380 con 0x56331f4ed8e0 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.844+0000 7f479effd640 1 --2- 172.21.15.60:0/638466092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4ed8e0 0x56331f4e9180 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4794009be0 tx=0x7f4794005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1309bceb7cb0bd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.845+0000 7f47857fa640 1 -- 172.21.15.60:0/638466092 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794004110 con 0x56331f4ed8e0 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.845+0000 7f47857fa640 1 -- 172.21.15.60:0/638466092 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f479401d880 con 0x56331f4ed8e0 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.845+0000 7f47857fa640 1 -- 172.21.15.60:0/638466092 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47940047a0 con 0x56331f4ed8e0 2024-03-04T12:26:34.856 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.845+0000 7f47a8e38740 1 -- 172.21.15.60:0/638466092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4ed8e0 msgr2=0x56331f4e9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.845+0000 7f47a8e38740 1 --2- 172.21.15.60:0/638466092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4ed8e0 0x56331f4e9180 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4794009be0 tx=0x7f4794005f90 comp rx=0 tx=0).stop 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.845+0000 7f47a8e38740 1 -- 172.21.15.60:0/638466092 shutdown_connections 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.845+0000 7f47a8e38740 1 --2- 172.21.15.60:0/638466092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4e96c0 0x56331f4e7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.845+0000 7f47a8e38740 1 --2- 172.21.15.60:0/638466092 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4ed8e0 0x56331f4e9180 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.845+0000 7f47a8e38740 1 --2- 172.21.15.60:0/638466092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4e50d0 0x56331f4ecfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.845+0000 7f47a8e38740 1 -- 172.21.15.60:0/638466092 >> 172.21.15.60:0/638466092 conn(0x56331f4abb90 msgr2=0x56331f4e5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.845+0000 7f47a8e38740 1 -- 172.21.15.60:0/638466092 shutdown_connections 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.845+0000 7f47a8e38740 1 -- 172.21.15.60:0/638466092 wait complete. 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f47a8e38740 1 -- start start 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f47a8e38740 1 Processor -- start 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f47a8e38740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 0x56331f4f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f47a8e38740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4e96c0 0x56331f4f4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f47a8e38740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4ed8e0 0x56331f554280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f47a8e38740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01ad4e0 con 0x56331f4e50d0 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f47a8e38740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01ace80 con 0x56331f4e96c0 2024-03-04T12:26:34.857 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f47a8e38740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01aa460 con 0x56331f4ed8e0 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f47a8e38740 10 client.0 _ll_register_callbacks cb 0x7f4790009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f479f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 0x56331f4f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f479f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 0x56331f4f3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:44024/0 (socket says 172.21.15.60:44024) 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f479f7fe640 1 -- 172.21.15.60:0/3436425898 learned_addr learned my addr 172.21.15.60:0/3436425898 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.846+0000 7f479effd640 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4e96c0 0x56331f4f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f479ffff640 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4ed8e0 0x56331f554280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f479f7fe640 1 -- 172.21.15.60:0/3436425898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4ed8e0 msgr2=0x56331f554280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f479f7fe640 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4ed8e0 0x56331f554280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f479f7fe640 1 -- 172.21.15.60:0/3436425898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4e96c0 msgr2=0x56331f4f4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f479f7fe640 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4e96c0 0x56331f4f4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f479f7fe640 1 -- 172.21.15.60:0/3436425898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56331f5548c0 con 0x56331f4e50d0 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f479ffff640 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4ed8e0 0x56331f554280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:34.858 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f479f7fe640 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 0x56331f4f3c50 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f47900b8bf0 tx=0x7f47900bcc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3211c383939208 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f477effd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f477effd640 1 -- 172.21.15.60:0/3436425898 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f479004b070 con 0x56331f4e50d0 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f477effd640 1 -- 172.21.15.60:0/3436425898 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f47900bbde0 con 0x56331f4e50d0 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f47a8e38740 1 -- 172.21.15.60:0/3436425898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f479401d880 con 0x56331f4e50d0 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f477effd640 1 -- 172.21.15.60:0/3436425898 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47900c0830 con 0x56331f4e50d0 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f47a8e38740 10 client.16023 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f47a8e38740 1 -- 172.21.15.60:0/3436425898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56331f555bd0 con 0x56331f4e50d0 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f477effd640 1 -- 172.21.15.60:0/3436425898 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f479401d880 con 0x56331f4e50d0 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f477effd640 1 -- 172.21.15.60:0/3436425898 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f47900b84e0 con 0x56331f4e50d0 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f47a8e38740 10 client.16023 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f47a8e38740 10 client.16023 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.847+0000 7f47a8e38740 20 client.16023 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.848+0000 7f47a8e38740 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56331f5560b0 0x56331f5584a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.848+0000 7f47a8e38740 4 client.16023 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T12:26:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.848+0000 7f47a8e38740 1 -- 172.21.15.60:0/3436425898 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56331f4e4050 con 0x56331f5560b0 2024-03-04T12:26:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.848+0000 7f479effd640 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56331f5560b0 0x56331f5584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.848+0000 7f477effd640 10 client.16023 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.848+0000 7f479effd640 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56331f5560b0 0x56331f5584a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=180ce4170c720df8 server_cookie=18cbdd707d3eef1b in_seq=0 out_seq=0 2024-03-04T12:26:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f477effd640 1 -- 172.21.15.60:0/3436425898 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56331f4e4050 con 0x56331f5560b0 2024-03-04T12:26:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f477effd640 10 client.16023 handle_command_reply: tid=0 2024-03-04T12:26:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f47a8e38740 1 client.16023 shutdown 2024-03-04T12:26:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f47a8e38740 1 -- 172.21.15.60:0/3436425898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 msgr2=0x56331f4f3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f47a8e38740 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 0x56331f4f3c50 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f47900b8bf0 tx=0x7f47900bcc00 comp rx=0 tx=0).stop 2024-03-04T12:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f47a8e38740 1 -- 172.21.15.60:0/3436425898 shutdown_connections 2024-03-04T12:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f47a8e38740 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56331f5560b0 0x56331f5584a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f47a8e38740 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56331f4ed8e0 0x56331f554280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f47a8e38740 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56331f4e96c0 0x56331f4f4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f477effd640 0 client.16023 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f47a8e38740 1 --2- 172.21.15.60:0/3436425898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56331f4e50d0 0x56331f4f3c50 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f47a8e38740 1 -- 172.21.15.60:0/3436425898 >> 172.21.15.60:0/3436425898 conn(0x56331f4abb90 msgr2=0x56331f4e5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f47a8e38740 1 -- 172.21.15.60:0/3436425898 shutdown_connections 2024-03-04T12:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f47a8e38740 1 -- 172.21.15.60:0/3436425898 wait complete. 2024-03-04T12:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.856+0000 7f47a8e38740 20 client.16023 trim_cache size 0 max 16384 2024-03-04T12:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:{ 2024-03-04T12:26:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "message": "", 2024-03-04T12:26:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "return_code": 0 2024-03-04T12:26:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:} 2024-03-04T12:26:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.857+0000 7f47a6259640 1 -- 172.21.15.60:0/850038813 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f47880a2000 msgr2=0x7f47880a44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:34.866 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.857+0000 7f47a6259640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f47880a2000 0x7f47880a44c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4798021000 tx=0x7f479801e000 comp rx=0 tx=0).stop 2024-03-04T12:26:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.857+0000 7f47a6259640 1 -- 172.21.15.60:0/850038813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47a0100440 msgr2=0x7f47a0197c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.857+0000 7f47a6259640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47a0100440 0x7f47a0197c60 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4790006010 tx=0x7f479002cb30 comp rx=0 tx=0).stop 2024-03-04T12:26:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.857+0000 7f47a6259640 1 -- 172.21.15.60:0/850038813 shutdown_connections 2024-03-04T12:26:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.857+0000 7f47a6259640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a019c550 0x7f47a019ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.857+0000 7f47a6259640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47a0105510 0x7f47a01981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.857+0000 7f47a6259640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f47880a2000 0x7f47880a44c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.857+0000 7f47a6259640 1 --2- 172.21.15.60:0/850038813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f47a0100440 0x7f47a0197c60 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.857+0000 7f47a6259640 1 -- 172.21.15.60:0/850038813 >> 172.21.15.60:0/850038813 conn(0x7f47a0077320 msgr2=0x7f47a0068e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.857+0000 7f47a6259640 1 -- 172.21.15.60:0/850038813 shutdown_connections 2024-03-04T12:26:34.867 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:34.857+0000 7f47a6259640 1 -- 172.21.15.60:0/850038813 wait complete. 2024-03-04T12:26:35.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:35 smithi086 ceph-mon[36851]: pgmap v23: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-03-04T12:26:35.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:35 smithi063 ceph-mon[36717]: pgmap v23: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-03-04T12:26:35.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:35 smithi060 ceph-mon[32745]: pgmap v23: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-03-04T12:26:37.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:37 smithi086 ceph-mon[36851]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-03-04T12:26:37.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:37 smithi063 ceph-mon[36717]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-03-04T12:26:37.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:37 smithi060 ceph-mon[32745]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-03-04T12:26:39.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:39 smithi086 ceph-mon[36851]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-03-04T12:26:39.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:39 smithi063 ceph-mon[36717]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-03-04T12:26:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:39 smithi060 ceph-mon[32745]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-03-04T12:26:41.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:41 smithi086 ceph-mon[36851]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.5 KiB/s wr, 14 op/s 2024-03-04T12:26:41.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:41 smithi063 ceph-mon[36717]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.5 KiB/s wr, 14 op/s 2024-03-04T12:26:41.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:41 smithi060 ceph-mon[32745]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.5 KiB/s wr, 14 op/s 2024-03-04T12:26:43.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:43 smithi086 ceph-mon[36851]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-03-04T12:26:43.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:43 smithi063 ceph-mon[36717]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-03-04T12:26:43.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:43 smithi060 ceph-mon[32745]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.4 KiB/s wr, 12 op/s 2024-03-04T12:26:45.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:45 smithi086 ceph-mon[36851]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-04T12:26:45.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:45 smithi063 ceph-mon[36717]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-04T12:26:45.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:45 smithi060 ceph-mon[32745]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-04T12:26:47.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:47 smithi063 ceph-mon[36717]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-04T12:26:47.425 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:47 smithi086 ceph-mon[36851]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-04T12:26:47.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:47 smithi060 ceph-mon[32745]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-04T12:26:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.958+0000 7f73e6aea640 1 -- 172.21.15.60:0/2690381359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73e0068e40 msgr2=0x7f73e0069220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.958+0000 7f73e6aea640 1 --2- 172.21.15.60:0/2690381359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73e0068e40 0x7f73e0069220 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f73d4009900 tx=0x7f73d402bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.959+0000 7f73e6aea640 1 -- 172.21.15.60:0/2690381359 shutdown_connections 2024-03-04T12:26:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.959+0000 7f73e6aea640 1 --2- 172.21.15.60:0/2690381359 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73e0106610 0x7f73e010e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.967 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.959+0000 7f73e6aea640 1 --2- 172.21.15.60:0/2690381359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e00697f0 0x7f73e0105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.959+0000 7f73e6aea640 1 --2- 172.21.15.60:0/2690381359 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73e0068e40 0x7f73e0069220 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.959+0000 7f73e6aea640 1 -- 172.21.15.60:0/2690381359 >> 172.21.15.60:0/2690381359 conn(0x7f73e0077320 msgr2=0x7f73e0077730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.959+0000 7f73e6aea640 1 -- 172.21.15.60:0/2690381359 shutdown_connections 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.959+0000 7f73e6aea640 1 -- 172.21.15.60:0/2690381359 wait complete. 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.959+0000 7f73e6aea640 1 Processor -- start 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.959+0000 7f73e6aea640 1 -- start start 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.959+0000 7f73e6aea640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0068e40 0x7f73e019d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.959+0000 7f73e6aea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73e0106610 0x7f73e019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73e6aea640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73e01979c0 0x7f73e0197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73e6aea640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e01116a0 con 0x7f73e0106610 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73e6aea640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0111520 con 0x7f73e01979c0 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73e6aea640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0111820 con 0x7f73e0068e40 2024-03-04T12:26:47.968 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73dffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73e0106610 0x7f73e019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73dffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73e0106610 0x7f73e019dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58796/0 (socket says 172.21.15.60:58796) 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73dffff640 1 -- 172.21.15.60:0/704139262 learned_addr learned my addr 172.21.15.60:0/704139262 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73e4886640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0068e40 0x7f73e019d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73e5087640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73e01979c0 0x7f73e0197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73dffff640 1 -- 172.21.15.60:0/704139262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0068e40 msgr2=0x7f73e019d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73dffff640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0068e40 0x7f73e019d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73dffff640 1 -- 172.21.15.60:0/704139262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73e01979c0 msgr2=0x7f73e0197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73dffff640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73e01979c0 0x7f73e0197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73dffff640 1 -- 172.21.15.60:0/704139262 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e01984e0 con 0x7f73e0106610 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73e4886640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0068e40 0x7f73e019d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73dffff640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73e0106610 0x7f73e019dd50 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00e9a0 tx=0x7f73cc00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73ddffb640 1 -- 172.21.15.60:0/704139262 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc00cd80 con 0x7f73e0106610 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73ddffb640 1 -- 172.21.15.60:0/704139262 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73cc0035f0 con 0x7f73e0106610 2024-03-04T12:26:47.969 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.960+0000 7f73e6aea640 1 -- 172.21.15.60:0/704139262 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e01987d0 con 0x7f73e0106610 2024-03-04T12:26:47.970 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.961+0000 7f73ddffb640 1 -- 172.21.15.60:0/704139262 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc010670 con 0x7f73e0106610 2024-03-04T12:26:47.970 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.961+0000 7f73e6aea640 1 -- 172.21.15.60:0/704139262 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e01a4a70 con 0x7f73e0106610 2024-03-04T12:26:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.963+0000 7f73ddffb640 1 -- 172.21.15.60:0/704139262 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f73cc0037a0 con 0x7f73e0106610 2024-03-04T12:26:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.963+0000 7f73e97a7740 1 Processor -- start 2024-03-04T12:26:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.963+0000 7f73ddffb640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f73d00a2080 0x7f73d00a4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.963+0000 7f73e97a7740 1 -- start start 2024-03-04T12:26:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73ddffb640 1 -- 172.21.15.60:0/704139262 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f73cc0cea50 con 0x7f73e0106610 2024-03-04T12:26:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e5087640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f73d00a2080 0x7f73d00a4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e97a7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3ace20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.975 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e97a7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 0x55cc5a3ad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e97a7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3b8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e97a7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5a3115f0 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e97a7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5a2ba930 con 0x55cc5a3b68e0 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e97a7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5a30d690 con 0x55cc5a3ad360 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e4886640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73dffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 0x55cc5a3ad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e4886640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58806/0 (socket says 172.21.15.60:58806) 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e4886640 1 -- 172.21.15.60:0/783368738 learned_addr learned my addr 172.21.15.60:0/783368738 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e5087640 1 --2- 172.21.15.60:0/783368738 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3b8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73dffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 0x55cc5a3ad7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:44332/0 (socket says 172.21.15.60:44332) 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e4886640 1 -- 172.21.15.60:0/783368738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 msgr2=0x55cc5a3ad7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e4886640 1 --2- 172.21.15.60:0/783368738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 0x55cc5a3ad7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.976 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e4886640 1 -- 172.21.15.60:0/783368738 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3b68e0 msgr2=0x55cc5a3b8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e4886640 1 --2- 172.21.15.60:0/783368738 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3b8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e4886640 1 -- 172.21.15.60:0/783368738 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc5a026210 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e5087640 1 --2- 172.21.15.60:0/783368738 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3b8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73dffff640 1 --2- 172.21.15.60:0/783368738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 0x55cc5a3ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.964+0000 7f73e5087640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f73d00a2080 0x7f73d00a4540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009fd0 tx=0x7f73d8014b70 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.965+0000 7f73e4886640 1 --2- 172.21.15.60:0/783368738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3ace20 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f73d402c170 tx=0x7f73d4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd625c6b93bdd144 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.965+0000 7f73caffd640 1 -- 172.21.15.60:0/783368738 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4042070 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.965+0000 7f73caffd640 1 -- 172.21.15.60:0/783368738 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73d4004a70 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.965+0000 7f73caffd640 1 -- 172.21.15.60:0/783368738 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4005210 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.965+0000 7f73e97a7740 1 -- 172.21.15.60:0/783368738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ae0d0 msgr2=0x55cc5a3ace20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.965+0000 7f73e97a7740 1 --2- 172.21.15.60:0/783368738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3ace20 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f73d402c170 tx=0x7f73d4005e80 comp rx=0 tx=0).stop 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 -- 172.21.15.60:0/783368738 shutdown_connections 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 --2- 172.21.15.60:0/783368738 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3b8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 --2- 172.21.15.60:0/783368738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 0x55cc5a3ad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.977 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 --2- 172.21.15.60:0/783368738 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3ace20 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 -- 172.21.15.60:0/783368738 >> 172.21.15.60:0/783368738 conn(0x55cc5a374b90 msgr2=0x55cc5a3a8050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 -- 172.21.15.60:0/783368738 shutdown_connections 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 -- 172.21.15.60:0/783368738 wait complete. 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 -- start start 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 Processor -- start 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ad360 0x55cc5a3b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3b4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5a3bd9c0 con 0x55cc5a3ad360 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5a2cdd90 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.966+0000 7f73e97a7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc5a3bdb40 con 0x55cc5a3b68e0 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73e4886640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ad360 0x55cc5a3b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.978 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73e97a7740 10 client.0 _ll_register_callbacks cb 0x55cc5a025a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73e4886640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ad360 0x55cc5a3b38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58820/0 (socket says 172.21.15.60:58820) 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73e4886640 1 -- 172.21.15.60:0/2973769463 learned_addr learned my addr 172.21.15.60:0/2973769463 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73e5087640 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73dffff640 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73e4886640 1 -- 172.21.15.60:0/2973769463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3b68e0 msgr2=0x55cc5a3b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73e4886640 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73e4886640 1 -- 172.21.15.60:0/2973769463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ae0d0 msgr2=0x55cc5a3b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73e4886640 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73e4886640 1 -- 172.21.15.60:0/2973769463 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc5a0f2620 con 0x55cc5a3ad360 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73dffff640 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73e4886640 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ad360 0x55cc5a3b38a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f73d402ce00 tx=0x7f73d402ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b4f594abfc4d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73bffff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73bffff640 1 -- 172.21.15.60:0/2973769463 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d404e060 con 0x55cc5a3ad360 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73e97a7740 1 -- 172.21.15.60:0/2973769463 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73d4004a70 con 0x55cc5a3ad360 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.967+0000 7f73bffff640 1 -- 172.21.15.60:0/2973769463 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f73d40051a0 con 0x55cc5a3ad360 2024-03-04T12:26:47.979 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.968+0000 7f73bffff640 1 -- 172.21.15.60:0/2973769463 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73d4055050 con 0x55cc5a3ad360 2024-03-04T12:26:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.968+0000 7f73e97a7740 10 client.16035 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.968+0000 7f73e97a7740 1 -- 172.21.15.60:0/2973769463 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55cc5a3c9c40 con 0x55cc5a3ad360 2024-03-04T12:26:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.968+0000 7f73bffff640 1 -- 172.21.15.60:0/2973769463 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73d4004a70 con 0x55cc5a3ad360 2024-03-04T12:26:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.968+0000 7f73bffff640 1 -- 172.21.15.60:0/2973769463 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f73d4046400 con 0x55cc5a3ad360 2024-03-04T12:26:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.968+0000 7f73e97a7740 10 client.16035 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.968+0000 7f73e97a7740 10 client.16035 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.968+0000 7f73e97a7740 20 client.16035 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.968+0000 7f73e97a7740 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc5a3caf90 0x55cc5a3cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.968+0000 7f73e97a7740 4 client.16035 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.968+0000 7f73e97a7740 1 -- 172.21.15.60:0/2973769463 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc5a31eb30 con 0x55cc5a3caf90 2024-03-04T12:26:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.968+0000 7f73dffff640 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc5a3caf90 0x55cc5a3cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.980 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.969+0000 7f73dffff640 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc5a3caf90 0x55cc5a3cd380 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8117ea788e10c81 server_cookie=46a9299afd31a6e2 in_seq=0 out_seq=0 2024-03-04T12:26:47.981 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.969+0000 7f73bffff640 10 client.16035 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.969+0000 7f73bffff640 1 -- 172.21.15.60:0/2973769463 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cc5a31eb30 con 0x55cc5a3caf90 2024-03-04T12:26:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.969+0000 7f73bffff640 10 client.16035 handle_command_reply: tid=0 2024-03-04T12:26:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.969+0000 7f73e97a7740 1 client.16035 shutdown 2024-03-04T12:26:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.970+0000 7f73e97a7740 1 -- 172.21.15.60:0/2973769463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ad360 msgr2=0x55cc5a3b38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.970+0000 7f73e97a7740 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ad360 0x55cc5a3b38a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f73d402ce00 tx=0x7f73d402ce30 comp rx=0 tx=0).stop 2024-03-04T12:26:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.970+0000 7f73e97a7740 1 -- 172.21.15.60:0/2973769463 shutdown_connections 2024-03-04T12:26:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.970+0000 7f73e97a7740 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc5a3caf90 0x55cc5a3cd380 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.970+0000 7f73bffff640 0 client.16035 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.970+0000 7f73e97a7740 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.970+0000 7f73e97a7740 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.970+0000 7f73e97a7740 1 --2- 172.21.15.60:0/2973769463 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3ad360 0x55cc5a3b38a0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.982 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.970+0000 7f73e97a7740 1 -- 172.21.15.60:0/2973769463 >> 172.21.15.60:0/2973769463 conn(0x55cc5a374b90 msgr2=0x55cc5a3a9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.970+0000 7f73e97a7740 1 -- 172.21.15.60:0/2973769463 shutdown_connections 2024-03-04T12:26:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.970+0000 7f73e97a7740 1 -- 172.21.15.60:0/2973769463 wait complete. 2024-03-04T12:26:47.983 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.970+0000 7f73e97a7740 20 client.16035 trim_cache size 0 max 16384 2024-03-04T12:26:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e97a7740 1 Processor -- start 2024-03-04T12:26:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e97a7740 1 -- start start 2024-03-04T12:26:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e97a7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 0x55cc5a3b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e97a7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e97a7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3af100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e97a7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e01ac200 con 0x55cc5a3b68e0 2024-03-04T12:26:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e97a7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e01adb80 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e97a7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e01ac860 con 0x55cc5a3ad360 2024-03-04T12:26:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e4886640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 0x55cc5a3b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.991 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e4886640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 0x55cc5a3b5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:44350/0 (socket says 172.21.15.60:44350) 2024-03-04T12:26:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e4886640 1 -- 172.21.15.60:0/531496653 learned_addr learned my addr 172.21.15.60:0/531496653 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e5087640 1 --2- 172.21.15.60:0/531496653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e4886640 1 -- 172.21.15.60:0/531496653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ae0d0 msgr2=0x55cc5a3aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73dffff640 1 --2- 172.21.15.60:0/531496653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e4886640 1 --2- 172.21.15.60:0/531496653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3aebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e4886640 1 -- 172.21.15.60:0/531496653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3b68e0 msgr2=0x55cc5a3af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e4886640 1 --2- 172.21.15.60:0/531496653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73e4886640 1 -- 172.21.15.60:0/531496653 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc5a3ac630 con 0x55cc5a3ad360 2024-03-04T12:26:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.984+0000 7f73dffff640 1 --2- 172.21.15.60:0/531496653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:47.992 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.985+0000 7f73e4886640 1 --2- 172.21.15.60:0/531496653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 0x55cc5a3b5fd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f73d404c040 tx=0x7f73d4004f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce9f20d527faf5f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.985+0000 7f73ca7fc640 1 -- 172.21.15.60:0/531496653 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4042070 con 0x55cc5a3ad360 2024-03-04T12:26:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.985+0000 7f73ca7fc640 1 -- 172.21.15.60:0/531496653 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73d402c5d0 con 0x55cc5a3ad360 2024-03-04T12:26:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.985+0000 7f73ca7fc640 1 -- 172.21.15.60:0/531496653 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4005620 con 0x55cc5a3ad360 2024-03-04T12:26:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.985+0000 7f73e97a7740 1 -- 172.21.15.60:0/531496653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 msgr2=0x55cc5a3b5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.985+0000 7f73e97a7740 1 --2- 172.21.15.60:0/531496653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 0x55cc5a3b5fd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f73d404c040 tx=0x7f73d4004f20 comp rx=0 tx=0).stop 2024-03-04T12:26:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.985+0000 7f73e97a7740 1 -- 172.21.15.60:0/531496653 shutdown_connections 2024-03-04T12:26:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.985+0000 7f73e97a7740 1 --2- 172.21.15.60:0/531496653 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.985+0000 7f73e97a7740 1 --2- 172.21.15.60:0/531496653 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.985+0000 7f73e97a7740 1 --2- 172.21.15.60:0/531496653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ad360 0x55cc5a3b5fd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.985+0000 7f73e97a7740 1 -- 172.21.15.60:0/531496653 >> 172.21.15.60:0/531496653 conn(0x55cc5a374b90 msgr2=0x55cc5a3d19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:47.996 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e97a7740 1 -- 172.21.15.60:0/531496653 shutdown_connections 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e97a7740 1 -- 172.21.15.60:0/531496653 wait complete. 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e97a7740 1 -- start start 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e97a7740 1 Processor -- start 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e97a7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ad360 0x55cc5a3c1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e97a7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3c0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e97a7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3c08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e97a7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e01ad520 con 0x55cc5a3b68e0 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e97a7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e01acec0 con 0x55cc5a3ad360 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e97a7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e01aa4a0 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e97a7740 10 client.0 _ll_register_callbacks cb 0x7f73cc0969d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73dffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73dffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3c0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:44352/0 (socket says 172.21.15.60:44352) 2024-03-04T12:26:47.997 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73dffff640 1 -- 172.21.15.60:0/2581307215 learned_addr learned my addr 172.21.15.60:0/2581307215 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e5087640 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3c08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.986+0000 7f73e4886640 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ad360 0x55cc5a3c1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73dffff640 1 -- 172.21.15.60:0/2581307215 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ad360 msgr2=0x55cc5a3c1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73dffff640 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ad360 0x55cc5a3c1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73dffff640 1 -- 172.21.15.60:0/2581307215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3b68e0 msgr2=0x55cc5a3c08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73dffff640 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3c08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73dffff640 1 -- 172.21.15.60:0/2581307215 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc5a3c0f70 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73dffff640 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3c0370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009380 tx=0x7f73cc09f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=125ad4c74923eca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73bffff640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73bffff640 1 -- 172.21.15.60:0/2581307215 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc0974c0 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73bffff640 1 -- 172.21.15.60:0/2581307215 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f73cc0a3160 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73e97a7740 1 -- 172.21.15.60:0/2581307215 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73d402c5d0 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73bffff640 1 -- 172.21.15.60:0/2581307215 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73cc097660 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.998 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73e97a7740 10 client.25582 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.987+0000 7f73e97a7740 1 -- 172.21.15.60:0/2581307215 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55cc5a422030 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.988+0000 7f73bffff640 1 -- 172.21.15.60:0/2581307215 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73cc09d040 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.988+0000 7f73bffff640 1 -- 172.21.15.60:0/2581307215 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f73cc0b3b80 con 0x55cc5a3ae0d0 2024-03-04T12:26:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.988+0000 7f73e97a7740 10 client.25582 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.988+0000 7f73e97a7740 10 client.25582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.988+0000 7f73e97a7740 20 client.25582 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.988+0000 7f73e97a7740 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc5a422760 0x55cc5a424b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.988+0000 7f73e97a7740 4 client.25582 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T12:26:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.988+0000 7f73e97a7740 1 -- 172.21.15.60:0/2581307215 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7f73d402c5d0 con 0x55cc5a422760 2024-03-04T12:26:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.988+0000 7f73e4886640 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc5a422760 0x55cc5a424b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.988+0000 7f73bffff640 10 client.25582 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.988+0000 7f73e4886640 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc5a422760 0x55cc5a424b80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eae4f7c71100033 server_cookie=9f58ace0701b4963 in_seq=0 out_seq=0 2024-03-04T12:26:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73bffff640 1 -- 172.21.15.60:0/2581307215 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7f73d402c5d0 con 0x55cc5a422760 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73bffff640 10 client.25582 handle_command_reply: tid=0 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73e97a7740 1 client.25582 shutdown 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73e97a7740 1 -- 172.21.15.60:0/2581307215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ae0d0 msgr2=0x55cc5a3c0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73e97a7740 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3c0370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f73cc009380 tx=0x7f73cc09f060 comp rx=0 tx=0).stop 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73e97a7740 1 -- 172.21.15.60:0/2581307215 shutdown_connections 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73e97a7740 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc5a422760 0x55cc5a424b80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73e97a7740 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc5a3b68e0 0x55cc5a3c08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73e97a7740 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc5a3ae0d0 0x55cc5a3c0370 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73bffff640 0 client.25582 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73e97a7740 1 --2- 172.21.15.60:0/2581307215 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc5a3ad360 0x55cc5a3c1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73e97a7740 1 -- 172.21.15.60:0/2581307215 >> 172.21.15.60:0/2581307215 conn(0x55cc5a374b90 msgr2=0x55cc5a3aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73e97a7740 1 -- 172.21.15.60:0/2581307215 shutdown_connections 2024-03-04T12:26:48.001 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73e97a7740 1 -- 172.21.15.60:0/2581307215 wait complete. 2024-03-04T12:26:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.992+0000 7f73e97a7740 20 client.25582 trim_cache size 0 max 16384 2024-03-04T12:26:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:{ 2024-03-04T12:26:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "message": "", 2024-03-04T12:26:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "return_code": 0 2024-03-04T12:26:48.002 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:} 2024-03-04T12:26:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.993+0000 7f73e6aea640 1 -- 172.21.15.60:0/704139262 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f73d00a2080 msgr2=0x7f73d00a4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.993+0000 7f73e6aea640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f73d00a2080 0x7f73d00a4540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009fd0 tx=0x7f73d8014b70 comp rx=0 tx=0).stop 2024-03-04T12:26:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.993+0000 7f73e6aea640 1 -- 172.21.15.60:0/704139262 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73e0106610 msgr2=0x7f73e019dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.993+0000 7f73e6aea640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73e0106610 0x7f73e019dd50 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00e9a0 tx=0x7f73cc00ee60 comp rx=0 tx=0).stop 2024-03-04T12:26:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.993+0000 7f73e6aea640 1 -- 172.21.15.60:0/704139262 shutdown_connections 2024-03-04T12:26:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.993+0000 7f73e6aea640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73e01979c0 0x7f73e0197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.993+0000 7f73e6aea640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f73d00a2080 0x7f73d00a4540 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.003 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.993+0000 7f73e6aea640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f73e0106610 0x7f73e019dd50 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.993+0000 7f73e6aea640 1 --2- 172.21.15.60:0/704139262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0068e40 0x7f73e019d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.993+0000 7f73e6aea640 1 -- 172.21.15.60:0/704139262 >> 172.21.15.60:0/704139262 conn(0x7f73e0077320 msgr2=0x7f73e01077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.993+0000 7f73e6aea640 1 -- 172.21.15.60:0/704139262 shutdown_connections 2024-03-04T12:26:48.004 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:26:47.993+0000 7f73e6aea640 1 -- 172.21.15.60:0/704139262 wait complete. 2024-03-04T12:26:48.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T12:26:48.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T12:26:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.141+0000 7f9517953640 1 -- 172.21.15.60:0/1406177223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95101054e0 msgr2=0x7f951010d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.141+0000 7f9517953640 1 --2- 172.21.15.60:0/1406177223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95101054e0 0x7f951010d0b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9508009ca0 tx=0x7f950802c0f0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.142+0000 7f9517953640 1 -- 172.21.15.60:0/1406177223 shutdown_connections 2024-03-04T12:26:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.142+0000 7f9517953640 1 --2- 172.21.15.60:0/1406177223 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f95101054e0 0x7f951010d0b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.142+0000 7f9517953640 1 --2- 172.21.15.60:0/1406177223 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9510100dc0 0x7f9510104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.142+0000 7f9517953640 1 --2- 172.21.15.60:0/1406177223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9510100410 0x7f95101007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.142+0000 7f9517953640 1 -- 172.21.15.60:0/1406177223 >> 172.21.15.60:0/1406177223 conn(0x7f9510077330 msgr2=0x7f9510077740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.142+0000 7f9517953640 1 -- 172.21.15.60:0/1406177223 shutdown_connections 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.142+0000 7f9517953640 1 -- 172.21.15.60:0/1406177223 wait complete. 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.142+0000 7f9517953640 1 Processor -- start 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.142+0000 7f9517953640 1 -- start start 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9517953640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9510100410 0x7f951019fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9517953640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95101054e0 0x7f951019ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9517953640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9510199b10 0x7f9510199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9517953640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9510110580 con 0x7f9510100410 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9517953640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9510110400 con 0x7f9510199b10 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9517953640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9510110700 con 0x7f95101054e0 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f95156ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9510100410 0x7f951019fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f95156ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9510100410 0x7f951019fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58872/0 (socket says 172.21.15.60:58872) 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f95156ef640 1 -- 172.21.15.60:0/2297286910 learned_addr learned my addr 172.21.15.60:0/2297286910 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9514eee640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95101054e0 0x7f951019ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9515ef0640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9510199b10 0x7f9510199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9514eee640 1 -- 172.21.15.60:0/2297286910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9510199b10 msgr2=0x7f9510199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9514eee640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9510199b10 0x7f9510199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9514eee640 1 -- 172.21.15.60:0/2297286910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9510100410 msgr2=0x7f951019fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9514eee640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9510100410 0x7f951019fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9514eee640 1 -- 172.21.15.60:0/2297286910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951019a710 con 0x7f95101054e0 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f95156ef640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9510100410 0x7f951019fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9515ef0640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9510199b10 0x7f9510199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f9514eee640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95101054e0 0x7f951019ff90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f950000e9c0 tx=0x7f950000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.143+0000 7f95067fc640 1 -- 172.21.15.60:0/2297286910 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950000ccc0 con 0x7f95101054e0 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.144+0000 7f9517953640 1 -- 172.21.15.60:0/2297286910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95101a67f0 con 0x7f95101054e0 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.144+0000 7f9517953640 1 -- 172.21.15.60:0/2297286910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95101a6c50 con 0x7f95101054e0 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.144+0000 7f95067fc640 1 -- 172.21.15.60:0/2297286910 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9500010910 con 0x7f95101054e0 2024-03-04T12:26:48.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.144+0000 7f95067fc640 1 -- 172.21.15.60:0/2297286910 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500018880 con 0x7f95101054e0 2024-03-04T12:26:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.146+0000 7f95067fc640 1 -- 172.21.15.60:0/2297286910 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f9500010ac0 con 0x7f95101054e0 2024-03-04T12:26:48.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.146+0000 7f95067fc640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f94fc0a2110 0x7f94fc0a45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.146+0000 7f95067fc640 1 -- 172.21.15.60:0/2297286910 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f95000cebc0 con 0x7f95101054e0 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.146+0000 7f95156ef640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f94fc0a2110 0x7f94fc0a45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.147+0000 7f95156ef640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f94fc0a2110 0x7f94fc0a45d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f950c0095f0 tx=0x7f950c005f90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.147+0000 7f951a611740 1 Processor -- start 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.147+0000 7f951a611740 1 -- start start 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.147+0000 7f951a611740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc69231790 0x55cc69231b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.147+0000 7f951a611740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 0x55cc6923a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.147+0000 7f951a611740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc6923ac90 0x55cc6923d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.147+0000 7f951a611740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc69194e00 con 0x55cc692320b0 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.147+0000 7f951a611740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc6913db10 con 0x55cc6923ac90 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.147+0000 7f951a611740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc69190ea0 con 0x55cc69231790 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f9514eee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 0x55cc6923a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f9514eee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 0x55cc6923a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58886/0 (socket says 172.21.15.60:58886) 2024-03-04T12:26:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f9514eee640 1 -- 172.21.15.60:0/2949754000 learned_addr learned my addr 172.21.15.60:0/2949754000 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f95156ef640 1 --2- 172.21.15.60:0/2949754000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc69231790 0x55cc69231b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f9515ef0640 1 --2- 172.21.15.60:0/2949754000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc6923ac90 0x55cc6923d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f9514eee640 1 -- 172.21.15.60:0/2949754000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc69231790 msgr2=0x55cc69231b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f9514eee640 1 --2- 172.21.15.60:0/2949754000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc69231790 0x55cc69231b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f9514eee640 1 -- 172.21.15.60:0/2949754000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc6923ac90 msgr2=0x55cc6923d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f9514eee640 1 --2- 172.21.15.60:0/2949754000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc6923ac90 0x55cc6923d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f9514eee640 1 -- 172.21.15.60:0/2949754000 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc68f737d0 con 0x55cc692320b0 2024-03-04T12:26:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f9515ef0640 1 --2- 172.21.15.60:0/2949754000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc6923ac90 0x55cc6923d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:48.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f9514eee640 1 --2- 172.21.15.60:0/2949754000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 0x55cc6923a750 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f95000973c0 tx=0x7f950009a980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4094bb0846b47127 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.148+0000 7f94fb7fe640 1 -- 172.21.15.60:0/2949754000 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95000b3070 con 0x55cc692320b0 2024-03-04T12:26:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.149+0000 7f94fb7fe640 1 -- 172.21.15.60:0/2949754000 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95000a3700 con 0x55cc692320b0 2024-03-04T12:26:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.149+0000 7f94fb7fe640 1 -- 172.21.15.60:0/2949754000 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950009a610 con 0x55cc692320b0 2024-03-04T12:26:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.149+0000 7f951a611740 1 -- 172.21.15.60:0/2949754000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 msgr2=0x55cc6923a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.149+0000 7f951a611740 1 --2- 172.21.15.60:0/2949754000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 0x55cc6923a750 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f95000973c0 tx=0x7f950009a980 comp rx=0 tx=0).stop 2024-03-04T12:26:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.149+0000 7f951a611740 1 -- 172.21.15.60:0/2949754000 shutdown_connections 2024-03-04T12:26:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.149+0000 7f951a611740 1 --2- 172.21.15.60:0/2949754000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc6923ac90 0x55cc6923d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.150+0000 7f951a611740 1 --2- 172.21.15.60:0/2949754000 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 0x55cc6923a750 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.150+0000 7f951a611740 1 --2- 172.21.15.60:0/2949754000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc69231790 0x55cc69231b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.150+0000 7f951a611740 1 -- 172.21.15.60:0/2949754000 >> 172.21.15.60:0/2949754000 conn(0x55cc6922bec0 msgr2=0x55cc6922c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.150+0000 7f951a611740 1 -- 172.21.15.60:0/2949754000 shutdown_connections 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.150+0000 7f951a611740 1 -- 172.21.15.60:0/2949754000 wait complete. 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.150+0000 7f951a611740 1 -- start start 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.150+0000 7f951a611740 1 Processor -- start 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f951a611740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 0x55cc6923a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f951a611740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc692320b0 0x55cc69237780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f951a611740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc6923ac90 0x55cc69237cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f951a611740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc6923da80 con 0x55cc69231790 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f951a611740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc691510a0 con 0x55cc692320b0 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f951a611740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc6923dc00 con 0x55cc6923ac90 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f951a611740 10 client.0 _ll_register_callbacks cb 0x55cc69150a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f95156ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 0x55cc6923a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f95156ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 0x55cc6923a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58894/0 (socket says 172.21.15.60:58894) 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f95156ef640 1 -- 172.21.15.60:0/2742016191 learned_addr learned my addr 172.21.15.60:0/2742016191 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f9515ef0640 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc6923ac90 0x55cc69237cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f9514eee640 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc692320b0 0x55cc69237780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f95156ef640 1 -- 172.21.15.60:0/2742016191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc6923ac90 msgr2=0x55cc69237cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f95156ef640 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc6923ac90 0x55cc69237cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f95156ef640 1 -- 172.21.15.60:0/2742016191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc692320b0 msgr2=0x55cc69237780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f95156ef640 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc692320b0 0x55cc69237780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f95156ef640 1 -- 172.21.15.60:0/2742016191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc68ea93f0 con 0x55cc69231790 2024-03-04T12:26:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.151+0000 7f9515ef0640 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc6923ac90 0x55cc69237cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.152+0000 7f95156ef640 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 0x55cc6923a3f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f950c00fc80 tx=0x7f950c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0761919d0fdffa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.152+0000 7f94f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.152+0000 7f94f8ff9640 1 -- 172.21.15.60:0/2742016191 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c004030 con 0x55cc69231790 2024-03-04T12:26:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.152+0000 7f94f8ff9640 1 -- 172.21.15.60:0/2742016191 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f950c0041d0 con 0x55cc69231790 2024-03-04T12:26:48.161 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.152+0000 7f951a611740 1 -- 172.21.15.60:0/2742016191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95000a3700 con 0x55cc69231790 2024-03-04T12:26:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.153+0000 7f94f8ff9640 1 -- 172.21.15.60:0/2742016191 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f950c030050 con 0x55cc69231790 2024-03-04T12:26:48.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.153+0000 7f951a611740 10 client.16053 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.153+0000 7f951a611740 1 -- 172.21.15.60:0/2742016191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55cc68e690f0 con 0x55cc69231790 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.153+0000 7f94f8ff9640 1 -- 172.21.15.60:0/2742016191 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95000a3700 con 0x55cc69231790 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.153+0000 7f94f8ff9640 1 -- 172.21.15.60:0/2742016191 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f950c003ad0 con 0x55cc69231790 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.154+0000 7f951a611740 10 client.16053 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.154+0000 7f951a611740 10 client.16053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.154+0000 7f951a611740 20 client.16053 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.154+0000 7f951a611740 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc6924a790 0x55cc6924cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.154+0000 7f951a611740 4 client.16053 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.154+0000 7f951a611740 1 -- 172.21.15.60:0/2742016191 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc691a2340 con 0x55cc6924a790 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.154+0000 7f9514eee640 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc6924a790 0x55cc6924cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.154+0000 7f9514eee640 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc6924a790 0x55cc6924cb80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5ee6ab2f7872823 server_cookie=a2ab6bbfbdb2d7e7 in_seq=0 out_seq=0 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f94f8ff9640 10 client.16053 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f94f8ff9640 1 -- 172.21.15.60:0/2742016191 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cc691a2340 con 0x55cc6924a790 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f94f8ff9640 10 client.16053 handle_command_reply: tid=0 2024-03-04T12:26:48.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f951a611740 1 client.16053 shutdown 2024-03-04T12:26:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f951a611740 1 -- 172.21.15.60:0/2742016191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 msgr2=0x55cc6923a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f951a611740 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 0x55cc6923a3f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f950c00fc80 tx=0x7f950c004730 comp rx=0 tx=0).stop 2024-03-04T12:26:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f951a611740 1 -- 172.21.15.60:0/2742016191 shutdown_connections 2024-03-04T12:26:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f951a611740 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc6924a790 0x55cc6924cb80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f951a611740 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc6923ac90 0x55cc69237cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f951a611740 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc692320b0 0x55cc69237780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f94f8ff9640 0 client.16053 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f951a611740 1 --2- 172.21.15.60:0/2742016191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 0x55cc6923a3f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f951a611740 1 -- 172.21.15.60:0/2742016191 >> 172.21.15.60:0/2742016191 conn(0x55cc6922bec0 msgr2=0x55cc6922dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f951a611740 1 -- 172.21.15.60:0/2742016191 shutdown_connections 2024-03-04T12:26:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f951a611740 1 -- 172.21.15.60:0/2742016191 wait complete. 2024-03-04T12:26:48.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.155+0000 7f951a611740 20 client.16053 trim_cache size 0 max 16384 2024-03-04T12:26:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f951a611740 1 Processor -- start 2024-03-04T12:26:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f951a611740 1 -- start start 2024-03-04T12:26:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f951a611740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc69231790 0x55cc69232ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f951a611740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 0x55cc69239eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.178 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f951a611740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc6923ac90 0x55cc6923a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f9514eee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 0x55cc69239eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f951a611740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95101b06d0 con 0x55cc692320b0 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f951a611740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95101b2050 con 0x55cc69231790 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f9514eee640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 0x55cc69239eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58896/0 (socket says 172.21.15.60:58896) 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f9514eee640 1 -- 172.21.15.60:0/2523970847 learned_addr learned my addr 172.21.15.60:0/2523970847 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f951a611740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95101b0d30 con 0x55cc6923ac90 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f9515ef0640 1 --2- 172.21.15.60:0/2523970847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc6923ac90 0x55cc6923a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f95156ef640 1 --2- 172.21.15.60:0/2523970847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc69231790 0x55cc69232ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.171+0000 7f9514eee640 1 -- 172.21.15.60:0/2523970847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc6923ac90 msgr2=0x55cc6923a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.172+0000 7f9514eee640 1 --2- 172.21.15.60:0/2523970847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc6923ac90 0x55cc6923a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.172+0000 7f9514eee640 1 -- 172.21.15.60:0/2523970847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc69231790 msgr2=0x55cc69232ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.172+0000 7f9514eee640 1 --2- 172.21.15.60:0/2523970847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc69231790 0x55cc69232ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.172+0000 7f9514eee640 1 -- 172.21.15.60:0/2523970847 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc6924cef0 con 0x55cc692320b0 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.172+0000 7f9515ef0640 1 --2- 172.21.15.60:0/2523970847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc6923ac90 0x55cc6923a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:48.179 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.172+0000 7f95156ef640 1 --2- 172.21.15.60:0/2523970847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc69231790 0x55cc69232ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.172+0000 7f9514eee640 1 --2- 172.21.15.60:0/2523970847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 0x55cc69239eb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f95000a31d0 tx=0x7f9500009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a669bdc03a45f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.172+0000 7f94faffd640 1 -- 172.21.15.60:0/2523970847 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95000b7070 con 0x55cc692320b0 2024-03-04T12:26:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.172+0000 7f94faffd640 1 -- 172.21.15.60:0/2523970847 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f950009f0b0 con 0x55cc692320b0 2024-03-04T12:26:48.180 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.172+0000 7f94faffd640 1 -- 172.21.15.60:0/2523970847 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95000b3040 con 0x55cc692320b0 2024-03-04T12:26:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.173+0000 7f951a611740 1 -- 172.21.15.60:0/2523970847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 msgr2=0x55cc69239eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.173+0000 7f951a611740 1 --2- 172.21.15.60:0/2523970847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 0x55cc69239eb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f95000a31d0 tx=0x7f9500009700 comp rx=0 tx=0).stop 2024-03-04T12:26:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.173+0000 7f951a611740 1 -- 172.21.15.60:0/2523970847 shutdown_connections 2024-03-04T12:26:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.173+0000 7f951a611740 1 --2- 172.21.15.60:0/2523970847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc6923ac90 0x55cc6923a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.173+0000 7f951a611740 1 --2- 172.21.15.60:0/2523970847 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc692320b0 0x55cc69239eb0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.173+0000 7f951a611740 1 --2- 172.21.15.60:0/2523970847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc69231790 0x55cc69232ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.173+0000 7f951a611740 1 -- 172.21.15.60:0/2523970847 >> 172.21.15.60:0/2523970847 conn(0x55cc6922bec0 msgr2=0x55cc692325d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.173+0000 7f951a611740 1 -- 172.21.15.60:0/2523970847 shutdown_connections 2024-03-04T12:26:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.173+0000 7f951a611740 1 -- 172.21.15.60:0/2523970847 wait complete. 2024-03-04T12:26:48.185 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f951a611740 1 -- start start 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f951a611740 1 Processor -- start 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f951a611740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 0x55cc69242310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f951a611740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc692320b0 0x55cc6923e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f951a611740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc6923ac90 0x55cc6923edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f951a611740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95101b19f0 con 0x55cc69231790 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f951a611740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95101b1390 con 0x55cc6923ac90 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f951a611740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95101ae970 con 0x55cc692320b0 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f951a611740 10 client.0 _ll_register_callbacks cb 0x7f950c00fa40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f95156ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 0x55cc69242310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f95156ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 0x55cc69242310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58910/0 (socket says 172.21.15.60:58910) 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f95156ef640 1 -- 172.21.15.60:0/1174878151 learned_addr learned my addr 172.21.15.60:0/1174878151 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.174+0000 7f9514eee640 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc692320b0 0x55cc6923e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f9515ef0640 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc6923ac90 0x55cc6923edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f95156ef640 1 -- 172.21.15.60:0/1174878151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc692320b0 msgr2=0x55cc6923e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f95156ef640 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc692320b0 0x55cc6923e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f95156ef640 1 -- 172.21.15.60:0/1174878151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc6923ac90 msgr2=0x55cc6923edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f95156ef640 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc6923ac90 0x55cc6923edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f95156ef640 1 -- 172.21.15.60:0/1174878151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc692a0c10 con 0x55cc69231790 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f9514eee640 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc692320b0 0x55cc6923e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f9515ef0640 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc6923ac90 0x55cc6923edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f95156ef640 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 0x55cc69242310 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f950c00fc50 tx=0x7f950c01b530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a960ed700f87ad4e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f94f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f94f8ff9640 1 -- 172.21.15.60:0/1174878151 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c01bc30 con 0x55cc69231790 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f94f8ff9640 1 -- 172.21.15.60:0/1174878151 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f950c01bdd0 con 0x55cc69231790 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f951a611740 1 -- 172.21.15.60:0/1174878151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f950009f0b0 con 0x55cc69231790 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f94f8ff9640 1 -- 172.21.15.60:0/1174878151 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f950c044050 con 0x55cc69231790 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f951a611740 10 client.16059 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.175+0000 7f951a611740 1 -- 172.21.15.60:0/1174878151 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55cc692a1d60 con 0x55cc69231790 2024-03-04T12:26:48.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.176+0000 7f94f8ff9640 1 -- 172.21.15.60:0/1174878151 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f950009f0b0 con 0x55cc69231790 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.176+0000 7f94f8ff9640 1 -- 172.21.15.60:0/1174878151 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f950c0035f0 con 0x55cc69231790 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.176+0000 7f951a611740 10 client.16059 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.176+0000 7f951a611740 10 client.16059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.176+0000 7f951a611740 20 client.16059 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.176+0000 7f951a611740 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc692a24e0 0x55cc692a4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.176+0000 7f951a611740 4 client.16059 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.176+0000 7f951a611740 1 -- 172.21.15.60:0/1174878151 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cc6919c690 con 0x55cc692a24e0 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.176+0000 7f9514eee640 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc692a24e0 0x55cc692a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.176+0000 7f94f8ff9640 10 client.16059 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f9514eee640 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc692a24e0 0x55cc692a4900 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58b55415ee72f1f3 server_cookie=a3603043f283746e in_seq=0 out_seq=0 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f94f8ff9640 1 -- 172.21.15.60:0/1174878151 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cc6919c690 con 0x55cc692a24e0 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f94f8ff9640 10 client.16059 handle_command_reply: tid=0 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f951a611740 1 client.16059 shutdown 2024-03-04T12:26:48.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f951a611740 1 -- 172.21.15.60:0/1174878151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 msgr2=0x55cc69242310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f951a611740 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 0x55cc69242310 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f950c00fc50 tx=0x7f950c01b530 comp rx=0 tx=0).stop 2024-03-04T12:26:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f951a611740 1 -- 172.21.15.60:0/1174878151 shutdown_connections 2024-03-04T12:26:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f951a611740 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55cc692a24e0 0x55cc692a4900 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f951a611740 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cc6923ac90 0x55cc6923edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f94f8ff9640 0 client.16059 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f951a611740 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc692320b0 0x55cc6923e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f951a611740 1 --2- 172.21.15.60:0/1174878151 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55cc69231790 0x55cc69242310 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f951a611740 1 -- 172.21.15.60:0/1174878151 >> 172.21.15.60:0/1174878151 conn(0x55cc6922bec0 msgr2=0x55cc69236050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f951a611740 1 -- 172.21.15.60:0/1174878151 shutdown_connections 2024-03-04T12:26:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.177+0000 7f951a611740 1 -- 172.21.15.60:0/1174878151 wait complete. 2024-03-04T12:26:48.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f951a611740 20 client.16059 trim_cache size 0 max 16384 2024-03-04T12:26:48.189 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:26:48.190 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "no active scrubs running", 2024-03-04T12:26:48.190 INFO:teuthology.orchestra.run.smithi060.stdout: "scrubs": {} 2024-03-04T12:26:48.190 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:26:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f9517953640 1 -- 172.21.15.60:0/2297286910 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f94fc0a2110 msgr2=0x7f94fc0a45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f9517953640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f94fc0a2110 0x7f94fc0a45d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f950c0095f0 tx=0x7f950c005f90 comp rx=0 tx=0).stop 2024-03-04T12:26:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f9517953640 1 -- 172.21.15.60:0/2297286910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95101054e0 msgr2=0x7f951019ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f9517953640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95101054e0 0x7f951019ff90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f950000e9c0 tx=0x7f950000ee80 comp rx=0 tx=0).stop 2024-03-04T12:26:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f9517953640 1 -- 172.21.15.60:0/2297286910 shutdown_connections 2024-03-04T12:26:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f9517953640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9510199b10 0x7f9510199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f9517953640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95101054e0 0x7f951019ff90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f9517953640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f94fc0a2110 0x7f94fc0a45d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f9517953640 1 --2- 172.21.15.60:0/2297286910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9510100410 0x7f951019fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f9517953640 1 -- 172.21.15.60:0/2297286910 >> 172.21.15.60:0/2297286910 conn(0x7f9510077330 msgr2=0x7f951010c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f9517953640 1 -- 172.21.15.60:0/2297286910 shutdown_connections 2024-03-04T12:26:48.191 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.178+0000 7f9517953640 1 -- 172.21.15.60:0/2297286910 wait complete. 2024-03-04T12:26:48.196 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T12:26:48.196 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:26:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.301+0000 7f2ac799e640 1 -- 172.21.15.60:0/1970640658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ac01009f0 msgr2=0x7f2ac010d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.301+0000 7f2ac799e640 1 --2- 172.21.15.60:0/1970640658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ac01009f0 0x7f2ac010d740 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009900 tx=0x7f2ab002bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.301+0000 7f2ac799e640 1 -- 172.21.15.60:0/1970640658 shutdown_connections 2024-03-04T12:26:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.301+0000 7f2ac799e640 1 --2- 172.21.15.60:0/1970640658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ac010dc80 0x7f2ac0110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.301+0000 7f2ac799e640 1 --2- 172.21.15.60:0/1970640658 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ac01009f0 0x7f2ac010d740 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.301+0000 7f2ac799e640 1 --2- 172.21.15.60:0/1970640658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01000d0 0x7f2ac01004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.301+0000 7f2ac799e640 1 -- 172.21.15.60:0/1970640658 >> 172.21.15.60:0/1970640658 conn(0x7f2ac00773b0 msgr2=0x7f2ac00777c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.301+0000 7f2ac799e640 1 -- 172.21.15.60:0/1970640658 shutdown_connections 2024-03-04T12:26:48.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.301+0000 7f2ac799e640 1 -- 172.21.15.60:0/1970640658 wait complete. 2024-03-04T12:26:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac799e640 1 Processor -- start 2024-03-04T12:26:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac799e640 1 -- start start 2024-03-04T12:26:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac799e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ac01000d0 0x7f2ac0199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac799e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01009f0 0x7f2ac019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac799e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ac010dc80 0x7f2ac019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac799e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac01138f0 con 0x7f2ac010dc80 2024-03-04T12:26:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac799e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac0113770 con 0x7f2ac01009f0 2024-03-04T12:26:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac799e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac0113a70 con 0x7f2ac01000d0 2024-03-04T12:26:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac5f3b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ac010dc80 0x7f2ac019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac573a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ac01000d0 0x7f2ac0199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac5f3b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ac010dc80 0x7f2ac019e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58932/0 (socket says 172.21.15.60:58932) 2024-03-04T12:26:48.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac5f3b640 1 -- 172.21.15.60:0/1685968693 learned_addr learned my addr 172.21.15.60:0/1685968693 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac4f39640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01009f0 0x7f2ac019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac573a640 1 -- 172.21.15.60:0/1685968693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01009f0 msgr2=0x7f2ac019a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac573a640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01009f0 0x7f2ac019a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac573a640 1 -- 172.21.15.60:0/1685968693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ac010dc80 msgr2=0x7f2ac019e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac573a640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ac010dc80 0x7f2ac019e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.302+0000 7f2ac573a640 1 -- 172.21.15.60:0/1685968693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac019ee30 con 0x7f2ac01000d0 2024-03-04T12:26:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.303+0000 7f2ac573a640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ac01000d0 0x7f2ac0199e90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00e9f0 tx=0x7f2abc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.303+0000 7f2ab67fc640 1 -- 172.21.15.60:0/1685968693 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc00ccc0 con 0x7f2ac01000d0 2024-03-04T12:26:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.303+0000 7f2ac799e640 1 -- 172.21.15.60:0/1685968693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac019f120 con 0x7f2ac01000d0 2024-03-04T12:26:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.303+0000 7f2ab67fc640 1 -- 172.21.15.60:0/1685968693 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2abc010910 con 0x7f2ac01000d0 2024-03-04T12:26:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.303+0000 7f2ab67fc640 1 -- 172.21.15.60:0/1685968693 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc018960 con 0x7f2ac01000d0 2024-03-04T12:26:48.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.303+0000 7f2ac799e640 1 -- 172.21.15.60:0/1685968693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac01a69b0 con 0x7f2ac01000d0 2024-03-04T12:26:48.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.304+0000 7f2ac799e640 1 -- 172.21.15.60:0/1685968693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac00681b0 con 0x7f2ac01000d0 2024-03-04T12:26:48.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.306+0000 7f2ab67fc640 1 -- 172.21.15.60:0/1685968693 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f2abc010ac0 con 0x7f2ac01000d0 2024-03-04T12:26:48.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.306+0000 7f2ab67fc640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2aac0a2010 0x7f2aac0a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.307+0000 7f2ac4f39640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2aac0a2010 0x7f2aac0a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.307+0000 7f2ab67fc640 1 -- 172.21.15.60:0/1685968693 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f2abc0cec80 con 0x7f2ac01000d0 2024-03-04T12:26:48.314 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.307+0000 7f2ac4f39640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2aac0a2010 0x7f2aac0a44d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00ff280 tx=0x7f2ab0000bc0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.311+0000 7f2ab67fc640 1 -- 172.21.15.60:0/1685968693 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2abc09b720 con 0x7f2ac01000d0 2024-03-04T12:26:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.532+0000 7f2ac799e640 1 -- 172.21.15.60:0/1685968693 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ac0108e10 con 0x7f2ac01000d0 2024-03-04T12:26:48.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.533+0000 7f2ab67fc640 1 -- 172.21.15.60:0/1685968693 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11373 (secure 0 0 0) 0x7f2abc09b020 con 0x7f2ac01000d0 2024-03-04T12:26:48.540 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:26:48.540 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:25:37.524434+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24463":{"gid":24463,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/472598778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":472598778},{"type":"v1","addr":"172.21.15.86:6837","nonce":472598778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:26:48.540 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 29 2024-03-04T12:26:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.535+0000 7f2ac799e640 1 -- 172.21.15.60:0/1685968693 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2aac0a2010 msgr2=0x7f2aac0a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.535+0000 7f2ac799e640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2aac0a2010 0x7f2aac0a44d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00ff280 tx=0x7f2ab0000bc0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.535+0000 7f2ac799e640 1 -- 172.21.15.60:0/1685968693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ac01000d0 msgr2=0x7f2ac0199e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.535+0000 7f2ac799e640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ac01000d0 0x7f2ac0199e90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2abc00e9f0 tx=0x7f2abc00eeb0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.535+0000 7f2ac799e640 1 -- 172.21.15.60:0/1685968693 shutdown_connections 2024-03-04T12:26:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.535+0000 7f2ac799e640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2aac0a2010 0x7f2aac0a44d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.535+0000 7f2ac799e640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2ac010dc80 0x7f2ac019e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.542 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.535+0000 7f2ac799e640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ac01009f0 0x7f2ac019a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.535+0000 7f2ac799e640 1 --2- 172.21.15.60:0/1685968693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ac01000d0 0x7f2ac0199e90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.536+0000 7f2ac799e640 1 -- 172.21.15.60:0/1685968693 >> 172.21.15.60:0/1685968693 conn(0x7f2ac00773b0 msgr2=0x7f2ac010f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.536+0000 7f2ac799e640 1 -- 172.21.15.60:0/1685968693 shutdown_connections 2024-03-04T12:26:48.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.536+0000 7f2ac799e640 1 -- 172.21.15.60:0/1685968693 wait complete. 2024-03-04T12:26:48.553 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:26:48.553 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-04T12:26:48.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.659+0000 7f66082c0640 1 -- 172.21.15.60:0/124064999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6600100570 msgr2=0x7f660010c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.668 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.659+0000 7f66082c0640 1 --2- 172.21.15.60:0/124064999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6600100570 0x7f660010c800 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f65f8009ca0 tx=0x7f65f802c0f0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.659+0000 7f66082c0640 1 -- 172.21.15.60:0/124064999 shutdown_connections 2024-03-04T12:26:48.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.659+0000 7f66082c0640 1 --2- 172.21.15.60:0/124064999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6600100570 0x7f660010c800 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.659+0000 7f66082c0640 1 --2- 172.21.15.60:0/124064999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66000ffbb0 0x7f6600100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.659+0000 7f66082c0640 1 --2- 172.21.15.60:0/124064999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6600105940 0x7f6600105d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.659+0000 7f66082c0640 1 -- 172.21.15.60:0/124064999 >> 172.21.15.60:0/124064999 conn(0x7f6600075490 msgr2=0x7f66000758a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.659+0000 7f66082c0640 1 -- 172.21.15.60:0/124064999 shutdown_connections 2024-03-04T12:26:48.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.659+0000 7f66082c0640 1 -- 172.21.15.60:0/124064999 wait complete. 2024-03-04T12:26:48.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f66082c0640 1 Processor -- start 2024-03-04T12:26:48.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f66082c0640 1 -- start start 2024-03-04T12:26:48.669 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f66082c0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66000ffbb0 0x7f6600199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f66082c0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6600100570 0x7f660019a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f66082c0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6600105940 0x7f660019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f66082c0640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6600111710 con 0x7f6600100570 2024-03-04T12:26:48.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f66082c0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6600111590 con 0x7f66000ffbb0 2024-03-04T12:26:48.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f66082c0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6600111890 con 0x7f6600105940 2024-03-04T12:26:48.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f660585b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6600100570 0x7f660019a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f660585b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6600100570 0x7f660019a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58936/0 (socket says 172.21.15.60:58936) 2024-03-04T12:26:48.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f660585b640 1 -- 172.21.15.60:0/19090156 learned_addr learned my addr 172.21.15.60:0/19090156 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f660605c640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66000ffbb0 0x7f6600199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f660685d640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6600105940 0x7f660019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f660585b640 1 -- 172.21.15.60:0/19090156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6600105940 msgr2=0x7f660019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f660585b640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6600105940 0x7f660019e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.671 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f660585b640 1 -- 172.21.15.60:0/19090156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66000ffbb0 msgr2=0x7f6600199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f660585b640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66000ffbb0 0x7f6600199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.660+0000 7f660585b640 1 -- 172.21.15.60:0/19090156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660019ed80 con 0x7f6600100570 2024-03-04T12:26:48.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.661+0000 7f660585b640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6600100570 0x7f660019a2d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f65f000e9f0 tx=0x7f65f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.661+0000 7f65f77fe640 1 -- 172.21.15.60:0/19090156 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f000cd80 con 0x7f6600100570 2024-03-04T12:26:48.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.661+0000 7f65f77fe640 1 -- 172.21.15.60:0/19090156 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65f0005500 con 0x7f6600100570 2024-03-04T12:26:48.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.661+0000 7f65f77fe640 1 -- 172.21.15.60:0/19090156 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f0010690 con 0x7f6600100570 2024-03-04T12:26:48.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.661+0000 7f66082c0640 1 -- 172.21.15.60:0/19090156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f660019f070 con 0x7f6600100570 2024-03-04T12:26:48.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.661+0000 7f66082c0640 1 -- 172.21.15.60:0/19090156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66001a6900 con 0x7f6600100570 2024-03-04T12:26:48.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.664+0000 7f660af7f740 1 Processor -- start 2024-03-04T12:26:48.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.664+0000 7f660af7f740 1 -- start start 2024-03-04T12:26:48.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.664+0000 7f660af7f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d36c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.664+0000 7f660af7f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d36cb00 0x55a08d3751a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.664+0000 7f660af7f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d377ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660af7f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08d32adc0 con 0x55a08d36cb00 2024-03-04T12:26:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660af7f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08d2c6ec0 con 0x55a08d36c1e0 2024-03-04T12:26:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660af7f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08d26fd30 con 0x55a08d3756e0 2024-03-04T12:26:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660605c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d36cb00 0x55a08d3751a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660605c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d36cb00 0x55a08d3751a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58950/0 (socket says 172.21.15.60:58950) 2024-03-04T12:26:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660605c640 1 -- 172.21.15.60:0/1287812361 learned_addr learned my addr 172.21.15.60:0/1287812361 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660685d640 1 --2- 172.21.15.60:0/1287812361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d36c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.682 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f65f77fe640 1 -- 172.21.15.60:0/19090156 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f65f00046c0 con 0x7f6600100570 2024-03-04T12:26:48.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660605c640 1 -- 172.21.15.60:0/1287812361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 msgr2=0x55a08d377ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T12:26:48.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660605c640 1 --2- 172.21.15.60:0/1287812361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d377ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660605c640 1 -- 172.21.15.60:0/1287812361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 msgr2=0x55a08d36c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660605c640 1 --2- 172.21.15.60:0/1287812361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d36c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660605c640 1 -- 172.21.15.60:0/1287812361 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a08cfdb3f0 con 0x55a08d36cb00 2024-03-04T12:26:48.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.665+0000 7f660685d640 1 --2- 172.21.15.60:0/1287812361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d36c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:48.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.666+0000 7f660605c640 1 --2- 172.21.15.60:0/1287812361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d36cb00 0x55a08d3751a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009c70 tx=0x7f65fc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549d5f27fd3056f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.666+0000 7f65f77fe640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f65ec0a2090 0x7f65ec0a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.666+0000 7f65f77fe640 1 -- 172.21.15.60:0/19090156 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f65f00ceb60 con 0x7f6600100570 2024-03-04T12:26:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.666+0000 7f660685d640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f65ec0a2090 0x7f65ec0a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.666+0000 7f65f4bfe640 1 -- 172.21.15.60:0/1287812361 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc004230 con 0x55a08d36cb00 2024-03-04T12:26:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.666+0000 7f65f4bfe640 1 -- 172.21.15.60:0/1287812361 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65fc0043d0 con 0x55a08d36cb00 2024-03-04T12:26:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.666+0000 7f65f4bfe640 1 -- 172.21.15.60:0/1287812361 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65fc019820 con 0x55a08d36cb00 2024-03-04T12:26:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.667+0000 7f660685d640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f65ec0a2090 0x7f65ec0a4550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f65f8005e30 tx=0x7f65f8005d80 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.667+0000 7f660af7f740 1 -- 172.21.15.60:0/1287812361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d36cb00 msgr2=0x55a08d3751a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.667+0000 7f660af7f740 1 --2- 172.21.15.60:0/1287812361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d36cb00 0x55a08d3751a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009c70 tx=0x7f65fc011810 comp rx=0 tx=0).stop 2024-03-04T12:26:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.667+0000 7f660af7f740 1 -- 172.21.15.60:0/1287812361 shutdown_connections 2024-03-04T12:26:48.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.667+0000 7f660af7f740 1 --2- 172.21.15.60:0/1287812361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d377ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.667+0000 7f660af7f740 1 --2- 172.21.15.60:0/1287812361 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d36cb00 0x55a08d3751a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.667+0000 7f660af7f740 1 --2- 172.21.15.60:0/1287812361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d36c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.667+0000 7f660af7f740 1 -- 172.21.15.60:0/1287812361 >> 172.21.15.60:0/1287812361 conn(0x55a08d366910 msgr2=0x55a08d366cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.667+0000 7f660af7f740 1 -- 172.21.15.60:0/1287812361 shutdown_connections 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.667+0000 7f660af7f740 1 -- 172.21.15.60:0/1287812361 wait complete. 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660af7f740 1 -- start start 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660af7f740 1 Processor -- start 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660af7f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d36c1e0 0x55a08d372bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660af7f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d373110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660af7f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d37f7b0 0x55a08d381ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660af7f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08d2654a0 con 0x55a08d36c1e0 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660af7f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08d2c2f60 con 0x55a08d37f7b0 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660af7f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a08d3784d0 con 0x55a08d3756e0 2024-03-04T12:26:48.685 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660af7f740 10 client.0 _ll_register_callbacks cb 0x55a08d2c11c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660605c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d36c1e0 0x55a08d372bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660585b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d373110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660585b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d373110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:44446/0 (socket says 172.21.15.60:44446) 2024-03-04T12:26:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660585b640 1 -- 172.21.15.60:0/3063427335 learned_addr learned my addr 172.21.15.60:0/3063427335 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.668+0000 7f660685d640 1 --2- 172.21.15.60:0/3063427335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d37f7b0 0x55a08d381ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f660585b640 1 -- 172.21.15.60:0/3063427335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d37f7b0 msgr2=0x55a08d381ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f660585b640 1 --2- 172.21.15.60:0/3063427335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d37f7b0 0x55a08d381ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f660585b640 1 -- 172.21.15.60:0/3063427335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d36c1e0 msgr2=0x55a08d372bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.686 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f660585b640 1 --2- 172.21.15.60:0/3063427335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d36c1e0 0x55a08d372bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f660585b640 1 -- 172.21.15.60:0/3063427335 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a08d0a57d0 con 0x55a08d3756e0 2024-03-04T12:26:48.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f660585b640 1 --2- 172.21.15.60:0/3063427335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d373110 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65f00cc510 tx=0x7f65f009a980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a26e8ca785f112b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f65e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-03-04T12:26:48.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f65e9ffb640 1 -- 172.21.15.60:0/3063427335 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f00b2020 con 0x55a08d3756e0 2024-03-04T12:26:48.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f65e9ffb640 1 -- 172.21.15.60:0/3063427335 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f65f00cc070 con 0x55a08d3756e0 2024-03-04T12:26:48.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f660af7f740 1 -- 172.21.15.60:0/3063427335 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65fc0043d0 con 0x55a08d3756e0 2024-03-04T12:26:48.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f65e9ffb640 1 -- 172.21.15.60:0/3063427335 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65f009f0b0 con 0x55a08d3756e0 2024-03-04T12:26:48.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f660af7f740 10 client.25615 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.669+0000 7f660af7f740 1 -- 172.21.15.60:0/3063427335 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55a08d384dc0 con 0x55a08d3756e0 2024-03-04T12:26:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.670+0000 7f65e9ffb640 1 -- 172.21.15.60:0/3063427335 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65fc0043d0 con 0x55a08d3756e0 2024-03-04T12:26:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.670+0000 7f65e9ffb640 1 -- 172.21.15.60:0/3063427335 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f65f009e070 con 0x55a08d3756e0 2024-03-04T12:26:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.670+0000 7f660af7f740 10 client.25615 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.670+0000 7f660af7f740 10 client.25615 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-04T12:26:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.670+0000 7f660af7f740 20 client.25615 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.670+0000 7f660af7f740 1 --2- 172.21.15.60:0/3063427335 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55a08d386280 0x55a08d388670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.670+0000 7f660af7f740 4 client.25615 mds_command: new command op to 24394 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:48.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.670+0000 7f660af7f740 1 -- 172.21.15.60:0/3063427335 --> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a08d2d4400 con 0x55a08d386280 2024-03-04T12:26:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.670+0000 7f660605c640 1 --2- 172.21.15.60:0/3063427335 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55a08d386280 0x55a08d388670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.671+0000 7f660605c640 1 --2- 172.21.15.60:0/3063427335 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55a08d386280 0x55a08d388670 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=37f11a188c1026c server_cookie=66b2f78b446960c4 in_seq=0 out_seq=0 2024-03-04T12:26:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.671+0000 7f65e9ffb640 10 client.25615 ms_handle_connect on v2:172.21.15.86:6832/777854669 2024-03-04T12:26:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.671+0000 7f65e9ffb640 1 -- 172.21.15.60:0/3063427335 <== mds.4 v2:172.21.15.86:6832/777854669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a08d2d4400 con 0x55a08d386280 2024-03-04T12:26:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.671+0000 7f65e9ffb640 10 client.25615 handle_command_reply: tid=0 2024-03-04T12:26:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.671+0000 7f660af7f740 1 client.25615 shutdown 2024-03-04T12:26:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.672+0000 7f660af7f740 1 -- 172.21.15.60:0/3063427335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 msgr2=0x55a08d373110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.672+0000 7f660af7f740 1 --2- 172.21.15.60:0/3063427335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d373110 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65f00cc510 tx=0x7f65f009a980 comp rx=0 tx=0).stop 2024-03-04T12:26:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.672+0000 7f660af7f740 1 -- 172.21.15.60:0/3063427335 shutdown_connections 2024-03-04T12:26:48.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.672+0000 7f660af7f740 1 --2- 172.21.15.60:0/3063427335 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55a08d386280 0x55a08d388670 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.672+0000 7f660af7f740 1 --2- 172.21.15.60:0/3063427335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d37f7b0 0x55a08d381ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.672+0000 7f660af7f740 1 --2- 172.21.15.60:0/3063427335 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d373110 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.672+0000 7f660af7f740 1 --2- 172.21.15.60:0/3063427335 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d36c1e0 0x55a08d372bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.672+0000 7f660af7f740 1 -- 172.21.15.60:0/3063427335 >> 172.21.15.60:0/3063427335 conn(0x55a08d366910 msgr2=0x55a08d368660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.672+0000 7f65e9ffb640 0 client.25615 ms_handle_reset on v2:172.21.15.86:6832/777854669 2024-03-04T12:26:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.672+0000 7f660af7f740 1 -- 172.21.15.60:0/3063427335 shutdown_connections 2024-03-04T12:26:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.672+0000 7f660af7f740 1 -- 172.21.15.60:0/3063427335 wait complete. 2024-03-04T12:26:48.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.672+0000 7f660af7f740 20 client.25615 trim_cache size 0 max 16384 2024-03-04T12:26:48.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660af7f740 1 Processor -- start 2024-03-04T12:26:48.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660af7f740 1 -- start start 2024-03-04T12:26:48.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660af7f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d36dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660af7f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d36e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660af7f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d37f7b0 0x55a08d3729a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660af7f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66001ac2a0 con 0x55a08d37f7b0 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660af7f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66001adc20 con 0x55a08d36c1e0 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660af7f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66001ac900 con 0x55a08d3756e0 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660585b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d36e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660585b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d36e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:44462/0 (socket says 172.21.15.60:44462) 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660585b640 1 -- 172.21.15.60:0/2993724996 learned_addr learned my addr 172.21.15.60:0/2993724996 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660685d640 1 --2- 172.21.15.60:0/2993724996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d37f7b0 0x55a08d3729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.695+0000 7f660605c640 1 --2- 172.21.15.60:0/2993724996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d36dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.696+0000 7f660585b640 1 -- 172.21.15.60:0/2993724996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 msgr2=0x55a08d36dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.696+0000 7f660585b640 1 --2- 172.21.15.60:0/2993724996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d36dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.696+0000 7f660585b640 1 -- 172.21.15.60:0/2993724996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d37f7b0 msgr2=0x55a08d3729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.696+0000 7f660585b640 1 --2- 172.21.15.60:0/2993724996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d37f7b0 0x55a08d3729a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.696+0000 7f660585b640 1 -- 172.21.15.60:0/2993724996 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a08d3783b0 con 0x55a08d3756e0 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.696+0000 7f660685d640 1 --2- 172.21.15.60:0/2993724996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d37f7b0 0x55a08d3729a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.696+0000 7f660585b640 1 --2- 172.21.15.60:0/2993724996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d36e220 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f65f009ac60 tx=0x7f65f000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bbdd28ee79fd182 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.696+0000 7f65ebfff640 1 -- 172.21.15.60:0/2993724996 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f00ab020 con 0x55a08d3756e0 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.696+0000 7f65ebfff640 1 -- 172.21.15.60:0/2993724996 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65f00cbcd0 con 0x55a08d3756e0 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.696+0000 7f65ebfff640 1 -- 172.21.15.60:0/2993724996 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f00b2030 con 0x55a08d3756e0 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.697+0000 7f660af7f740 1 -- 172.21.15.60:0/2993724996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 msgr2=0x55a08d36e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.697+0000 7f660af7f740 1 --2- 172.21.15.60:0/2993724996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d36e220 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f65f009ac60 tx=0x7f65f000cf90 comp rx=0 tx=0).stop 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.697+0000 7f660af7f740 1 -- 172.21.15.60:0/2993724996 shutdown_connections 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.697+0000 7f660af7f740 1 --2- 172.21.15.60:0/2993724996 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d37f7b0 0x55a08d3729a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.697+0000 7f660af7f740 1 --2- 172.21.15.60:0/2993724996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d3756e0 0x55a08d36e220 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.697+0000 7f660af7f740 1 --2- 172.21.15.60:0/2993724996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d36dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.697+0000 7f660af7f740 1 -- 172.21.15.60:0/2993724996 >> 172.21.15.60:0/2993724996 conn(0x55a08d366910 msgr2=0x55a08d38cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.698+0000 7f660af7f740 1 -- 172.21.15.60:0/2993724996 shutdown_connections 2024-03-04T12:26:48.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.698+0000 7f660af7f740 1 -- 172.21.15.60:0/2993724996 wait complete. 2024-03-04T12:26:48.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.698+0000 7f660af7f740 1 -- start start 2024-03-04T12:26:48.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.698+0000 7f660af7f740 1 Processor -- start 2024-03-04T12:26:48.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.698+0000 7f660af7f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d37c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.698+0000 7f660af7f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d3756e0 0x55a08d37ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660af7f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d37f7b0 0x55a08d37b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.711 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660af7f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66001ad5c0 con 0x55a08d3756e0 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660585b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d3756e0 0x55a08d37ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660585b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d3756e0 0x55a08d37ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58966/0 (socket says 172.21.15.60:58966) 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660585b640 1 -- 172.21.15.60:0/2455611626 learned_addr learned my addr 172.21.15.60:0/2455611626 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660af7f740 1 -- 172.21.15.60:0/2455611626 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66001acf60 con 0x55a08d36c1e0 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660af7f740 1 -- 172.21.15.60:0/2455611626 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66001aa560 con 0x55a08d37f7b0 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660685d640 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d37f7b0 0x55a08d37b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660605c640 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d37c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660af7f740 10 client.0 _ll_register_callbacks cb 0x7f65fc0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660685d640 1 -- 172.21.15.60:0/2455611626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 msgr2=0x55a08d37c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660685d640 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d37c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660685d640 1 -- 172.21.15.60:0/2455611626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d3756e0 msgr2=0x55a08d37ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660685d640 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d3756e0 0x55a08d37ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.712 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660685d640 1 -- 172.21.15.60:0/2455611626 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a08d37b710 con 0x55a08d37f7b0 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660585b640 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d3756e0 0x55a08d37ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f660685d640 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d37f7b0 0x55a08d37b1d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f65f8009bf0 tx=0x7f65f80095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce6fcd27624fe829 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.699+0000 7f65e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f65e9ffb640 1 -- 172.21.15.60:0/2455611626 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f8051070 con 0x55a08d37f7b0 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f65e9ffb640 1 -- 172.21.15.60:0/2455611626 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f65f8003fc0 con 0x55a08d37f7b0 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f660af7f740 1 -- 172.21.15.60:0/2455611626 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65f00cbcd0 con 0x55a08d37f7b0 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f65e9ffb640 1 -- 172.21.15.60:0/2455611626 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65f8039040 con 0x55a08d37f7b0 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f660af7f740 10 client.25621 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f660af7f740 1 -- 172.21.15.60:0/2455611626 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55a08d3dd380 con 0x55a08d37f7b0 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f65e9ffb640 1 -- 172.21.15.60:0/2455611626 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65f80046a0 con 0x55a08d37f7b0 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f65e9ffb640 1 -- 172.21.15.60:0/2455611626 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f65f8056b90 con 0x55a08d37f7b0 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f660af7f740 10 client.25621 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f660af7f740 10 client.25621 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-04T12:26:48.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f660af7f740 20 client.25621 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f660af7f740 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55a08d3dda80 0x55a08d3dfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f660af7f740 4 client.25621 mds_command: new command op to 24394 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.700+0000 7f660af7f740 1 -- 172.21.15.60:0/2455611626 --> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f65f00cbcd0 con 0x55a08d3dda80 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.701+0000 7f660605c640 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55a08d3dda80 0x55a08d3dfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.701+0000 7f65e9ffb640 10 client.25621 ms_handle_connect on v2:172.21.15.86:6832/777854669 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.701+0000 7f660605c640 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55a08d3dda80 0x55a08d3dfea0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a9a45fbfa685670e server_cookie=5fc70125851e2fd4 in_seq=0 out_seq=0 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.701+0000 7f65e9ffb640 1 -- 172.21.15.60:0/2455611626 <== mds.4 v2:172.21.15.86:6832/777854669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f65f00cbcd0 con 0x55a08d3dda80 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.701+0000 7f65e9ffb640 10 client.25621 handle_command_reply: tid=0 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.701+0000 7f660af7f740 1 client.25621 shutdown 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.701+0000 7f660af7f740 1 -- 172.21.15.60:0/2455611626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d37f7b0 msgr2=0x55a08d37b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.701+0000 7f660af7f740 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d37f7b0 0x55a08d37b1d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f65f8009bf0 tx=0x7f65f80095c0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.702+0000 7f660af7f740 1 -- 172.21.15.60:0/2455611626 shutdown_connections 2024-03-04T12:26:48.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.702+0000 7f660af7f740 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a08d37f7b0 0x55a08d37b1d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.702+0000 7f660af7f740 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a08d3756e0 0x55a08d37ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.702+0000 7f660af7f740 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55a08d3dda80 0x55a08d3dfea0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.702+0000 7f660af7f740 1 --2- 172.21.15.60:0/2455611626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a08d36c1e0 0x55a08d37c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.702+0000 7f65e9ffb640 0 client.25621 ms_handle_reset on v2:172.21.15.86:6832/777854669 2024-03-04T12:26:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.702+0000 7f660af7f740 1 -- 172.21.15.60:0/2455611626 >> 172.21.15.60:0/2455611626 conn(0x55a08d366910 msgr2=0x55a08d36dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.702+0000 7f660af7f740 1 -- 172.21.15.60:0/2455611626 shutdown_connections 2024-03-04T12:26:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.702+0000 7f660af7f740 1 -- 172.21.15.60:0/2455611626 wait complete. 2024-03-04T12:26:48.715 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.702+0000 7f660af7f740 20 client.25621 trim_cache size 0 max 16384 2024-03-04T12:26:48.715 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:26:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.702+0000 7f66082c0640 1 -- 172.21.15.60:0/19090156 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f65ec0a2090 msgr2=0x7f65ec0a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.703+0000 7f66082c0640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f65ec0a2090 0x7f65ec0a4550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f65f8005e30 tx=0x7f65f8005d80 comp rx=0 tx=0).stop 2024-03-04T12:26:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.703+0000 7f66082c0640 1 -- 172.21.15.60:0/19090156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6600100570 msgr2=0x7f660019a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.703+0000 7f66082c0640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6600100570 0x7f660019a2d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f65f000e9f0 tx=0x7f65f000edf0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.703+0000 7f66082c0640 1 -- 172.21.15.60:0/19090156 shutdown_connections 2024-03-04T12:26:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.703+0000 7f66082c0640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6600105940 0x7f660019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.703+0000 7f66082c0640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f65ec0a2090 0x7f65ec0a4550 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.703+0000 7f66082c0640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6600100570 0x7f660019a2d0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.703+0000 7f66082c0640 1 --2- 172.21.15.60:0/19090156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66000ffbb0 0x7f6600199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.703+0000 7f66082c0640 1 -- 172.21.15.60:0/19090156 >> 172.21.15.60:0/19090156 conn(0x7f6600075490 msgr2=0x7f66000fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.703+0000 7f66082c0640 1 -- 172.21.15.60:0/19090156 shutdown_connections 2024-03-04T12:26:48.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.703+0000 7f66082c0640 1 -- 172.21.15.60:0/19090156 wait complete. 2024-03-04T12:26:48.724 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:26:48.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-04T12:26:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.825+0000 7f4e4d324640 1 -- 172.21.15.60:0/4206450294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e48100df0 msgr2=0x7f4e48104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.825+0000 7f4e4d324640 1 --2- 172.21.15.60:0/4206450294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e48100df0 0x7f4e48104e30 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c009900 tx=0x7f4e3c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.825+0000 7f4e4d324640 1 -- 172.21.15.60:0/4206450294 shutdown_connections 2024-03-04T12:26:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.825+0000 7f4e4d324640 1 --2- 172.21.15.60:0/4206450294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e48105510 0x7f4e4810d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.825+0000 7f4e4d324640 1 --2- 172.21.15.60:0/4206450294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e48100df0 0x7f4e48104e30 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.825+0000 7f4e4d324640 1 --2- 172.21.15.60:0/4206450294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e48100440 0x7f4e48100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.825+0000 7f4e4d324640 1 -- 172.21.15.60:0/4206450294 >> 172.21.15.60:0/4206450294 conn(0x7f4e480773e0 msgr2=0x7f4e480777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.832 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.825+0000 7f4e4d324640 1 -- 172.21.15.60:0/4206450294 shutdown_connections 2024-03-04T12:26:48.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.825+0000 7f4e4d324640 1 -- 172.21.15.60:0/4206450294 wait complete. 2024-03-04T12:26:48.833 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.826+0000 7f4e4d324640 1 Processor -- start 2024-03-04T12:26:48.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.826+0000 7f4e4d324640 1 -- start start 2024-03-04T12:26:48.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.826+0000 7f4e4d324640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e48100440 0x7f4e4819b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.826+0000 7f4e4d324640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e48105510 0x7f4e4819bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.826+0000 7f4e4d324640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e48195840 0x7f4e48195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.835 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.826+0000 7f4e4d324640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4810f4c0 con 0x7f4e48195840 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.826+0000 7f4e4d324640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4810f340 con 0x7f4e48105510 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.826+0000 7f4e4d324640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4810f640 con 0x7f4e48100440 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e46ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e48100440 0x7f4e4819b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e46ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e48100440 0x7f4e4819b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:44494/0 (socket says 172.21.15.60:44494) 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e46ffd640 1 -- 172.21.15.60:0/1348616248 learned_addr learned my addr 172.21.15.60:0/1348616248 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e477fe640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e48195840 0x7f4e48195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e467fc640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e48105510 0x7f4e4819bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e46ffd640 1 -- 172.21.15.60:0/1348616248 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e48105510 msgr2=0x7f4e4819bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e48105510 0x7f4e4819bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e46ffd640 1 -- 172.21.15.60:0/1348616248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e48195840 msgr2=0x7f4e48195cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e48195840 0x7f4e48195cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e46ffd640 1 -- 172.21.15.60:0/1348616248 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e481963d0 con 0x7f4e48100440 2024-03-04T12:26:48.836 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e467fc640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e48105510 0x7f4e4819bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:48.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e477fe640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e48195840 0x7f4e48195cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:48.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e48100440 0x7f4e4819b6d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800e9f0 tx=0x7f4e3800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e2ffff640 1 -- 172.21.15.60:0/1348616248 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3800cd80 con 0x7f4e48100440 2024-03-04T12:26:48.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e2ffff640 1 -- 172.21.15.60:0/1348616248 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e38010910 con 0x7f4e48100440 2024-03-04T12:26:48.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.827+0000 7f4e2ffff640 1 -- 172.21.15.60:0/1348616248 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38018990 con 0x7f4e48100440 2024-03-04T12:26:48.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.828+0000 7f4e4d324640 1 -- 172.21.15.60:0/1348616248 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e481966c0 con 0x7f4e48100440 2024-03-04T12:26:48.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.828+0000 7f4e4d324640 1 -- 172.21.15.60:0/1348616248 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e481a2820 con 0x7f4e48100440 2024-03-04T12:26:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.830+0000 7f4e2ffff640 1 -- 172.21.15.60:0/1348616248 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f4e38010ac0 con 0x7f4e48100440 2024-03-04T12:26:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.831+0000 7f4e4ffd7740 1 Processor -- start 2024-03-04T12:26:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.831+0000 7f4e4ffd7740 1 -- start start 2024-03-04T12:26:48.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.831+0000 7f4e2ffff640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4e300a4220 0x7f4e300a66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.831+0000 7f4e4ffd7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 0x55c941b85db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.831+0000 7f4e4ffd7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.831+0000 7f4e4ffd7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b8eed0 0x55c941b912c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.831+0000 7f4e4ffd7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c941b4cdc0 con 0x55c941b859d0 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.831+0000 7f4e4ffd7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c941ae8ec0 con 0x55c941b862f0 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.831+0000 7f4e4ffd7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c941a91d30 con 0x55c941b8eed0 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.831+0000 7f4e2ffff640 1 -- 172.21.15.60:0/1348616248 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f4e380cec30 con 0x7f4e48100440 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e46ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 0x55c941b85db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e477fe640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4e300a4220 0x7f4e300a66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e46ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 0x55c941b85db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58986/0 (socket says 172.21.15.60:58986) 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e46ffd640 1 -- 172.21.15.60:0/1931825461 learned_addr learned my addr 172.21.15.60:0/1931825461 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e467fc640 1 --2- 172.21.15.60:0/1931825461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e46ffd640 1 -- 172.21.15.60:0/1931825461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b8eed0 msgr2=0x55c941b912c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-04T12:26:48.841 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/1931825461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b8eed0 0x55c941b912c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e46ffd640 1 -- 172.21.15.60:0/1931825461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 msgr2=0x55c941b8e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/1931825461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e46ffd640 1 -- 172.21.15.60:0/1931825461 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9417fd3f0 con 0x55c941b859d0 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/1931825461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 0x55c941b85db0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800c6a0 tx=0x7f4e380659c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850d0786a0a05466 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e477fe640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4e300a4220 0x7f4e300a66e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009fd0 tx=0x7f4e40009500 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.832+0000 7f4e2d7fa640 1 -- 172.21.15.60:0/1931825461 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3809b6d0 con 0x55c941b859d0 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.833+0000 7f4e2d7fa640 1 -- 172.21.15.60:0/1931825461 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e3809afd0 con 0x55c941b859d0 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.833+0000 7f4e2d7fa640 1 -- 172.21.15.60:0/1931825461 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38064ad0 con 0x55c941b859d0 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.833+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1931825461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 msgr2=0x55c941b85db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.833+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/1931825461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 0x55c941b85db0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800c6a0 tx=0x7f4e380659c0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.834+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1931825461 shutdown_connections 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.834+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/1931825461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b8eed0 0x55c941b912c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.842 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.834+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/1931825461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.834+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/1931825461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 0x55c941b85db0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.834+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1931825461 >> 172.21.15.60:0/1931825461 conn(0x55c941b80100 msgr2=0x55c941b804e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.834+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1931825461 shutdown_connections 2024-03-04T12:26:48.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.834+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1931825461 wait complete. 2024-03-04T12:26:48.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.834+0000 7f4e4ffd7740 1 -- start start 2024-03-04T12:26:48.843 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.834+0000 7f4e4ffd7740 1 Processor -- start 2024-03-04T12:26:48.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.834+0000 7f4e4ffd7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 0x55c941b8bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.834+0000 7f4e4ffd7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e4ffd7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b8eed0 0x55c941b8c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e4ffd7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c941a874a0 con 0x55c941b859d0 2024-03-04T12:26:48.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e4ffd7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c941ae4f60 con 0x55c941b862f0 2024-03-04T12:26:48.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e4ffd7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c941b91c00 con 0x55c941b8eed0 2024-03-04T12:26:48.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e4ffd7740 10 client.0 _ll_register_callbacks cb 0x55c941ae31c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:48.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e46ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 0x55c941b8bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e46ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 0x55c941b8bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58992/0 (socket says 172.21.15.60:58992) 2024-03-04T12:26:48.847 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e46ffd640 1 -- 172.21.15.60:0/1324262188 learned_addr learned my addr 172.21.15.60:0/1324262188 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e477fe640 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b8eed0 0x55c941b8c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e467fc640 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e46ffd640 1 -- 172.21.15.60:0/1324262188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b8eed0 msgr2=0x55c941b8c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b8eed0 0x55c941b8c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e46ffd640 1 -- 172.21.15.60:0/1324262188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 msgr2=0x55c941b8c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e46ffd640 1 -- 172.21.15.60:0/1324262188 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c9418c77d0 con 0x55c941b859d0 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e467fc640 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e477fe640 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b8eed0 0x55c941b8c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 0x55c941b8bf40 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4e380a3240 tx=0x7f4e38009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acf629aa767d0964 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.835+0000 7f4e2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e2aff5640 1 -- 172.21.15.60:0/1324262188 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e380b3030 con 0x55c941b859d0 2024-03-04T12:26:48.848 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e2aff5640 1 -- 172.21.15.60:0/1324262188 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f4e38018d30 con 0x55c941b859d0 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1324262188 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e3809afd0 con 0x55c941b859d0 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e2aff5640 1 -- 172.21.15.60:0/1324262188 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e380cbd40 con 0x55c941b859d0 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e4ffd7740 10 client.16095 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1324262188 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55c941b9dd70 con 0x55c941b859d0 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e2aff5640 1 -- 172.21.15.60:0/1324262188 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e3809afd0 con 0x55c941b859d0 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e2aff5640 1 -- 172.21.15.60:0/1324262188 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4e380cc0e0 con 0x55c941b859d0 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e4ffd7740 10 client.16095 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e4ffd7740 10 client.16095 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e4ffd7740 20 client.16095 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55c941b9f0c0 0x55c941ba14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e4ffd7740 4 client.16095 mds_command: new command op to 24407 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.836+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1324262188 --> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c941af6400 con 0x55c941b9f0c0 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.837+0000 7f4e467fc640 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55c941b9f0c0 0x55c941ba14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.849 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.837+0000 7f4e2aff5640 10 client.16095 ms_handle_connect on v2:172.21.15.63:6832/2270964219 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.837+0000 7f4e467fc640 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55c941b9f0c0 0x55c941ba14b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dfb4fbc9c72c9c59 server_cookie=e252500a0ac3fb2d in_seq=0 out_seq=0 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e2aff5640 1 -- 172.21.15.60:0/1324262188 <== mds.3 v2:172.21.15.63:6832/2270964219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c941af6400 con 0x55c941b9f0c0 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e2aff5640 10 client.16095 handle_command_reply: tid=0 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e4ffd7740 1 client.16095 shutdown 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1324262188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 msgr2=0x55c941b8bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 0x55c941b8bf40 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4e380a3240 tx=0x7f4e38009310 comp rx=0 tx=0).stop 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1324262188 shutdown_connections 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55c941b9f0c0 0x55c941ba14b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b8eed0 0x55c941b8c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e2aff5640 0 client.16095 ms_handle_reset on v2:172.21.15.63:6832/2270964219 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/1324262188 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b859d0 0x55c941b8bf40 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1324262188 >> 172.21.15.60:0/1324262188 conn(0x55c941b80100 msgr2=0x55c941b81e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1324262188 shutdown_connections 2024-03-04T12:26:48.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.838+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/1324262188 wait complete. 2024-03-04T12:26:48.851 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.839+0000 7f4e4ffd7740 20 client.16095 trim_cache size 0 max 16384 2024-03-04T12:26:48.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.861+0000 7f4e4ffd7740 1 Processor -- start 2024-03-04T12:26:48.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e4ffd7740 1 -- start start 2024-03-04T12:26:48.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e4ffd7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e4ffd7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 0x55c941b8b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e4ffd7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b81c60 0x55c941b891c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e4ffd7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e481ac260 con 0x55c941b8eed0 2024-03-04T12:26:48.869 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e4ffd7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e481adbe0 con 0x55c941b862f0 2024-03-04T12:26:48.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e4ffd7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e481ac8c0 con 0x55c941b81c60 2024-03-04T12:26:48.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e467fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 0x55c941b8b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e467fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 0x55c941b8b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58998/0 (socket says 172.21.15.60:58998) 2024-03-04T12:26:48.870 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e467fc640 1 -- 172.21.15.60:0/883840868 learned_addr learned my addr 172.21.15.60:0/883840868 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.874 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/883840868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e477fe640 1 --2- 172.21.15.60:0/883840868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b81c60 0x55c941b891c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e467fc640 1 -- 172.21.15.60:0/883840868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b81c60 msgr2=0x55c941b891c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e467fc640 1 --2- 172.21.15.60:0/883840868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b81c60 0x55c941b891c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e467fc640 1 -- 172.21.15.60:0/883840868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 msgr2=0x55c941b8e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e467fc640 1 --2- 172.21.15.60:0/883840868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e467fc640 1 -- 172.21.15.60:0/883840868 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c941b7f920 con 0x55c941b8eed0 2024-03-04T12:26:48.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.862+0000 7f4e477fe640 1 --2- 172.21.15.60:0/883840868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b81c60 0x55c941b891c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:48.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.863+0000 7f4e467fc640 1 --2- 172.21.15.60:0/883840868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 0x55c941b8b0a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c02cfd0 tx=0x7f4e3c0058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa891a26942afc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.863+0000 7f4e2cff9640 1 -- 172.21.15.60:0/883840868 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c02cce0 con 0x55c941b8eed0 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.863+0000 7f4e2cff9640 1 -- 172.21.15.60:0/883840868 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e3c044b70 con 0x55c941b8eed0 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.863+0000 7f4e2cff9640 1 -- 172.21.15.60:0/883840868 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c005040 con 0x55c941b8eed0 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.863+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/883840868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 msgr2=0x55c941b8b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.863+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/883840868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 0x55c941b8b0a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c02cfd0 tx=0x7f4e3c0058b0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/883840868 shutdown_connections 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/883840868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b81c60 0x55c941b891c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/883840868 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 0x55c941b8b0a0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/883840868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941b8e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/883840868 >> 172.21.15.60:0/883840868 conn(0x55c941b80100 msgr2=0x55c941b81550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/883840868 shutdown_connections 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/883840868 wait complete. 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 -- start start 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 Processor -- start 2024-03-04T12:26:48.876 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b81c60 0x55c941ba39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941ba1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 0x55c941ba2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e481ad580 con 0x55c941b8eed0 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e481acf20 con 0x55c941b862f0 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e481aa500 con 0x55c941b81c60 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.864+0000 7f4e4ffd7740 10 client.0 _ll_register_callbacks cb 0x7f4e38096b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e467fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941ba1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 0x55c941ba2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e467fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941ba1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:39926/0 (socket says 172.21.15.60:39926) 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e467fc640 1 -- 172.21.15.60:0/693781506 learned_addr learned my addr 172.21.15.60:0/693781506 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b81c60 0x55c941ba39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e467fc640 1 -- 172.21.15.60:0/693781506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b81c60 msgr2=0x55c941ba39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e467fc640 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b81c60 0x55c941ba39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e467fc640 1 -- 172.21.15.60:0/693781506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 msgr2=0x55c941ba2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:48.877 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e467fc640 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 0x55c941ba2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e467fc640 1 -- 172.21.15.60:0/693781506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c941bf51a0 con 0x55c941b862f0 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e477fe640 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 0x55c941ba2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e467fc640 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941ba1d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c009230 tx=0x7f4e3c0443b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f07125bee7096efb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e2aff5640 1 -- 172.21.15.60:0/693781506 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c052020 con 0x55c941b862f0 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e2aff5640 1 -- 172.21.15.60:0/693781506 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f4e3c0446d0 con 0x55c941b862f0 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.865+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/693781506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e3c044b70 con 0x55c941b862f0 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.866+0000 7f4e2aff5640 1 -- 172.21.15.60:0/693781506 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e3c0448d0 con 0x55c941b862f0 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.866+0000 7f4e4ffd7740 10 client.25568 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.866+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/693781506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55c941bf6300 con 0x55c941b862f0 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.866+0000 7f4e2aff5640 1 -- 172.21.15.60:0/693781506 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e3c005050 con 0x55c941b862f0 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.866+0000 7f4e2aff5640 1 -- 172.21.15.60:0/693781506 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4e3c055020 con 0x55c941b862f0 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.866+0000 7f4e4ffd7740 10 client.25568 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.866+0000 7f4e4ffd7740 10 client.25568 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-04T12:26:48.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.866+0000 7f4e4ffd7740 20 client.25568 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:48.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.866+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55c941bf69d0 0x55c941bf8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:48.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.866+0000 7f4e4ffd7740 4 client.25568 mds_command: new command op to 24407 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:26:48.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.866+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/693781506 --> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f4e3c044b70 con 0x55c941bf69d0 2024-03-04T12:26:48.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.866+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55c941bf69d0 0x55c941bf8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:48.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e2aff5640 10 client.25568 ms_handle_connect on v2:172.21.15.63:6832/2270964219 2024-03-04T12:26:48.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e46ffd640 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55c941bf69d0 0x55c941bf8df0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=58bc200f6aca9499 server_cookie=2e92dee11e82431c in_seq=0 out_seq=0 2024-03-04T12:26:48.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e2aff5640 1 -- 172.21.15.60:0/693781506 <== mds.3 v2:172.21.15.63:6832/2270964219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f4e3c044b70 con 0x55c941bf69d0 2024-03-04T12:26:48.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e2aff5640 10 client.25568 handle_command_reply: tid=0 2024-03-04T12:26:48.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e4ffd7740 1 client.25568 shutdown 2024-03-04T12:26:48.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/693781506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 msgr2=0x55c941ba1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.880 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941ba1d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c009230 tx=0x7f4e3c0443b0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/693781506 shutdown_connections 2024-03-04T12:26:48.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55c941bf69d0 0x55c941bf8df0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c941b8eed0 0x55c941ba2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e2aff5640 0 client.25568 ms_handle_reset on v2:172.21.15.63:6832/2270964219 2024-03-04T12:26:48.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c941b862f0 0x55c941ba1d40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e4ffd7740 1 --2- 172.21.15.60:0/693781506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c941b81c60 0x55c941ba39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/693781506 >> 172.21.15.60:0/693781506 conn(0x55c941b80100 msgr2=0x55c941b8b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/693781506 shutdown_connections 2024-03-04T12:26:48.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e4ffd7740 1 -- 172.21.15.60:0/693781506 wait complete. 2024-03-04T12:26:48.881 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.867+0000 7f4e4ffd7740 20 client.25568 trim_cache size 0 max 16384 2024-03-04T12:26:48.881 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:26:48.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.868+0000 7f4e4d324640 1 -- 172.21.15.60:0/1348616248 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4e300a4220 msgr2=0x7f4e300a66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.868+0000 7f4e4d324640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4e300a4220 0x7f4e300a66e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009fd0 tx=0x7f4e40009500 comp rx=0 tx=0).stop 2024-03-04T12:26:48.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.868+0000 7f4e4d324640 1 -- 172.21.15.60:0/1348616248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e48100440 msgr2=0x7f4e4819b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.868+0000 7f4e4d324640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e48100440 0x7f4e4819b6d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800e9f0 tx=0x7f4e3800eeb0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.868+0000 7f4e4d324640 1 -- 172.21.15.60:0/1348616248 shutdown_connections 2024-03-04T12:26:48.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.868+0000 7f4e4d324640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4e300a4220 0x7f4e300a66e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.868+0000 7f4e4d324640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4e48195840 0x7f4e48195cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.868+0000 7f4e4d324640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4e48105510 0x7f4e4819bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.868+0000 7f4e4d324640 1 --2- 172.21.15.60:0/1348616248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e48100440 0x7f4e4819b6d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.868+0000 7f4e4d324640 1 -- 172.21.15.60:0/1348616248 >> 172.21.15.60:0/1348616248 conn(0x7f4e480773e0 msgr2=0x7f4e480ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.868+0000 7f4e4d324640 1 -- 172.21.15.60:0/1348616248 shutdown_connections 2024-03-04T12:26:48.883 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.868+0000 7f4e4d324640 1 -- 172.21.15.60:0/1348616248 wait complete. 2024-03-04T12:26:48.886 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:26:48.886 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-04T12:26:48.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.990+0000 7fd3b7716640 1 -- 172.21.15.60:0/3342371318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3b0101ef0 msgr2=0x7fd3b0105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:48.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.990+0000 7fd3b7716640 1 --2- 172.21.15.60:0/3342371318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3b0101ef0 0x7fd3b0105f30 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009900 tx=0x7fd3a002bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:48.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.990+0000 7fd3b7716640 1 -- 172.21.15.60:0/3342371318 shutdown_connections 2024-03-04T12:26:48.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.990+0000 7fd3b7716640 1 --2- 172.21.15.60:0/3342371318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3b0106610 0x7fd3b010e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.990+0000 7fd3b7716640 1 --2- 172.21.15.60:0/3342371318 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3b0101ef0 0x7fd3b0105f30 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.990+0000 7fd3b7716640 1 --2- 172.21.15.60:0/3342371318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3b0101540 0x7fd3b0101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:48.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.990+0000 7fd3b7716640 1 -- 172.21.15.60:0/3342371318 >> 172.21.15.60:0/3342371318 conn(0x7fd3b00fcdc0 msgr2=0x7fd3b00ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:48.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.990+0000 7fd3b7716640 1 -- 172.21.15.60:0/3342371318 shutdown_connections 2024-03-04T12:26:48.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.990+0000 7fd3b7716640 1 -- 172.21.15.60:0/3342371318 wait complete. 2024-03-04T12:26:48.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.991+0000 7fd3b7716640 1 Processor -- start 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.991+0000 7fd3b7716640 1 -- start start 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.991+0000 7fd3b7716640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3b0101540 0x7fd3b0197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.991+0000 7fd3b7716640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3b0106610 0x7fd3b0198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.991+0000 7fd3b7716640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3b019c6a0 0x7fd3b019cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.991+0000 7fd3b7716640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b01116d0 con 0x7fd3b0101540 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.991+0000 7fd3b7716640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b0111550 con 0x7fd3b0106610 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.991+0000 7fd3b7716640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b0111850 con 0x7fd3b019c6a0 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.991+0000 7fd3b54b2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3b0101540 0x7fd3b0197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.991+0000 7fd3b54b2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3b0101540 0x7fd3b0197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59024/0 (socket says 172.21.15.60:59024) 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.991+0000 7fd3b54b2640 1 -- 172.21.15.60:0/2264206588 learned_addr learned my addr 172.21.15.60:0/2264206588 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.991+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3b019c6a0 0x7fd3b019cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.992+0000 7fd3b54b2640 1 -- 172.21.15.60:0/2264206588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3b019c6a0 msgr2=0x7fd3b019cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.992+0000 7fd3b54b2640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3b019c6a0 0x7fd3b019cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.992+0000 7fd3b54b2640 1 -- 172.21.15.60:0/2264206588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3b0106610 msgr2=0x7fd3b0198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.992+0000 7fd3b4cb1640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3b0106610 0x7fd3b0198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.992+0000 7fd3b54b2640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3b0106610 0x7fd3b0198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.992+0000 7fd3b54b2640 1 -- 172.21.15.60:0/2264206588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b019d190 con 0x7fd3b0101540 2024-03-04T12:26:49.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.992+0000 7fd3b54b2640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3b0101540 0x7fd3b0197de0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00d950 tx=0x7fd3ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.992+0000 7fd3a67fc640 1 -- 172.21.15.60:0/2264206588 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac017030 con 0x7fd3b0101540 2024-03-04T12:26:49.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.992+0000 7fd3a67fc640 1 -- 172.21.15.60:0/2264206588 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3ac004020 con 0x7fd3b0101540 2024-03-04T12:26:49.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.992+0000 7fd3a67fc640 1 -- 172.21.15.60:0/2264206588 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0047a0 con 0x7fd3b0101540 2024-03-04T12:26:49.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.992+0000 7fd3b7716640 1 -- 172.21.15.60:0/2264206588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b019d480 con 0x7fd3b0101540 2024-03-04T12:26:49.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.992+0000 7fd3b7716640 1 -- 172.21.15.60:0/2264206588 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b0073820 con 0x7fd3b0101540 2024-03-04T12:26:49.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3ba3cc740 1 Processor -- start 2024-03-04T12:26:49.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3ba3cc740 1 -- start start 2024-03-04T12:26:49.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3ba3cc740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f18bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3ba3cc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f2a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.004 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3ba3cc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f2cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3ba3cc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c209edfdc0 con 0x55c209f2a6e0 2024-03-04T12:26:49.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3ba3cc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c209e7bec0 con 0x55c209f19190 2024-03-04T12:26:49.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3ba3cc740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c209e24d30 con 0x55c209f187e0 2024-03-04T12:26:49.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3a67fc640 1 -- 172.21.15.60:0/2264206588 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7fd3ac0041d0 con 0x7fd3b0101540 2024-03-04T12:26:49.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3b4cb1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f2cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.005 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3b4cb1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f2cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59036/0 (socket says 172.21.15.60:59036) 2024-03-04T12:26:49.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3b4cb1640 1 -- 172.21.15.60:0/1922637706 learned_addr learned my addr 172.21.15.60:0/1922637706 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/1922637706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f18bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3b54b2640 1 --2- 172.21.15.60:0/1922637706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f2a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.996+0000 7fd3b5cb3640 1 -- 172.21.15.60:0/1922637706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 msgr2=0x55c209f2a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.997+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/1922637706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f2a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.997+0000 7fd3b5cb3640 1 -- 172.21.15.60:0/1922637706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 msgr2=0x55c209f2cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.006 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.997+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/1922637706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f2cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.997+0000 7fd3b5cb3640 1 -- 172.21.15.60:0/1922637706 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c209b903f0 con 0x55c209f187e0 2024-03-04T12:26:49.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.997+0000 7fd3b4cb1640 1 --2- 172.21.15.60:0/1922637706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f2cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:49.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.997+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/1922637706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f18bc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009e80 tx=0x7fd3a80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6be02a7464879a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.997+0000 7fd3a67fc640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd39c0a2090 0x7fd39c0a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.007 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.997+0000 7fd39b7fe640 1 -- 172.21.15.60:0/1922637706 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8003b70 con 0x55c209f187e0 2024-03-04T12:26:49.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.997+0000 7fd39b7fe640 1 -- 172.21.15.60:0/1922637706 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3a8015070 con 0x55c209f187e0 2024-03-04T12:26:49.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.997+0000 7fd39b7fe640 1 -- 172.21.15.60:0/1922637706 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8011860 con 0x55c209f187e0 2024-03-04T12:26:49.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.997+0000 7fd3b54b2640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd39c0a2090 0x7fd39c0a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.008 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.997+0000 7fd3a67fc640 1 -- 172.21.15.60:0/2264206588 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fd3ac0ce240 con 0x7fd3b0101540 2024-03-04T12:26:49.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.998+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1922637706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 msgr2=0x55c209f18bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.998+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/1922637706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f18bc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009e80 tx=0x7fd3a80107a0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.998+0000 7fd3b54b2640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd39c0a2090 0x7fd39c0a4550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0cb4d0 tx=0x7fd3ac000bc0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.998+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1922637706 shutdown_connections 2024-03-04T12:26:49.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.998+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/1922637706 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f2cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.998+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/1922637706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f2a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.998+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/1922637706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f18bc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.998+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1922637706 >> 172.21.15.60:0/1922637706 conn(0x55c209f13100 msgr2=0x55c209f134e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.998+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1922637706 shutdown_connections 2024-03-04T12:26:49.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.998+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1922637706 wait complete. 2024-03-04T12:26:49.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.998+0000 7fd3ba3cc740 1 -- start start 2024-03-04T12:26:49.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3ba3cc740 1 Processor -- start 2024-03-04T12:26:49.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3ba3cc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f187e0 0x55c209f276b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3ba3cc740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f19190 0x55c209f27bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3ba3cc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f2a6e0 0x55c209f28130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3ba3cc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c209e1a4a0 con 0x55c209f187e0 2024-03-04T12:26:49.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3ba3cc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c209e77f60 con 0x55c209f2a6e0 2024-03-04T12:26:49.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3ba3cc740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c209f2d4a0 con 0x55c209f19190 2024-03-04T12:26:49.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3ba3cc740 10 client.0 _ll_register_callbacks cb 0x55c209e761c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:49.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3b5cb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f187e0 0x55c209f276b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3b5cb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f187e0 0x55c209f276b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59050/0 (socket says 172.21.15.60:59050) 2024-03-04T12:26:49.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3b5cb3640 1 -- 172.21.15.60:0/3956562603 learned_addr learned my addr 172.21.15.60:0/3956562603 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3b4cb1640 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f19190 0x55c209f27bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3b5cb3640 1 -- 172.21.15.60:0/3956562603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f19190 msgr2=0x55c209f27bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:48.999+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f19190 0x55c209f27bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd3b5cb3640 1 -- 172.21.15.60:0/3956562603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f2a6e0 msgr2=0x55c209f28130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f2a6e0 0x55c209f28130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd3b5cb3640 1 -- 172.21.15.60:0/3956562603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c209c5a7d0 con 0x55c209f187e0 2024-03-04T12:26:49.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd3b4cb1640 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f19190 0x55c209f27bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:49.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f187e0 0x55c209f276b0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8010ca0 tx=0x7fd3a8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2996f511450b37df server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd398ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:49.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd398ff9640 1 -- 172.21.15.60:0/3956562603 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a802a060 con 0x55c209f187e0 2024-03-04T12:26:49.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd398ff9640 1 -- 172.21.15.60:0/3956562603 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fd3a8011b90 con 0x55c209f187e0 2024-03-04T12:26:49.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/3956562603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd3a8015070 con 0x55c209f187e0 2024-03-04T12:26:49.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd398ff9640 1 -- 172.21.15.60:0/3956562603 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3a8011d90 con 0x55c209f187e0 2024-03-04T12:26:49.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd3ba3cc740 10 client.16113 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:49.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/3956562603 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55c209f395a0 con 0x55c209f187e0 2024-03-04T12:26:49.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.000+0000 7fd398ff9640 1 -- 172.21.15.60:0/3956562603 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3a8015070 con 0x55c209f187e0 2024-03-04T12:26:49.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.001+0000 7fd398ff9640 1 -- 172.21.15.60:0/3956562603 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd3a8025640 con 0x55c209f187e0 2024-03-04T12:26:49.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.001+0000 7fd3ba3cc740 10 client.16113 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:49.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.001+0000 7fd3ba3cc740 10 client.16113 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-04T12:26:49.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.001+0000 7fd3ba3cc740 20 client.16113 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:49.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.001+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55c209f3aa00 0x55c209f3cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.001+0000 7fd3ba3cc740 4 client.16113 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:49.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.001+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/3956562603 --> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c209e89400 con 0x55c209f3aa00 2024-03-04T12:26:49.019 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.001+0000 7fd3b4cb1640 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55c209f3aa00 0x55c209f3cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.001+0000 7fd398ff9640 10 client.16113 ms_handle_connect on v2:172.21.15.63:6836/3974574804 2024-03-04T12:26:49.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.001+0000 7fd3b4cb1640 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55c209f3aa00 0x55c209f3cdf0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=70105a45c2eda56e server_cookie=a82b15bea8977e41 in_seq=0 out_seq=0 2024-03-04T12:26:49.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.002+0000 7fd398ff9640 1 -- 172.21.15.60:0/3956562603 <== mds.2 v2:172.21.15.63:6836/3974574804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c209e89400 con 0x55c209f3aa00 2024-03-04T12:26:49.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.002+0000 7fd398ff9640 10 client.16113 handle_command_reply: tid=0 2024-03-04T12:26:49.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.002+0000 7fd3ba3cc740 1 client.16113 shutdown 2024-03-04T12:26:49.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.002+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/3956562603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f187e0 msgr2=0x55c209f276b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.002+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f187e0 0x55c209f276b0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8010ca0 tx=0x7fd3a8011810 comp rx=0 tx=0).stop 2024-03-04T12:26:49.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.002+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/3956562603 shutdown_connections 2024-03-04T12:26:49.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.002+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55c209f3aa00 0x55c209f3cdf0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.002+0000 7fd398ff9640 0 client.16113 ms_handle_reset on v2:172.21.15.63:6836/3974574804 2024-03-04T12:26:49.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.002+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f2a6e0 0x55c209f28130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.002+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f19190 0x55c209f27bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.002+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/3956562603 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f187e0 0x55c209f276b0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.002+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/3956562603 >> 172.21.15.60:0/3956562603 conn(0x55c209f13100 msgr2=0x55c209f14e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.003+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/3956562603 shutdown_connections 2024-03-04T12:26:49.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.003+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/3956562603 wait complete. 2024-03-04T12:26:49.021 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.003+0000 7fd3ba3cc740 20 client.16113 trim_cache size 0 max 16384 2024-03-04T12:26:49.028 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3ba3cc740 1 Processor -- start 2024-03-04T12:26:49.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3ba3cc740 1 -- start start 2024-03-04T12:26:49.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3ba3cc740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f29de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3ba3cc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f31710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3ba3cc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f31c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3ba3cc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b01ac890 con 0x55c209f2a6e0 2024-03-04T12:26:49.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3ba3cc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b01ae210 con 0x55c209f19190 2024-03-04T12:26:49.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3ba3cc740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b01acef0 con 0x55c209f187e0 2024-03-04T12:26:49.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3b5cb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f31c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3b5cb3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f29de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.029 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3b4cb1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f31710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3b5cb3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f29de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:44580/0 (socket says 172.21.15.60:44580) 2024-03-04T12:26:49.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3b5cb3640 1 -- 172.21.15.60:0/2438246684 learned_addr learned my addr 172.21.15.60:0/2438246684 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3b5cb3640 1 -- 172.21.15.60:0/2438246684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 msgr2=0x55c209f31710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/2438246684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f31710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3b5cb3640 1 -- 172.21.15.60:0/2438246684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 msgr2=0x55c209f31c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/2438246684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f31c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.015+0000 7fd3b5cb3640 1 -- 172.21.15.60:0/2438246684 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c209f2ce00 con 0x55c209f187e0 2024-03-04T12:26:49.030 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.016+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/2438246684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f29de0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8005a20 tx=0x7fd3a80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4a35402ab36464a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.016+0000 7fd39affd640 1 -- 172.21.15.60:0/2438246684 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a802a040 con 0x55c209f187e0 2024-03-04T12:26:49.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.016+0000 7fd39affd640 1 -- 172.21.15.60:0/2438246684 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3a8015040 con 0x55c209f187e0 2024-03-04T12:26:49.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.016+0000 7fd39affd640 1 -- 172.21.15.60:0/2438246684 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a801c6d0 con 0x55c209f187e0 2024-03-04T12:26:49.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.017+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/2438246684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 msgr2=0x55c209f29de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.017+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/2438246684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f29de0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8005a20 tx=0x7fd3a80107a0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.017+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/2438246684 shutdown_connections 2024-03-04T12:26:49.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.017+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/2438246684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f31c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.017+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/2438246684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f31710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.017+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/2438246684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f29de0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.017+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/2438246684 >> 172.21.15.60:0/2438246684 conn(0x55c209f13100 msgr2=0x55c209f413b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.031 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.017+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/2438246684 shutdown_connections 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.017+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/2438246684 wait complete. 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.017+0000 7fd3ba3cc740 1 -- start start 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.017+0000 7fd3ba3cc740 1 Processor -- start 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3ba3cc740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f2f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3ba3cc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f2fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3ba3cc740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f8fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3ba3cc740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b01adbb0 con 0x55c209f2a6e0 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3ba3cc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b01ad550 con 0x55c209f19190 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3ba3cc740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b01aab30 con 0x55c209f187e0 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3ba3cc740 10 client.0 _ll_register_callbacks cb 0x7fd3a80099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3b5cb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f8fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3b5cb3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f8fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59066/0 (socket says 172.21.15.60:59066) 2024-03-04T12:26:49.032 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3b5cb3640 1 -- 172.21.15.60:0/1452379396 learned_addr learned my addr 172.21.15.60:0/1452379396 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f2f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3b4cb1640 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f2fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3b4cb1640 1 -- 172.21.15.60:0/1452379396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 msgr2=0x55c209f2f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3b4cb1640 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f2f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3b4cb1640 1 -- 172.21.15.60:0/1452379396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 msgr2=0x55c209f8fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3b4cb1640 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f8fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3b4cb1640 1 -- 172.21.15.60:0/1452379396 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c209f90070 con 0x55c209f19190 2024-03-04T12:26:49.033 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.018+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f8fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.019+0000 7fd3b4cb1640 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f2fc40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55c209f14290 tx=0x7fd3a00044b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eecc558796d8d6c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.019+0000 7fd398ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.019+0000 7fd398ff9640 1 -- 172.21.15.60:0/1452379396 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0050060 con 0x55c209f19190 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.019+0000 7fd398ff9640 1 -- 172.21.15.60:0/1452379396 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fd3a003a040 con 0x55c209f19190 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.019+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1452379396 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd3a8015040 con 0x55c209f19190 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.019+0000 7fd398ff9640 1 -- 172.21.15.60:0/1452379396 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3a0057050 con 0x55c209f19190 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.019+0000 7fd3ba3cc740 10 client.25580 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.019+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1452379396 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55c209f91150 con 0x55c209f19190 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.019+0000 7fd398ff9640 1 -- 172.21.15.60:0/1452379396 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3a002ca90 con 0x55c209f19190 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.020+0000 7fd398ff9640 1 -- 172.21.15.60:0/1452379396 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd3a0034540 con 0x55c209f19190 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.020+0000 7fd3ba3cc740 10 client.25580 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.020+0000 7fd3ba3cc740 10 client.25580 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.020+0000 7fd3ba3cc740 20 client.25580 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:49.034 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.020+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55c209f3aa00 0x55c209f93870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.020+0000 7fd3ba3cc740 4 client.25580 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:26:49.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.020+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1452379396 --> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7fd3a8015040 con 0x55c209f3aa00 2024-03-04T12:26:49.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.020+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55c209f3aa00 0x55c209f93870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.020+0000 7fd398ff9640 10 client.25580 ms_handle_connect on v2:172.21.15.63:6836/3974574804 2024-03-04T12:26:49.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3b5cb3640 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55c209f3aa00 0x55c209f93870 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3110565d8aeb8527 server_cookie=3c3f40dfc87c21d9 in_seq=0 out_seq=0 2024-03-04T12:26:49.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd398ff9640 1 -- 172.21.15.60:0/1452379396 <== mds.2 v2:172.21.15.63:6836/3974574804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7fd3a8015040 con 0x55c209f3aa00 2024-03-04T12:26:49.035 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd398ff9640 10 client.25580 handle_command_reply: tid=0 2024-03-04T12:26:49.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3ba3cc740 1 client.25580 shutdown 2024-03-04T12:26:49.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1452379396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 msgr2=0x55c209f2fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f2fc40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55c209f14290 tx=0x7fd3a00044b0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1452379396 shutdown_connections 2024-03-04T12:26:49.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55c209f3aa00 0x55c209f93870 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c209f2a6e0 0x55c209f8fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.036 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c209f19190 0x55c209f2fc40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd398ff9640 0 client.25580 ms_handle_reset on v2:172.21.15.63:6836/3974574804 2024-03-04T12:26:49.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3ba3cc740 1 --2- 172.21.15.60:0/1452379396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c209f187e0 0x55c209f2f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1452379396 >> 172.21.15.60:0/1452379396 conn(0x55c209f13100 msgr2=0x55c209f27670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1452379396 shutdown_connections 2024-03-04T12:26:49.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3ba3cc740 1 -- 172.21.15.60:0/1452379396 wait complete. 2024-03-04T12:26:49.037 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.021+0000 7fd3ba3cc740 20 client.25580 trim_cache size 0 max 16384 2024-03-04T12:26:49.038 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:26:49.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.022+0000 7fd3b7716640 1 -- 172.21.15.60:0/2264206588 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd39c0a2090 msgr2=0x7fd39c0a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.022+0000 7fd3b7716640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd39c0a2090 0x7fd39c0a4550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0cb4d0 tx=0x7fd3ac000bc0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.022+0000 7fd3b7716640 1 -- 172.21.15.60:0/2264206588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3b0101540 msgr2=0x7fd3b0197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.022+0000 7fd3b7716640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3b0101540 0x7fd3b0197de0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00d950 tx=0x7fd3ac00de10 comp rx=0 tx=0).stop 2024-03-04T12:26:49.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.022+0000 7fd3b7716640 1 -- 172.21.15.60:0/2264206588 shutdown_connections 2024-03-04T12:26:49.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.022+0000 7fd3b7716640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3b019c6a0 0x7fd3b019cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.022+0000 7fd3b7716640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3b0106610 0x7fd3b0198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.040 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.022+0000 7fd3b7716640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd39c0a2090 0x7fd39c0a4550 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.022+0000 7fd3b7716640 1 --2- 172.21.15.60:0/2264206588 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd3b0101540 0x7fd3b0197de0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.022+0000 7fd3b7716640 1 -- 172.21.15.60:0/2264206588 >> 172.21.15.60:0/2264206588 conn(0x7fd3b00fcdc0 msgr2=0x7fd3b00fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.022+0000 7fd3b7716640 1 -- 172.21.15.60:0/2264206588 shutdown_connections 2024-03-04T12:26:49.041 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.022+0000 7fd3b7716640 1 -- 172.21.15.60:0/2264206588 wait complete. 2024-03-04T12:26:49.041 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:26:49.041 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-04T12:26:49.152 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.143+0000 7f582408e640 1 -- 172.21.15.60:0/553478492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f581c102e70 msgr2=0x7f581c10b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.143+0000 7f582408e640 1 --2- 172.21.15.60:0/553478492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f581c102e70 0x7f581c10b350 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f580c009900 tx=0x7f580c02bcd0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.144+0000 7f582408e640 1 -- 172.21.15.60:0/553478492 shutdown_connections 2024-03-04T12:26:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.144+0000 7f582408e640 1 --2- 172.21.15.60:0/553478492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f581c10b920 0x7f581c10dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.144+0000 7f582408e640 1 --2- 172.21.15.60:0/553478492 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f581c102e70 0x7f581c10b350 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.144+0000 7f582408e640 1 --2- 172.21.15.60:0/553478492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f581c102550 0x7f581c102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.144+0000 7f582408e640 1 -- 172.21.15.60:0/553478492 >> 172.21.15.60:0/553478492 conn(0x7f581c0773e0 msgr2=0x7f581c0777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.144+0000 7f582408e640 1 -- 172.21.15.60:0/553478492 shutdown_connections 2024-03-04T12:26:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.144+0000 7f582408e640 1 -- 172.21.15.60:0/553478492 wait complete. 2024-03-04T12:26:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.144+0000 7f582408e640 1 Processor -- start 2024-03-04T12:26:49.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.144+0000 7f582408e640 1 -- start start 2024-03-04T12:26:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.144+0000 7f582408e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f581c102550 0x7f581c19d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.144+0000 7f582408e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f581c102e70 0x7f581c19dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.144+0000 7f582408e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f581c10b920 0x7f581c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f582408e640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c1116a0 con 0x7f581c10b920 2024-03-04T12:26:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f582408e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c111520 con 0x7f581c102e70 2024-03-04T12:26:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f582408e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c111820 con 0x7f581c102550 2024-03-04T12:26:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f582262b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f581c10b920 0x7f581c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f582262b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f581c10b920 0x7f581c197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59076/0 (socket says 172.21.15.60:59076) 2024-03-04T12:26:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f582262b640 1 -- 172.21.15.60:0/1593538501 learned_addr learned my addr 172.21.15.60:0/1593538501 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f5821e2a640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f581c102550 0x7f581c19d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f5821629640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f581c102e70 0x7f581c19dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f582262b640 1 -- 172.21.15.60:0/1593538501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f581c102550 msgr2=0x7f581c19d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f582262b640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f581c102550 0x7f581c19d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f582262b640 1 -- 172.21.15.60:0/1593538501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f581c102e70 msgr2=0x7f581c19dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f582262b640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f581c102e70 0x7f581c19dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f582262b640 1 -- 172.21.15.60:0/1593538501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581c198180 con 0x7f581c10b920 2024-03-04T12:26:49.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f5821e2a640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f581c102550 0x7f581c19d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:49.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f582262b640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f581c10b920 0x7f581c197a00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f581400da40 tx=0x7f581400df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.145+0000 7f5812ffd640 1 -- 172.21.15.60:0/1593538501 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5814017070 con 0x7f581c10b920 2024-03-04T12:26:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.146+0000 7f582408e640 1 -- 172.21.15.60:0/1593538501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581c198470 con 0x7f581c10b920 2024-03-04T12:26:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.146+0000 7f582408e640 1 -- 172.21.15.60:0/1593538501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581c1a4710 con 0x7f581c10b920 2024-03-04T12:26:49.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.146+0000 7f5812ffd640 1 -- 172.21.15.60:0/1593538501 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5814003e50 con 0x7f581c10b920 2024-03-04T12:26:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.146+0000 7f5812ffd640 1 -- 172.21.15.60:0/1593538501 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5814020060 con 0x7f581c10b920 2024-03-04T12:26:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.148+0000 7f5826c5f740 1 Processor -- start 2024-03-04T12:26:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.149+0000 7f5812ffd640 1 -- 172.21.15.60:0/1593538501 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f5814010480 con 0x7f581c10b920 2024-03-04T12:26:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.149+0000 7f5826c5f740 1 -- start start 2024-03-04T12:26:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.149+0000 7f5826c5f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 0x5587f39aadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.149+0000 7f5826c5f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39ab2f0 0x5587f39b3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.149+0000 7f5826c5f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39b3ed0 0x5587f39b62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.149+0000 7f5826c5f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f3971dc0 con 0x5587f39aa9d0 2024-03-04T12:26:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.149+0000 7f5826c5f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f390dec0 con 0x5587f39ab2f0 2024-03-04T12:26:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.149+0000 7f5826c5f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f38b6d30 con 0x5587f39b3ed0 2024-03-04T12:26:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.149+0000 7f5821629640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 0x5587f39aadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.149+0000 7f5821629640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 0x5587f39aadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59090/0 (socket says 172.21.15.60:59090) 2024-03-04T12:26:49.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.149+0000 7f5821629640 1 -- 172.21.15.60:0/3036357944 learned_addr learned my addr 172.21.15.60:0/3036357944 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.149+0000 7f5821e2a640 1 --2- 172.21.15.60:0/3036357944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39ab2f0 0x5587f39b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.150+0000 7f5812ffd640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f58080a2010 0x7f58080a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.150+0000 7f5821629640 1 -- 172.21.15.60:0/3036357944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39b3ed0 msgr2=0x5587f39b62c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T12:26:49.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.150+0000 7f5821629640 1 --2- 172.21.15.60:0/3036357944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39b3ed0 0x5587f39b62c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.150+0000 7f5821629640 1 -- 172.21.15.60:0/3036357944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39ab2f0 msgr2=0x5587f39b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.165 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.150+0000 7f5821629640 1 --2- 172.21.15.60:0/3036357944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39ab2f0 0x5587f39b3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.150+0000 7f5821629640 1 -- 172.21.15.60:0/3036357944 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587f36223f0 con 0x5587f39aa9d0 2024-03-04T12:26:49.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.150+0000 7f582262b640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f58080a2010 0x7f58080a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.150+0000 7f5812ffd640 1 -- 172.21.15.60:0/1593538501 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f58140ce270 con 0x7f581c10b920 2024-03-04T12:26:49.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.150+0000 7f5821629640 1 --2- 172.21.15.60:0/3036357944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 0x5587f39aadb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f580c006010 tx=0x7f580c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb5e2849d7d9575e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.150+0000 7f5807fff640 1 -- 172.21.15.60:0/3036357944 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c02c920 con 0x5587f39aa9d0 2024-03-04T12:26:49.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.150+0000 7f5807fff640 1 -- 172.21.15.60:0/3036357944 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f580c02cac0 con 0x5587f39aa9d0 2024-03-04T12:26:49.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.150+0000 7f5807fff640 1 -- 172.21.15.60:0/3036357944 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c004f10 con 0x5587f39aa9d0 2024-03-04T12:26:49.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.151+0000 7f5826c5f740 1 -- 172.21.15.60:0/3036357944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 msgr2=0x5587f39aadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.166 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.151+0000 7f582262b640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f58080a2010 0x7f58080a44d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f58140cb500 tx=0x7f5814096380 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.151+0000 7f5826c5f740 1 --2- 172.21.15.60:0/3036357944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 0x5587f39aadb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f580c006010 tx=0x7f580c005d50 comp rx=0 tx=0).stop 2024-03-04T12:26:49.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.151+0000 7f5826c5f740 1 -- 172.21.15.60:0/3036357944 shutdown_connections 2024-03-04T12:26:49.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.151+0000 7f5826c5f740 1 --2- 172.21.15.60:0/3036357944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39b3ed0 0x5587f39b62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.151+0000 7f5826c5f740 1 --2- 172.21.15.60:0/3036357944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39ab2f0 0x5587f39b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.151+0000 7f5826c5f740 1 --2- 172.21.15.60:0/3036357944 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 0x5587f39aadb0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.167 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.151+0000 7f5826c5f740 1 -- 172.21.15.60:0/3036357944 >> 172.21.15.60:0/3036357944 conn(0x5587f39a5100 msgr2=0x5587f39a54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.151+0000 7f5826c5f740 1 -- 172.21.15.60:0/3036357944 shutdown_connections 2024-03-04T12:26:49.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.151+0000 7f5826c5f740 1 -- 172.21.15.60:0/3036357944 wait complete. 2024-03-04T12:26:49.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5826c5f740 1 -- start start 2024-03-04T12:26:49.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5826c5f740 1 Processor -- start 2024-03-04T12:26:49.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5826c5f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 0x5587f39b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5826c5f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39ab2f0 0x5587f39b1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5826c5f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39b3ed0 0x5587f39b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.168 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5826c5f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f38ac4a0 con 0x5587f39ab2f0 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5826c5f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f3909f60 con 0x5587f39b3ed0 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5826c5f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f39b6c60 con 0x5587f39aa9d0 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5826c5f740 10 client.0 _ll_register_callbacks cb 0x5587f39081c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5821629640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 0x5587f39b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5821e2a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39ab2f0 0x5587f39b1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5821629640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 0x5587f39b0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:44620/0 (socket says 172.21.15.60:44620) 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5821629640 1 -- 172.21.15.60:0/4288727032 learned_addr learned my addr 172.21.15.60:0/4288727032 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.152+0000 7f5821629640 1 --2- 172.21.15.60:0/4288727032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39b3ed0 0x5587f39b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f5821629640 1 -- 172.21.15.60:0/4288727032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39b3ed0 msgr2=0x5587f39b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f5821629640 1 --2- 172.21.15.60:0/4288727032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39b3ed0 0x5587f39b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f5821629640 1 -- 172.21.15.60:0/4288727032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39ab2f0 msgr2=0x5587f39b1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f5821629640 1 --2- 172.21.15.60:0/4288727032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39ab2f0 0x5587f39b1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f5821629640 1 -- 172.21.15.60:0/4288727032 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587f36ec7d0 con 0x5587f39aa9d0 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f5821629640 1 --2- 172.21.15.60:0/4288727032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 0x5587f39b0e20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f580c02ce50 tx=0x7f580c02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6aca3ff95d34e068 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.169 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f58057fa640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f58057fa640 1 -- 172.21.15.60:0/4288727032 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c0047c0 con 0x5587f39aa9d0 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f58057fa640 1 -- 172.21.15.60:0/4288727032 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f580c004230 con 0x5587f39aa9d0 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f5826c5f740 1 -- 172.21.15.60:0/4288727032 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f580c02cac0 con 0x5587f39aa9d0 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f58057fa640 1 -- 172.21.15.60:0/4288727032 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f580c05b050 con 0x5587f39aa9d0 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f5826c5f740 10 client.25663 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f5826c5f740 1 -- 172.21.15.60:0/4288727032 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5587f39c2d30 con 0x5587f39aa9d0 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.153+0000 7f58057fa640 1 -- 172.21.15.60:0/4288727032 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f580c005600 con 0x5587f39aa9d0 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.154+0000 7f58057fa640 1 -- 172.21.15.60:0/4288727032 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f580c003710 con 0x5587f39aa9d0 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.154+0000 7f5826c5f740 10 client.25663 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.154+0000 7f5826c5f740 10 client.25663 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.154+0000 7f5826c5f740 20 client.25663 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.154+0000 7f5826c5f740 1 --2- 172.21.15.60:0/4288727032 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5587f39c4080 0x5587f39c6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.154+0000 7f5826c5f740 4 client.25663 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.154+0000 7f5826c5f740 1 -- 172.21.15.60:0/4288727032 --> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f580c02cac0 con 0x5587f39c4080 2024-03-04T12:26:49.170 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.155+0000 7f5821e2a640 1 --2- 172.21.15.60:0/4288727032 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5587f39c4080 0x5587f39c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.155+0000 7f58057fa640 10 client.25663 ms_handle_connect on v2:172.21.15.86:6838/3347800322 2024-03-04T12:26:49.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.155+0000 7f5821e2a640 1 --2- 172.21.15.60:0/4288727032 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5587f39c4080 0x5587f39c6470 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3b2797455bdb0a91 server_cookie=b449b29153b0d8ae in_seq=0 out_seq=0 2024-03-04T12:26:49.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.155+0000 7f58057fa640 1 -- 172.21.15.60:0/4288727032 <== mds.1 v2:172.21.15.86:6838/3347800322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7f580c02cac0 con 0x5587f39c4080 2024-03-04T12:26:49.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.155+0000 7f58057fa640 10 client.25663 handle_command_reply: tid=0 2024-03-04T12:26:49.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f5826c5f740 1 client.25663 shutdown 2024-03-04T12:26:49.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f5826c5f740 1 -- 172.21.15.60:0/4288727032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 msgr2=0x5587f39b0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f5826c5f740 1 --2- 172.21.15.60:0/4288727032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 0x5587f39b0e20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f580c02ce50 tx=0x7f580c02e040 comp rx=0 tx=0).stop 2024-03-04T12:26:49.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f5826c5f740 1 -- 172.21.15.60:0/4288727032 shutdown_connections 2024-03-04T12:26:49.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f5826c5f740 1 --2- 172.21.15.60:0/4288727032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39b3ed0 0x5587f39b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f5826c5f740 1 --2- 172.21.15.60:0/4288727032 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5587f39c4080 0x5587f39c6470 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f5826c5f740 1 --2- 172.21.15.60:0/4288727032 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39ab2f0 0x5587f39b1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f58057fa640 0 client.25663 ms_handle_reset on v2:172.21.15.86:6838/3347800322 2024-03-04T12:26:49.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f5826c5f740 1 --2- 172.21.15.60:0/4288727032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 0x5587f39b0e20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f5826c5f740 1 -- 172.21.15.60:0/4288727032 >> 172.21.15.60:0/4288727032 conn(0x5587f39a5100 msgr2=0x5587f39a6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f5826c5f740 1 -- 172.21.15.60:0/4288727032 shutdown_connections 2024-03-04T12:26:49.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f5826c5f740 1 -- 172.21.15.60:0/4288727032 wait complete. 2024-03-04T12:26:49.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.156+0000 7f5826c5f740 20 client.25663 trim_cache size 0 max 16384 2024-03-04T12:26:49.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:49 smithi060 ceph-mon[32745]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-04T12:26:49.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:49 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1685968693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:26:49.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5826c5f740 1 Processor -- start 2024-03-04T12:26:49.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5826c5f740 1 -- start start 2024-03-04T12:26:49.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5826c5f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 0x5587f39baf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5826c5f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39ab2f0 0x5587f39bb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5826c5f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39b3ed0 0x5587f39ae080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.193 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5826c5f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580000fbb0 con 0x5587f39ab2f0 2024-03-04T12:26:49.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5826c5f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5800011530 con 0x5587f39b3ed0 2024-03-04T12:26:49.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5826c5f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5800010210 con 0x5587f39aa9d0 2024-03-04T12:26:49.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5821e2a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39ab2f0 0x5587f39bb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5821e2a640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39ab2f0 0x5587f39bb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59118/0 (socket says 172.21.15.60:59118) 2024-03-04T12:26:49.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5821e2a640 1 -- 172.21.15.60:0/988540451 learned_addr learned my addr 172.21.15.60:0/988540451 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.194 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5821629640 1 --2- 172.21.15.60:0/988540451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 0x5587f39baf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5821629640 1 --2- 172.21.15.60:0/988540451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39b3ed0 0x5587f39ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5821e2a640 1 -- 172.21.15.60:0/988540451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 msgr2=0x5587f39baf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5821e2a640 1 --2- 172.21.15.60:0/988540451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 0x5587f39baf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5821e2a640 1 -- 172.21.15.60:0/988540451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39b3ed0 msgr2=0x5587f39ae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.177+0000 7f5821e2a640 1 --2- 172.21.15.60:0/988540451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39b3ed0 0x5587f39ae080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.178+0000 7f5821e2a640 1 -- 172.21.15.60:0/988540451 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587f39b62c0 con 0x5587f39ab2f0 2024-03-04T12:26:49.195 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.178+0000 7f5821629640 1 --2- 172.21.15.60:0/988540451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 0x5587f39baf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.178+0000 7f5821e2a640 1 --2- 172.21.15.60:0/988540451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39ab2f0 0x5587f39bb460 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f581c198b40 tx=0x7f581800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc5b0c1d098725c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.178+0000 7f58077fe640 1 -- 172.21.15.60:0/988540451 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5818005220 con 0x5587f39ab2f0 2024-03-04T12:26:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.178+0000 7f58077fe640 1 -- 172.21.15.60:0/988540451 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5818023030 con 0x5587f39ab2f0 2024-03-04T12:26:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.178+0000 7f58077fe640 1 -- 172.21.15.60:0/988540451 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581801f350 con 0x5587f39ab2f0 2024-03-04T12:26:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.179+0000 7f5826c5f740 1 -- 172.21.15.60:0/988540451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39ab2f0 msgr2=0x5587f39bb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.179+0000 7f5826c5f740 1 --2- 172.21.15.60:0/988540451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39ab2f0 0x5587f39bb460 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f581c198b40 tx=0x7f581800b390 comp rx=0 tx=0).stop 2024-03-04T12:26:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.179+0000 7f5826c5f740 1 -- 172.21.15.60:0/988540451 shutdown_connections 2024-03-04T12:26:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.179+0000 7f5826c5f740 1 --2- 172.21.15.60:0/988540451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39b3ed0 0x5587f39ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.196 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.179+0000 7f5826c5f740 1 --2- 172.21.15.60:0/988540451 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39ab2f0 0x5587f39bb460 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.179+0000 7f5826c5f740 1 --2- 172.21.15.60:0/988540451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39aa9d0 0x5587f39baf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.179+0000 7f5826c5f740 1 -- 172.21.15.60:0/988540451 >> 172.21.15.60:0/988540451 conn(0x5587f39a5100 msgr2=0x5587f39caa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.179+0000 7f5826c5f740 1 -- 172.21.15.60:0/988540451 shutdown_connections 2024-03-04T12:26:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.179+0000 7f5826c5f740 1 -- 172.21.15.60:0/988540451 wait complete. 2024-03-04T12:26:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.180+0000 7f5826c5f740 1 -- start start 2024-03-04T12:26:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.180+0000 7f5826c5f740 1 Processor -- start 2024-03-04T12:26:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.180+0000 7f5826c5f740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 0x5587f39b9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.180+0000 7f5826c5f740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39b3ed0 0x5587f39b9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.180+0000 7f5826c5f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39ba410 0x5587f3a1a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.197 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5826c5f740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5800010ed0 con 0x5587f39aa9d0 2024-03-04T12:26:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5826c5f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5800010870 con 0x5587f39ba410 2024-03-04T12:26:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5826c5f740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580000de50 con 0x5587f39b3ed0 2024-03-04T12:26:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5821629640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 0x5587f39b9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5821629640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 0x5587f39b9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59128/0 (socket says 172.21.15.60:59128) 2024-03-04T12:26:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5821629640 1 -- 172.21.15.60:0/479910519 learned_addr learned my addr 172.21.15.60:0/479910519 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5826c5f740 10 client.0 _ll_register_callbacks cb 0x7f580c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5821629640 1 -- 172.21.15.60:0/479910519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39b3ed0 msgr2=0x5587f39b9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.198 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5821e2a640 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39b3ed0 0x5587f39b9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5821629640 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39b3ed0 0x5587f39b9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5821629640 1 -- 172.21.15.60:0/479910519 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39ba410 msgr2=0x5587f3a1a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5821629640 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39ba410 0x5587f3a1a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5821629640 1 -- 172.21.15.60:0/479910519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587f39aca00 con 0x5587f39aa9d0 2024-03-04T12:26:49.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5821e2a640 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39b3ed0 0x5587f39b9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:49.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f5821629640 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 0x5587f39b9990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f580c059040 tx=0x7f580c02cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d160d011107945d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f58057fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:49.199 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f58057fa640 1 -- 172.21.15.60:0/479910519 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580c064060 con 0x5587f39aa9d0 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.181+0000 7f58057fa640 1 -- 172.21.15.60:0/479910519 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f580c005320 con 0x5587f39aa9d0 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.182+0000 7f5826c5f740 1 -- 172.21.15.60:0/479910519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5818023030 con 0x5587f39aa9d0 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.182+0000 7f58057fa640 1 -- 172.21.15.60:0/479910519 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f580c005520 con 0x5587f39aa9d0 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.182+0000 7f5826c5f740 10 client.16137 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.182+0000 7f5826c5f740 1 -- 172.21.15.60:0/479910519 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5587f3a1b7c0 con 0x5587f39aa9d0 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.182+0000 7f58057fa640 1 -- 172.21.15.60:0/479910519 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5818023030 con 0x5587f39aa9d0 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.182+0000 7f58057fa640 1 -- 172.21.15.60:0/479910519 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f580c03e400 con 0x5587f39aa9d0 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.182+0000 7f5826c5f740 10 client.16137 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.182+0000 7f5826c5f740 10 client.16137 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.183+0000 7f5826c5f740 20 client.16137 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.183+0000 7f5826c5f740 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5587f3a1bdb0 0x5587f3a1e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.183+0000 7f5826c5f740 4 client.16137 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.183+0000 7f5826c5f740 1 -- 172.21.15.60:0/479910519 --> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5587f391b400 con 0x5587f3a1bdb0 2024-03-04T12:26:49.200 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.183+0000 7f5821e2a640 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5587f3a1bdb0 0x5587f3a1e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.183+0000 7f58057fa640 10 client.16137 ms_handle_connect on v2:172.21.15.86:6838/3347800322 2024-03-04T12:26:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.183+0000 7f5821e2a640 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5587f3a1bdb0 0x5587f3a1e1d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7692471520aa0cf0 server_cookie=978d4d781d7f33af in_seq=0 out_seq=0 2024-03-04T12:26:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.184+0000 7f58057fa640 1 -- 172.21.15.60:0/479910519 <== mds.1 v2:172.21.15.86:6838/3347800322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5587f391b400 con 0x5587f3a1bdb0 2024-03-04T12:26:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.184+0000 7f58057fa640 10 client.16137 handle_command_reply: tid=0 2024-03-04T12:26:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.184+0000 7f5826c5f740 1 client.16137 shutdown 2024-03-04T12:26:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.184+0000 7f5826c5f740 1 -- 172.21.15.60:0/479910519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 msgr2=0x5587f39b9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.184+0000 7f5826c5f740 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 0x5587f39b9990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f580c059040 tx=0x7f580c02cc90 comp rx=0 tx=0).stop 2024-03-04T12:26:49.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.184+0000 7f5826c5f740 1 -- 172.21.15.60:0/479910519 shutdown_connections 2024-03-04T12:26:49.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.184+0000 7f5826c5f740 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5587f39ba410 0x5587f3a1a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.184+0000 7f5826c5f740 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5587f39b3ed0 0x5587f39b9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.184+0000 7f5826c5f740 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x5587f3a1bdb0 0x5587f3a1e1d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.184+0000 7f5826c5f740 1 --2- 172.21.15.60:0/479910519 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5587f39aa9d0 0x5587f39b9990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.184+0000 7f58057fa640 0 client.16137 ms_handle_reset on v2:172.21.15.86:6838/3347800322 2024-03-04T12:26:49.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.184+0000 7f5826c5f740 1 -- 172.21.15.60:0/479910519 >> 172.21.15.60:0/479910519 conn(0x5587f39a5100 msgr2=0x5587f39af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.185+0000 7f5826c5f740 1 -- 172.21.15.60:0/479910519 shutdown_connections 2024-03-04T12:26:49.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.185+0000 7f5826c5f740 1 -- 172.21.15.60:0/479910519 wait complete. 2024-03-04T12:26:49.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.185+0000 7f5826c5f740 20 client.16137 trim_cache size 0 max 16384 2024-03-04T12:26:49.203 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:26:49.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.185+0000 7f582408e640 1 -- 172.21.15.60:0/1593538501 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f58080a2010 msgr2=0x7f58080a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.185+0000 7f582408e640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f58080a2010 0x7f58080a44d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f58140cb500 tx=0x7f5814096380 comp rx=0 tx=0).stop 2024-03-04T12:26:49.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.185+0000 7f582408e640 1 -- 172.21.15.60:0/1593538501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f581c10b920 msgr2=0x7f581c197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.185+0000 7f582408e640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f581c10b920 0x7f581c197a00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f581400da40 tx=0x7f581400df00 comp rx=0 tx=0).stop 2024-03-04T12:26:49.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.185+0000 7f582408e640 1 -- 172.21.15.60:0/1593538501 shutdown_connections 2024-03-04T12:26:49.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.185+0000 7f582408e640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f58080a2010 0x7f58080a44d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.185+0000 7f582408e640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f581c10b920 0x7f581c197a00 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.185+0000 7f582408e640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f581c102e70 0x7f581c19dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.185+0000 7f582408e640 1 --2- 172.21.15.60:0/1593538501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f581c102550 0x7f581c19d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.186+0000 7f582408e640 1 -- 172.21.15.60:0/1593538501 >> 172.21.15.60:0/1593538501 conn(0x7f581c0773e0 msgr2=0x7f581c0fd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.186+0000 7f582408e640 1 -- 172.21.15.60:0/1593538501 shutdown_connections 2024-03-04T12:26:49.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.186+0000 7f582408e640 1 -- 172.21.15.60:0/1593538501 wait complete. 2024-03-04T12:26:49.207 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:26:49.207 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T12:26:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.307+0000 7fc71b6e8640 1 -- 172.21.15.60:0/3495684372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc71410dc80 msgr2=0x7fc714110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.307+0000 7fc71b6e8640 1 --2- 172.21.15.60:0/3495684372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc71410dc80 0x7fc714110070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009ca0 tx=0x7fc70c02c180 comp rx=0 tx=0).stop 2024-03-04T12:26:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.307+0000 7fc71b6e8640 1 -- 172.21.15.60:0/3495684372 shutdown_connections 2024-03-04T12:26:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.307+0000 7fc71b6e8640 1 --2- 172.21.15.60:0/3495684372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc71410dc80 0x7fc714110070 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.307+0000 7fc71b6e8640 1 --2- 172.21.15.60:0/3495684372 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc7141009f0 0x7fc71410d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.307+0000 7fc71b6e8640 1 --2- 172.21.15.60:0/3495684372 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7141000d0 0x7fc7141004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.307+0000 7fc71b6e8640 1 -- 172.21.15.60:0/3495684372 >> 172.21.15.60:0/3495684372 conn(0x7fc7140773e0 msgr2=0x7fc7140777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.307+0000 7fc71b6e8640 1 -- 172.21.15.60:0/3495684372 shutdown_connections 2024-03-04T12:26:49.316 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.307+0000 7fc71b6e8640 1 -- 172.21.15.60:0/3495684372 wait complete. 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc71b6e8640 1 Processor -- start 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc71b6e8640 1 -- start start 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc71b6e8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7141000d0 0x7fc714199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc71b6e8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7141009f0 0x7fc71419a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc71b6e8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71410dc80 0x7fc71419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc71b6e8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7141138a0 con 0x7fc71410dc80 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc71b6e8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc714113720 con 0x7fc7141000d0 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc71b6e8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc714113a20 con 0x7fc7141009f0 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc719c85640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71410dc80 0x7fc71419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc719c85640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71410dc80 0x7fc71419e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59142/0 (socket says 172.21.15.60:59142) 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc719c85640 1 -- 172.21.15.60:0/593977540 learned_addr learned my addr 172.21.15.60:0/593977540 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc719484640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7141000d0 0x7fc714199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.308+0000 7fc718c83640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7141009f0 0x7fc71419a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.309+0000 7fc719c85640 1 -- 172.21.15.60:0/593977540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7141009f0 msgr2=0x7fc71419a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.309+0000 7fc719c85640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7141009f0 0x7fc71419a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.309+0000 7fc719c85640 1 -- 172.21.15.60:0/593977540 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7141000d0 msgr2=0x7fc714199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.309+0000 7fc719c85640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7141000d0 0x7fc714199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.309+0000 7fc719c85640 1 -- 172.21.15.60:0/593977540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71419edb0 con 0x7fc71410dc80 2024-03-04T12:26:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.309+0000 7fc719c85640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71410dc80 0x7fc71419e680 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc70c005cc0 tx=0x7fc70c035e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.309+0000 7fc70a7fc640 1 -- 172.21.15.60:0/593977540 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c039040 con 0x7fc71410dc80 2024-03-04T12:26:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.309+0000 7fc71b6e8640 1 -- 172.21.15.60:0/593977540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71419f040 con 0x7fc71410dc80 2024-03-04T12:26:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.309+0000 7fc70a7fc640 1 -- 172.21.15.60:0/593977540 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc70c044070 con 0x7fc71410dc80 2024-03-04T12:26:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.309+0000 7fc71b6e8640 1 -- 172.21.15.60:0/593977540 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7141a6930 con 0x7fc71410dc80 2024-03-04T12:26:49.318 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.309+0000 7fc70a7fc640 1 -- 172.21.15.60:0/593977540 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c032960 con 0x7fc71410dc80 2024-03-04T12:26:49.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.312+0000 7fc70a7fc640 1 -- 172.21.15.60:0/593977540 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7fc70c032b00 con 0x7fc71410dc80 2024-03-04T12:26:49.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.313+0000 7fc71e3a1740 1 Processor -- start 2024-03-04T12:26:49.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.313+0000 7fc71e3a1740 1 -- start start 2024-03-04T12:26:49.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.313+0000 7fc71e3a1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 0x5624ec742db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.319 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.313+0000 7fc70a7fc640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc7000a2010 0x7fc7000a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.313+0000 7fc71e3a1740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7432f0 0x5624ec74b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.313+0000 7fc71e3a1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec74e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.313+0000 7fc71e3a1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ec709dc0 con 0x5624ec74bed0 2024-03-04T12:26:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.313+0000 7fc71e3a1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ec6a5ec0 con 0x5624ec7429d0 2024-03-04T12:26:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.313+0000 7fc71e3a1740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ec64ed30 con 0x5624ec7432f0 2024-03-04T12:26:49.321 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.313+0000 7fc70a7fc640 1 -- 172.21.15.60:0/593977540 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fc70c0f0af0 con 0x7fc71410dc80 2024-03-04T12:26:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.313+0000 7fc719484640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc7000a2010 0x7fc7000a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.313+0000 7fc719c85640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec74e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc718c83640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 0x5624ec742db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc718c83640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 0x5624ec742db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:40068/0 (socket says 172.21.15.60:40068) 2024-03-04T12:26:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc718c83640 1 -- 172.21.15.60:0/3392266088 learned_addr learned my addr 172.21.15.60:0/3392266088 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc718c83640 1 --2- 172.21.15.60:0/3392266088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7432f0 0x5624ec74b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc718c83640 1 -- 172.21.15.60:0/3392266088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7432f0 msgr2=0x5624ec74b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc718c83640 1 --2- 172.21.15.60:0/3392266088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7432f0 0x5624ec74b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.322 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc718c83640 1 -- 172.21.15.60:0/3392266088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 msgr2=0x5624ec74e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc718c83640 1 --2- 172.21.15.60:0/3392266088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec74e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc718c83640 1 -- 172.21.15.60:0/3392266088 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624ec3ba3f0 con 0x5624ec7429d0 2024-03-04T12:26:49.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc718c83640 1 --2- 172.21.15.60:0/3392266088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 0x5624ec742db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc704012770 tx=0x7fc704009660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93cdd1288b88925c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc719484640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc7000a2010 0x7fc7000a44d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc710009bd0 tx=0x7fc710005f90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc6ff7fe640 1 -- 172.21.15.60:0/3392266088 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704004020 con 0x5624ec7429d0 2024-03-04T12:26:49.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc6ff7fe640 1 -- 172.21.15.60:0/3392266088 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7040041c0 con 0x5624ec7429d0 2024-03-04T12:26:49.323 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.314+0000 7fc6ff7fe640 1 -- 172.21.15.60:0/3392266088 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70401f760 con 0x5624ec7429d0 2024-03-04T12:26:49.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.315+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3392266088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 msgr2=0x5624ec742db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.315+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/3392266088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 0x5624ec742db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc704012770 tx=0x7fc704009660 comp rx=0 tx=0).stop 2024-03-04T12:26:49.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.315+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3392266088 shutdown_connections 2024-03-04T12:26:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.315+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/3392266088 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec74e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.315+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/3392266088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7432f0 0x5624ec74b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.315+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/3392266088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 0x5624ec742db0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.315+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3392266088 >> 172.21.15.60:0/3392266088 conn(0x5624ec73d100 msgr2=0x5624ec73d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.315+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3392266088 shutdown_connections 2024-03-04T12:26:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.315+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3392266088 wait complete. 2024-03-04T12:26:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.316+0000 7fc71e3a1740 1 -- start start 2024-03-04T12:26:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.316+0000 7fc71e3a1740 1 Processor -- start 2024-03-04T12:26:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.316+0000 7fc71e3a1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 0x5624ec748f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.316+0000 7fc71e3a1740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7432f0 0x5624ec749460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.316+0000 7fc71e3a1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec7499a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc719c85640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec7499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc718c83640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7432f0 0x5624ec749460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc719c85640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec7499a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59154/0 (socket says 172.21.15.60:59154) 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc719c85640 1 -- 172.21.15.60:0/4172719980 learned_addr learned my addr 172.21.15.60:0/4172719980 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc719484640 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 0x5624ec748f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc71e3a1740 1 -- 172.21.15.60:0/4172719980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ec6444a0 con 0x5624ec74bed0 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc71e3a1740 1 -- 172.21.15.60:0/4172719980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ec6a1f60 con 0x5624ec7429d0 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc71e3a1740 1 -- 172.21.15.60:0/4172719980 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ec74ecc0 con 0x5624ec7432f0 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc71e3a1740 10 client.0 _ll_register_callbacks cb 0x5624ec6a01c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc719c85640 1 -- 172.21.15.60:0/4172719980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7432f0 msgr2=0x5624ec749460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc719c85640 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7432f0 0x5624ec749460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc719c85640 1 -- 172.21.15.60:0/4172719980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 msgr2=0x5624ec748f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc719c85640 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 0x5624ec748f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc719c85640 1 -- 172.21.15.60:0/4172719980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624ec4847d0 con 0x5624ec74bed0 2024-03-04T12:26:49.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc719484640 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 0x5624ec748f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc719c85640 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec7499a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0bd020 tx=0x7fc70c0bcc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff2c86c58bfd410 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc6fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.317+0000 7fc6fcff9640 1 -- 172.21.15.60:0/4172719980 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c0de070 con 0x5624ec74bed0 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc6fcff9640 1 -- 172.21.15.60:0/4172719980 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fc70c0bb390 con 0x5624ec74bed0 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc71e3a1740 1 -- 172.21.15.60:0/4172719980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc7040041c0 con 0x5624ec74bed0 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc6fcff9640 1 -- 172.21.15.60:0/4172719980 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc70c0bd090 con 0x5624ec74bed0 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc71e3a1740 10 client.16149 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc71e3a1740 1 -- 172.21.15.60:0/4172719980 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5624ec75ade0 con 0x5624ec74bed0 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc6fcff9640 1 -- 172.21.15.60:0/4172719980 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc70c0c11e0 con 0x5624ec74bed0 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc6fcff9640 1 -- 172.21.15.60:0/4172719980 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc70c0e0070 con 0x5624ec74bed0 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc71e3a1740 10 client.16149 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc71e3a1740 10 client.16149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:49.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc71e3a1740 20 client.16149 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5624ec75c2e0 0x5624ec75e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc71e3a1740 4 client.16149 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.318+0000 7fc71e3a1740 1 -- 172.21.15.60:0/4172719980 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7040041c0 con 0x5624ec75c2e0 2024-03-04T12:26:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.319+0000 7fc719484640 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5624ec75c2e0 0x5624ec75e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.319+0000 7fc6fcff9640 10 client.16149 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.319+0000 7fc719484640 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5624ec75c2e0 0x5624ec75e6f0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=196877afcd75dd6 server_cookie=1be3e1612c3468f2 in_seq=0 out_seq=0 2024-03-04T12:26:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc6fcff9640 1 -- 172.21.15.60:0/4172719980 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7fc7040041c0 con 0x5624ec75c2e0 2024-03-04T12:26:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc6fcff9640 10 client.16149 handle_command_reply: tid=0 2024-03-04T12:26:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc71e3a1740 1 client.16149 shutdown 2024-03-04T12:26:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc71e3a1740 1 -- 172.21.15.60:0/4172719980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 msgr2=0x5624ec7499a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.336 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec7499a0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0bd020 tx=0x7fc70c0bcc90 comp rx=0 tx=0).stop 2024-03-04T12:26:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc71e3a1740 1 -- 172.21.15.60:0/4172719980 shutdown_connections 2024-03-04T12:26:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5624ec75c2e0 0x5624ec75e6f0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec7499a0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc6fcff9640 0 client.16149 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7432f0 0x5624ec749460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/4172719980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7429d0 0x5624ec748f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc71e3a1740 1 -- 172.21.15.60:0/4172719980 >> 172.21.15.60:0/4172719980 conn(0x5624ec73d100 msgr2=0x5624ec73ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc71e3a1740 1 -- 172.21.15.60:0/4172719980 shutdown_connections 2024-03-04T12:26:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc71e3a1740 1 -- 172.21.15.60:0/4172719980 wait complete. 2024-03-04T12:26:49.337 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.320+0000 7fc71e3a1740 20 client.16149 trim_cache size 0 max 16384 2024-03-04T12:26:49.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.343+0000 7fc71e3a1740 1 Processor -- start 2024-03-04T12:26:49.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.343+0000 7fc71e3a1740 1 -- start start 2024-03-04T12:26:49.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc71e3a1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec7429d0 0x5624ec74b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc71e3a1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7432f0 0x5624ec752ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.351 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc719484640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec7429d0 0x5624ec74b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc719484640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec7429d0 0x5624ec74b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59168/0 (socket says 172.21.15.60:59168) 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc719484640 1 -- 172.21.15.60:0/1252998237 learned_addr learned my addr 172.21.15.60:0/1252998237 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/1252998237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec74bed0 0x5624ec753430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc71e3a1740 1 -- 172.21.15.60:0/1252998237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7141b06b0 con 0x5624ec7429d0 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc71e3a1740 1 -- 172.21.15.60:0/1252998237 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7141b2030 con 0x5624ec7432f0 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc71e3a1740 1 -- 172.21.15.60:0/1252998237 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7141b0d10 con 0x5624ec74bed0 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc718c83640 1 --2- 172.21.15.60:0/1252998237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7432f0 0x5624ec752ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc719c85640 1 --2- 172.21.15.60:0/1252998237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec74bed0 0x5624ec753430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc719484640 1 -- 172.21.15.60:0/1252998237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec74bed0 msgr2=0x5624ec753430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc719484640 1 --2- 172.21.15.60:0/1252998237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec74bed0 0x5624ec753430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc719484640 1 -- 172.21.15.60:0/1252998237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7432f0 msgr2=0x5624ec752ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc719484640 1 --2- 172.21.15.60:0/1252998237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7432f0 0x5624ec752ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc719484640 1 -- 172.21.15.60:0/1252998237 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624ec73cc30 con 0x5624ec7429d0 2024-03-04T12:26:49.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.344+0000 7fc719c85640 1 --2- 172.21.15.60:0/1252998237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec74bed0 0x5624ec753430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:26:49.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.345+0000 7fc719484640 1 --2- 172.21.15.60:0/1252998237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec7429d0 0x5624ec74b650 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc710009730 tx=0x7fc710014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3706dd506ec6e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.345+0000 7fc6feffd640 1 -- 172.21.15.60:0/1252998237 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7100043c0 con 0x5624ec7429d0 2024-03-04T12:26:49.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.345+0000 7fc6feffd640 1 -- 172.21.15.60:0/1252998237 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc71002c030 con 0x5624ec7429d0 2024-03-04T12:26:49.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.345+0000 7fc6feffd640 1 -- 172.21.15.60:0/1252998237 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71001d410 con 0x5624ec7429d0 2024-03-04T12:26:49.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:49 smithi063 ceph-mon[36717]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-04T12:26:49.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:49 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1685968693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:26:49.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.346+0000 7fc71e3a1740 1 -- 172.21.15.60:0/1252998237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec7429d0 msgr2=0x5624ec74b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.346+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/1252998237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec7429d0 0x5624ec74b650 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc710009730 tx=0x7fc710014000 comp rx=0 tx=0).stop 2024-03-04T12:26:49.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.346+0000 7fc71e3a1740 1 -- 172.21.15.60:0/1252998237 shutdown_connections 2024-03-04T12:26:49.357 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.346+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/1252998237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec74bed0 0x5624ec753430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.346+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/1252998237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7432f0 0x5624ec752ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.346+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/1252998237 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec7429d0 0x5624ec74b650 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.346+0000 7fc71e3a1740 1 -- 172.21.15.60:0/1252998237 >> 172.21.15.60:0/1252998237 conn(0x5624ec73d100 msgr2=0x5624ec763020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.346+0000 7fc71e3a1740 1 -- 172.21.15.60:0/1252998237 shutdown_connections 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.346+0000 7fc71e3a1740 1 -- 172.21.15.60:0/1252998237 wait complete. 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.347+0000 7fc71e3a1740 1 -- start start 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.347+0000 7fc71e3a1740 1 Processor -- start 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.347+0000 7fc71e3a1740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7429d0 0x5624ec7532d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.347+0000 7fc71e3a1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7432f0 0x5624ec751710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.347+0000 7fc71e3a1740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec751cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc71e3a1740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7141b19d0 con 0x5624ec74bed0 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc71e3a1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7141b1370 con 0x5624ec7432f0 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc719484640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7429d0 0x5624ec7532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc71e3a1740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7141ae950 con 0x5624ec7429d0 2024-03-04T12:26:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc719c85640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec751cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc719484640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7429d0 0x5624ec7532d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:44664/0 (socket says 172.21.15.60:44664) 2024-03-04T12:26:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc719484640 1 -- 172.21.15.60:0/3458536954 learned_addr learned my addr 172.21.15.60:0/3458536954 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc718c83640 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7432f0 0x5624ec751710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc71e3a1740 10 client.0 _ll_register_callbacks cb 0x7fc7040099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc718c83640 1 -- 172.21.15.60:0/3458536954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7429d0 msgr2=0x5624ec7532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc718c83640 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7429d0 0x5624ec7532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc718c83640 1 -- 172.21.15.60:0/3458536954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 msgr2=0x5624ec751cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc718c83640 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec751cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc718c83640 1 -- 172.21.15.60:0/3458536954 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624ec752500 con 0x5624ec7432f0 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc719c85640 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec751cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc718c83640 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7432f0 0x5624ec751710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc704006010 tx=0x7fc704003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4dd33c0396fe1fb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc6fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc6fcff9640 1 -- 172.21.15.60:0/3458536954 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7040055f0 con 0x5624ec7432f0 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3458536954 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc71002c030 con 0x5624ec7432f0 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.348+0000 7fc6fcff9640 1 -- 172.21.15.60:0/3458536954 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fc70401f470 con 0x5624ec7432f0 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.349+0000 7fc6fcff9640 1 -- 172.21.15.60:0/3458536954 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc70402f050 con 0x5624ec7432f0 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.349+0000 7fc71e3a1740 10 client.25613 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.349+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3458536954 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5624ec7b37e0 con 0x5624ec7432f0 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.349+0000 7fc6fcff9640 1 -- 172.21.15.60:0/3458536954 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc71002c030 con 0x5624ec7432f0 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.349+0000 7fc6fcff9640 1 -- 172.21.15.60:0/3458536954 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc704028670 con 0x5624ec7432f0 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.349+0000 7fc71e3a1740 10 client.25613 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.349+0000 7fc71e3a1740 10 client.25613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:49.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.349+0000 7fc71e3a1740 20 client.25613 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.349+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5624ec75c2e0 0x5624ec7b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.350+0000 7fc71e3a1740 4 client.25613 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.350+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3458536954 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5624ec743ea0 con 0x5624ec75c2e0 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.350+0000 7fc719484640 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5624ec75c2e0 0x5624ec7b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.350+0000 7fc6fcff9640 10 client.25613 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.350+0000 7fc719484640 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5624ec75c2e0 0x5624ec7b5c80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8661166b868bd2f5 server_cookie=f1b8d289f32d43f9 in_seq=0 out_seq=0 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.350+0000 7fc6fcff9640 1 -- 172.21.15.60:0/3458536954 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5624ec743ea0 con 0x5624ec75c2e0 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.350+0000 7fc6fcff9640 10 client.25613 handle_command_reply: tid=0 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.350+0000 7fc71e3a1740 1 client.25613 shutdown 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.350+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3458536954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7432f0 msgr2=0x5624ec751710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.350+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7432f0 0x5624ec751710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc704006010 tx=0x7fc704003820 comp rx=0 tx=0).stop 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3458536954 shutdown_connections 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5624ec75c2e0 0x5624ec7b5c80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5624ec74bed0 0x5624ec751cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5624ec7432f0 0x5624ec751710 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71e3a1740 1 --2- 172.21.15.60:0/3458536954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5624ec7429d0 0x5624ec7532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3458536954 >> 172.21.15.60:0/3458536954 conn(0x5624ec73d100 msgr2=0x5624ec747720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc6fcff9640 0 client.25613 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:49.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3458536954 shutdown_connections 2024-03-04T12:26:49.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71e3a1740 1 -- 172.21.15.60:0/3458536954 wait complete. 2024-03-04T12:26:49.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71e3a1740 20 client.25613 trim_cache size 0 max 16384 2024-03-04T12:26:49.362 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:26:49.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71b6e8640 1 -- 172.21.15.60:0/593977540 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc7000a2010 msgr2=0x7fc7000a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71b6e8640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc7000a2010 0x7fc7000a44d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc710009bd0 tx=0x7fc710005f90 comp rx=0 tx=0).stop 2024-03-04T12:26:49.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71b6e8640 1 -- 172.21.15.60:0/593977540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71410dc80 msgr2=0x7fc71419e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:49.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71b6e8640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71410dc80 0x7fc71419e680 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc70c005cc0 tx=0x7fc70c035e00 comp rx=0 tx=0).stop 2024-03-04T12:26:49.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71b6e8640 1 -- 172.21.15.60:0/593977540 shutdown_connections 2024-03-04T12:26:49.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71b6e8640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc7000a2010 0x7fc7000a44d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71b6e8640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc71410dc80 0x7fc71419e680 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71b6e8640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7141009f0 0x7fc71419a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71b6e8640 1 --2- 172.21.15.60:0/593977540 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc7141000d0 0x7fc714199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:49.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.351+0000 7fc71b6e8640 1 -- 172.21.15.60:0/593977540 >> 172.21.15.60:0/593977540 conn(0x7fc7140773e0 msgr2=0x7fc71410f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:49.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.352+0000 7fc71b6e8640 1 -- 172.21.15.60:0/593977540 shutdown_connections 2024-03-04T12:26:49.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:49.352+0000 7fc71b6e8640 1 -- 172.21.15.60:0/593977540 wait complete. 2024-03-04T12:26:49.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:49 smithi086 ceph-mon[36851]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-04T12:26:49.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:49 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1685968693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:26:50.369 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T12:26:50.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:26:50.370 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T12:26:50.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.533+0000 7f1caa1cd640 1 -- 172.21.15.60:0/519758906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ca4100df0 msgr2=0x7f1ca4104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.540 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.533+0000 7f1caa1cd640 1 --2- 172.21.15.60:0/519758906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ca4100df0 0x7f1ca4104e30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009900 tx=0x7f1c9802bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:50.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.533+0000 7f1caa1cd640 1 -- 172.21.15.60:0/519758906 shutdown_connections 2024-03-04T12:26:50.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.533+0000 7f1caa1cd640 1 --2- 172.21.15.60:0/519758906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ca4105510 0x7f1ca410d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.533+0000 7f1caa1cd640 1 --2- 172.21.15.60:0/519758906 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ca4100df0 0x7f1ca4104e30 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.533+0000 7f1caa1cd640 1 --2- 172.21.15.60:0/519758906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ca4100440 0x7f1ca4100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.533+0000 7f1caa1cd640 1 -- 172.21.15.60:0/519758906 >> 172.21.15.60:0/519758906 conn(0x7f1ca40773e0 msgr2=0x7f1ca40777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:50.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.533+0000 7f1caa1cd640 1 -- 172.21.15.60:0/519758906 shutdown_connections 2024-03-04T12:26:50.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.533+0000 7f1caa1cd640 1 -- 172.21.15.60:0/519758906 wait complete. 2024-03-04T12:26:50.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.534+0000 7f1caa1cd640 1 Processor -- start 2024-03-04T12:26:50.541 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.534+0000 7f1caa1cd640 1 -- start start 2024-03-04T12:26:50.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.534+0000 7f1caa1cd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ca4100440 0x7f1ca419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.534+0000 7f1caa1cd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ca4105510 0x7f1ca419cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.534+0000 7f1caa1cd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ca41968c0 0x7f1ca4196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.543 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.534+0000 7f1caa1cd640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca4110580 con 0x7f1ca41968c0 2024-03-04T12:26:50.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.534+0000 7f1caa1cd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca4110400 con 0x7f1ca4100440 2024-03-04T12:26:50.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.534+0000 7f1caa1cd640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca4110700 con 0x7f1ca4105510 2024-03-04T12:26:50.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.535+0000 7f1ca3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ca41968c0 0x7f1ca4196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.535+0000 7f1ca37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ca4100440 0x7f1ca419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.535+0000 7f1ca37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ca4100440 0x7f1ca419c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:40114/0 (socket says 172.21.15.60:40114) 2024-03-04T12:26:50.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.535+0000 7f1ca37fe640 1 -- 172.21.15.60:0/3345615182 learned_addr learned my addr 172.21.15.60:0/3345615182 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:50.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.535+0000 7f1ca2ffd640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ca4105510 0x7f1ca419cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.535+0000 7f1ca37fe640 1 -- 172.21.15.60:0/3345615182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ca4105510 msgr2=0x7f1ca419cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.544 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.535+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ca4105510 0x7f1ca419cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.535+0000 7f1ca37fe640 1 -- 172.21.15.60:0/3345615182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ca41968c0 msgr2=0x7f1ca4196d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.535+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ca41968c0 0x7f1ca4196d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.535+0000 7f1ca37fe640 1 -- 172.21.15.60:0/3345615182 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca41973e0 con 0x7f1ca4100440 2024-03-04T12:26:50.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.535+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ca4100440 0x7f1ca419c710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400d950 tx=0x7f1c9400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:50.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.535+0000 7f1ca0ff9640 1 -- 172.21.15.60:0/3345615182 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94017030 con 0x7f1ca4100440 2024-03-04T12:26:50.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.536+0000 7f1ca0ff9640 1 -- 172.21.15.60:0/3345615182 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c94004020 con 0x7f1ca4100440 2024-03-04T12:26:50.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.536+0000 7f1caa1cd640 1 -- 172.21.15.60:0/3345615182 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca41976d0 con 0x7f1ca4100440 2024-03-04T12:26:50.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.536+0000 7f1ca0ff9640 1 -- 172.21.15.60:0/3345615182 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c940047a0 con 0x7f1ca4100440 2024-03-04T12:26:50.545 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.537+0000 7f1caa1cd640 1 -- 172.21.15.60:0/3345615182 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca4068950 con 0x7f1ca4100440 2024-03-04T12:26:50.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.539+0000 7f1ca0ff9640 1 -- 172.21.15.60:0/3345615182 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f1c940041d0 con 0x7f1ca4100440 2024-03-04T12:26:50.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.540+0000 7f1ca0ff9640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f1c8c0a1ef0 0x7f1c8c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.540+0000 7f1ca0ff9640 1 -- 172.21.15.60:0/3345615182 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f1c940cdb00 con 0x7f1ca4100440 2024-03-04T12:26:50.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.540+0000 7f1cacdac740 1 Processor -- start 2024-03-04T12:26:50.546 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.540+0000 7f1ca2ffd640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f1c8c0a1ef0 0x7f1c8c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.540+0000 7f1cacdac740 1 -- start start 2024-03-04T12:26:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.540+0000 7f1cacdac740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336b59c0 0x5606336b5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.540+0000 7f1cacdac740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b6370 0x5606336b67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.540+0000 7f1cacdac740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336c7b80 0x5606336c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.540+0000 7f1cacdac740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x560633619b40 con 0x5606336c7b80 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.540+0000 7f1cacdac740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606335c2d80 con 0x5606336b59c0 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.540+0000 7f1cacdac740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x560633615be0 con 0x5606336b6370 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1ca2ffd640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f1c8c0a1ef0 0x7f1c8c0a43b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009230 tx=0x7f1c98005fd0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1ca37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336b59c0 0x5606336b5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1ca3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336c7b80 0x5606336c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1ca37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336b59c0 0x5606336b5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:40124/0 (socket says 172.21.15.60:40124) 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1ca37fe640 1 -- 172.21.15.60:0/2365196380 learned_addr learned my addr 172.21.15.60:0/2365196380 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1ca2ffd640 1 --2- 172.21.15.60:0/2365196380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b6370 0x5606336b67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1ca37fe640 1 -- 172.21.15.60:0/2365196380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b6370 msgr2=0x5606336b67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/2365196380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b6370 0x5606336b67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1ca37fe640 1 -- 172.21.15.60:0/2365196380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336c7b80 msgr2=0x5606336c9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/2365196380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336c7b80 0x5606336c9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1ca37fe640 1 -- 172.21.15.60:0/2365196380 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606333f67e0 con 0x5606336b59c0 2024-03-04T12:26:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/2365196380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336b59c0 0x5606336b5da0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1c94099c00 tx=0x7f1c9409a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30dbc40694d6b2cb server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.541+0000 7f1c89ffb640 1 -- 172.21.15.60:0/2365196380 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c940a26a0 con 0x5606336b59c0 2024-03-04T12:26:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.542+0000 7f1c89ffb640 1 -- 172.21.15.60:0/2365196380 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c940b2070 con 0x5606336b59c0 2024-03-04T12:26:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.542+0000 7f1c89ffb640 1 -- 172.21.15.60:0/2365196380 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c940995b0 con 0x5606336b59c0 2024-03-04T12:26:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.542+0000 7f1cacdac740 1 -- 172.21.15.60:0/2365196380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336b59c0 msgr2=0x5606336b5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.542+0000 7f1cacdac740 1 --2- 172.21.15.60:0/2365196380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336b59c0 0x5606336b5da0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1c94099c00 tx=0x7f1c9409a6c0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.542+0000 7f1cacdac740 1 -- 172.21.15.60:0/2365196380 shutdown_connections 2024-03-04T12:26:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.542+0000 7f1cacdac740 1 --2- 172.21.15.60:0/2365196380 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336c7b80 0x5606336c9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.542+0000 7f1cacdac740 1 --2- 172.21.15.60:0/2365196380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b6370 0x5606336b67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.542+0000 7f1cacdac740 1 --2- 172.21.15.60:0/2365196380 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336b59c0 0x5606336b5da0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.542+0000 7f1cacdac740 1 -- 172.21.15.60:0/2365196380 >> 172.21.15.60:0/2365196380 conn(0x5606336b0680 msgr2=0x5606336b0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.542+0000 7f1cacdac740 1 -- 172.21.15.60:0/2365196380 shutdown_connections 2024-03-04T12:26:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.542+0000 7f1cacdac740 1 -- 172.21.15.60:0/2365196380 wait complete. 2024-03-04T12:26:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.543+0000 7f1cacdac740 1 -- start start 2024-03-04T12:26:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.543+0000 7f1cacdac740 1 Processor -- start 2024-03-04T12:26:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.543+0000 7f1cacdac740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b59c0 0x5606336ba920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.543+0000 7f1cacdac740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b6370 0x5606336b7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.543+0000 7f1cacdac740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336c7b80 0x5606336b82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.543+0000 7f1cacdac740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606336ca9d0 con 0x5606336b6370 2024-03-04T12:26:50.557 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.543+0000 7f1cacdac740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606335d61c0 con 0x5606336c7b80 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.543+0000 7f1ca2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b6370 0x5606336b7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1ca2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b6370 0x5606336b7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59232/0 (socket says 172.21.15.60:59232) 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1ca2ffd640 1 -- 172.21.15.60:0/2691260019 learned_addr learned my addr 172.21.15.60:0/2691260019 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.543+0000 7f1cacdac740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606336cab50 con 0x5606336b59c0 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1cacdac740 10 client.0 _ll_register_callbacks cb 0x5606335bbf80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b59c0 0x5606336ba920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1ca3fff640 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336c7b80 0x5606336b82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1ca2ffd640 1 -- 172.21.15.60:0/2691260019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b59c0 msgr2=0x5606336ba920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1ca2ffd640 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b59c0 0x5606336ba920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1ca2ffd640 1 -- 172.21.15.60:0/2691260019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336c7b80 msgr2=0x5606336b82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1ca2ffd640 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336c7b80 0x5606336b82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1ca2ffd640 1 -- 172.21.15.60:0/2691260019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560633456f70 con 0x5606336b6370 2024-03-04T12:26:50.558 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1ca2ffd640 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b6370 0x5606336b7cb0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009900 tx=0x7f1c9802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eefa74cc3e2dee3c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1c82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.544+0000 7f1c82ffd640 1 -- 172.21.15.60:0/2691260019 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9804f070 con 0x5606336b6370 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1c82ffd640 1 -- 172.21.15.60:0/2691260019 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f1c98004ec0 con 0x5606336b6370 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1cacdac740 1 -- 172.21.15.60:0/2691260019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c940b2070 con 0x5606336b6370 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1c82ffd640 1 -- 172.21.15.60:0/2691260019 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c980050c0 con 0x5606336b6370 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1cacdac740 10 client.16167 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1cacdac740 1 -- 172.21.15.60:0/2691260019 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5606332ee170 con 0x5606336b6370 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1c82ffd640 1 -- 172.21.15.60:0/2691260019 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c940b2070 con 0x5606336b6370 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1c82ffd640 1 -- 172.21.15.60:0/2691260019 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1c98034440 con 0x5606336b6370 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1cacdac740 10 client.16167 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1cacdac740 10 client.16167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1cacdac740 20 client.16167 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:50.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1cacdac740 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5606336d7cd0 0x5606336da0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1cacdac740 4 client.16167 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.545+0000 7f1cacdac740 1 -- 172.21.15.60:0/2691260019 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5606335ef960 con 0x5606336d7cd0 2024-03-04T12:26:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.546+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5606336d7cd0 0x5606336da0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.546+0000 7f1c82ffd640 10 client.16167 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.546+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5606336d7cd0 0x5606336da0e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57f398d313a110c server_cookie=1e5ce036cd9555b2 in_seq=0 out_seq=0 2024-03-04T12:26:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1c82ffd640 1 -- 172.21.15.60:0/2691260019 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5606335ef960 con 0x5606336d7cd0 2024-03-04T12:26:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1c82ffd640 10 client.16167 handle_command_reply: tid=0 2024-03-04T12:26:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1cacdac740 1 client.16167 shutdown 2024-03-04T12:26:50.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1cacdac740 1 -- 172.21.15.60:0/2691260019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b6370 msgr2=0x5606336b7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1cacdac740 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b6370 0x5606336b7cb0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009900 tx=0x7f1c9802bc30 comp rx=0 tx=0).stop 2024-03-04T12:26:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1cacdac740 1 -- 172.21.15.60:0/2691260019 shutdown_connections 2024-03-04T12:26:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1cacdac740 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5606336d7cd0 0x5606336da0e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1cacdac740 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336c7b80 0x5606336b82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1cacdac740 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b6370 0x5606336b7cb0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1cacdac740 1 --2- 172.21.15.60:0/2691260019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b59c0 0x5606336ba920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1cacdac740 1 -- 172.21.15.60:0/2691260019 >> 172.21.15.60:0/2691260019 conn(0x5606336b0680 msgr2=0x5606336b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1c82ffd640 0 client.16167 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1cacdac740 1 -- 172.21.15.60:0/2691260019 shutdown_connections 2024-03-04T12:26:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.547+0000 7f1cacdac740 1 -- 172.21.15.60:0/2691260019 wait complete. 2024-03-04T12:26:50.561 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.548+0000 7f1cacdac740 20 client.16167 trim_cache size 0 max 16384 2024-03-04T12:26:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.576+0000 7f1cacdac740 1 Processor -- start 2024-03-04T12:26:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.576+0000 7f1cacdac740 1 -- start start 2024-03-04T12:26:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.576+0000 7f1cacdac740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b59c0 0x5606336dea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1cacdac740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336c7b80 0x5606336ba3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1ca2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336c7b80 0x5606336ba3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1cacdac740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336def80 0x5606336ba920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1ca2ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336c7b80 0x5606336ba3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59242/0 (socket says 172.21.15.60:59242) 2024-03-04T12:26:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1cacdac740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca41ac010 con 0x5606336c7b80 2024-03-04T12:26:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1ca2ffd640 1 -- 172.21.15.60:0/929917140 learned_addr learned my addr 172.21.15.60:0/929917140 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1cacdac740 1 -- 172.21.15.60:0/929917140 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca41ad990 con 0x5606336def80 2024-03-04T12:26:50.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1cacdac740 1 -- 172.21.15.60:0/929917140 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca41ac670 con 0x5606336b59c0 2024-03-04T12:26:50.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/929917140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b59c0 0x5606336dea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1ca37fe640 1 -- 172.21.15.60:0/929917140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336def80 msgr2=0x5606336ba920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/929917140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336def80 0x5606336ba920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1ca37fe640 1 -- 172.21.15.60:0/929917140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336c7b80 msgr2=0x5606336ba3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/929917140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336c7b80 0x5606336ba3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1ca37fe640 1 -- 172.21.15.60:0/929917140 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606336b8630 con 0x5606336b59c0 2024-03-04T12:26:50.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1ca2ffd640 1 --2- 172.21.15.60:0/929917140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336c7b80 0x5606336ba3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:50.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.577+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/929917140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b59c0 0x5606336dea40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1c94099c00 tx=0x7f1c940962d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b58fe4959d935b24 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:50.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.578+0000 7f1c897fa640 1 -- 172.21.15.60:0/929917140 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c940a26a0 con 0x5606336b59c0 2024-03-04T12:26:50.592 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.578+0000 7f1c897fa640 1 -- 172.21.15.60:0/929917140 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c940b2040 con 0x5606336b59c0 2024-03-04T12:26:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.578+0000 7f1c897fa640 1 -- 172.21.15.60:0/929917140 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94098dc0 con 0x5606336b59c0 2024-03-04T12:26:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.579+0000 7f1cacdac740 1 -- 172.21.15.60:0/929917140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b59c0 msgr2=0x5606336dea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.579+0000 7f1cacdac740 1 --2- 172.21.15.60:0/929917140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b59c0 0x5606336dea40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1c94099c00 tx=0x7f1c940962d0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.579+0000 7f1cacdac740 1 -- 172.21.15.60:0/929917140 shutdown_connections 2024-03-04T12:26:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.579+0000 7f1cacdac740 1 --2- 172.21.15.60:0/929917140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336def80 0x5606336ba920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.579+0000 7f1cacdac740 1 --2- 172.21.15.60:0/929917140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336c7b80 0x5606336ba3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.579+0000 7f1cacdac740 1 --2- 172.21.15.60:0/929917140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336b59c0 0x5606336dea40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.579+0000 7f1cacdac740 1 -- 172.21.15.60:0/929917140 >> 172.21.15.60:0/929917140 conn(0x5606336b0680 msgr2=0x5606336d9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.579+0000 7f1cacdac740 1 -- 172.21.15.60:0/929917140 shutdown_connections 2024-03-04T12:26:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.579+0000 7f1cacdac740 1 -- 172.21.15.60:0/929917140 wait complete. 2024-03-04T12:26:50.593 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.579+0000 7f1cacdac740 1 -- start start 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.580+0000 7f1cacdac740 1 Processor -- start 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.580+0000 7f1cacdac740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b59c0 0x5606337286a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.580+0000 7f1cacdac740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336c7b80 0x560633728be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.580+0000 7f1cacdac740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336def80 0x5606336ce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.580+0000 7f1ca37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b59c0 0x5606337286a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.580+0000 7f1ca37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b59c0 0x5606337286a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59248/0 (socket says 172.21.15.60:59248) 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.580+0000 7f1ca37fe640 1 -- 172.21.15.60:0/1362558898 learned_addr learned my addr 172.21.15.60:0/1362558898 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1cacdac740 1 -- 172.21.15.60:0/1362558898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca41ad330 con 0x5606336b59c0 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1cacdac740 1 -- 172.21.15.60:0/1362558898 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca41accd0 con 0x5606336c7b80 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1ca2ffd640 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336c7b80 0x560633728be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1cacdac740 1 -- 172.21.15.60:0/1362558898 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca41aa2b0 con 0x5606336def80 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1ca3fff640 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336def80 0x5606336ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1ca37fe640 1 -- 172.21.15.60:0/1362558898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336def80 msgr2=0x5606336ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336def80 0x5606336ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.594 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1ca37fe640 1 -- 172.21.15.60:0/1362558898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336c7b80 msgr2=0x560633728be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336c7b80 0x560633728be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1ca37fe640 1 -- 172.21.15.60:0/1362558898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606336ce8f0 con 0x5606336b59c0 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1ca3fff640 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336def80 0x5606336ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1cacdac740 10 client.0 _ll_register_callbacks cb 0x7f1c940044c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1cacdac740 1 -- 172.21.15.60:0/1362558898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c940b2040 con 0x5606336b59c0 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1c82ffd640 10 client.16173 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1ca37fe640 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b59c0 0x5606337286a0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1c94099e90 tx=0x7f1c940cad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5f44fafdd930fe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.581+0000 7f1c82ffd640 1 -- 172.21.15.60:0/1362558898 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9407c070 con 0x5606336b59c0 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.582+0000 7f1c82ffd640 1 -- 172.21.15.60:0/1362558898 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f1c94083120 con 0x5606336b59c0 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.582+0000 7f1c82ffd640 1 -- 172.21.15.60:0/1362558898 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c940b6030 con 0x5606336b59c0 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.582+0000 7f1cacdac740 10 client.16173 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.582+0000 7f1cacdac740 1 -- 172.21.15.60:0/1362558898 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({fsmap=29}) v3 -- 0x56063372fd10 con 0x5606336b59c0 2024-03-04T12:26:50.595 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.582+0000 7f1c82ffd640 1 -- 172.21.15.60:0/1362558898 <== mon.0 v2:172.21.15.60:3300/0 4 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1c94096570 con 0x5606336b59c0 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.582+0000 7f1cacdac740 10 client.16173 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.582+0000 7f1cacdac740 10 client.16173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.582+0000 7f1cacdac740 20 client.16173 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.582+0000 7f1cacdac740 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56063372fee0 0x5606337322d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.582+0000 7f1cacdac740 4 client.16173 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.582+0000 7f1cacdac740 1 -- 172.21.15.60:0/1362558898 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5606335db080 con 0x56063372fee0 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.583+0000 7f1ca2ffd640 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56063372fee0 0x5606337322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.583+0000 7f1ca2ffd640 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56063372fee0 0x5606337322d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed31d78211b39704 server_cookie=11fd12fdb3df89e1 in_seq=0 out_seq=0 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.583+0000 7f1c82ffd640 10 client.16173 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.584+0000 7f1c82ffd640 1 -- 172.21.15.60:0/1362558898 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5606335db080 con 0x56063372fee0 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.584+0000 7f1c82ffd640 10 client.16173 handle_command_reply: tid=0 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.584+0000 7f1cacdac740 1 client.16173 shutdown 2024-03-04T12:26:50.596 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.585+0000 7f1cacdac740 1 -- 172.21.15.60:0/1362558898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b59c0 msgr2=0x5606337286a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.585+0000 7f1cacdac740 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b59c0 0x5606337286a0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f1c94099e90 tx=0x7f1c940cad00 comp rx=0 tx=0).stop 2024-03-04T12:26:50.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.585+0000 7f1cacdac740 1 -- 172.21.15.60:0/1362558898 shutdown_connections 2024-03-04T12:26:50.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.585+0000 7f1cacdac740 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56063372fee0 0x5606337322d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.585+0000 7f1cacdac740 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606336def80 0x5606336ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.585+0000 7f1cacdac740 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606336c7b80 0x560633728be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.585+0000 7f1c82ffd640 0 client.16173 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:50.597 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.585+0000 7f1cacdac740 1 --2- 172.21.15.60:0/1362558898 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5606336b59c0 0x5606337286a0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.585+0000 7f1cacdac740 1 -- 172.21.15.60:0/1362558898 >> 172.21.15.60:0/1362558898 conn(0x5606336b0680 msgr2=0x5606336b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:50.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.585+0000 7f1cacdac740 1 -- 172.21.15.60:0/1362558898 shutdown_connections 2024-03-04T12:26:50.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.585+0000 7f1cacdac740 1 -- 172.21.15.60:0/1362558898 wait complete. 2024-03-04T12:26:50.598 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.585+0000 7f1cacdac740 20 client.16173 trim_cache size 0 max 16384 2024-03-04T12:26:50.598 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:26:50.598 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": 0, 2024-03-04T12:26:50.598 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_tag": "b8a23e46-1261-4201-bbac-188a95ded3fe", 2024-03-04T12:26:50.598 INFO:teuthology.orchestra.run.smithi060.stdout: "mode": "asynchronous" 2024-03-04T12:26:50.599 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:26:50.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.586+0000 7f1caa1cd640 1 -- 172.21.15.60:0/3345615182 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f1c8c0a1ef0 msgr2=0x7f1c8c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.586+0000 7f1caa1cd640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f1c8c0a1ef0 0x7f1c8c0a43b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009230 tx=0x7f1c98005fd0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.599 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.586+0000 7f1caa1cd640 1 -- 172.21.15.60:0/3345615182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ca4100440 msgr2=0x7f1ca419c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.586+0000 7f1caa1cd640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ca4100440 0x7f1ca419c710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400d950 tx=0x7f1c9400de10 comp rx=0 tx=0).stop 2024-03-04T12:26:50.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.586+0000 7f1caa1cd640 1 -- 172.21.15.60:0/3345615182 shutdown_connections 2024-03-04T12:26:50.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.586+0000 7f1caa1cd640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f1c8c0a1ef0 0x7f1c8c0a43b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.586+0000 7f1caa1cd640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1ca41968c0 0x7f1ca4196d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.586+0000 7f1caa1cd640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ca4105510 0x7f1ca419cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.586+0000 7f1caa1cd640 1 --2- 172.21.15.60:0/3345615182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ca4100440 0x7f1ca419c710 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.586+0000 7f1caa1cd640 1 -- 172.21.15.60:0/3345615182 >> 172.21.15.60:0/3345615182 conn(0x7f1ca40773e0 msgr2=0x7f1ca40fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:50.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.586+0000 7f1caa1cd640 1 -- 172.21.15.60:0/3345615182 shutdown_connections 2024-03-04T12:26:50.600 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.586+0000 7f1caa1cd640 1 -- 172.21.15.60:0/3345615182 wait complete. 2024-03-04T12:26:50.604 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T12:26:50.604 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T12:26:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.706+0000 7fe1aa644640 1 -- 172.21.15.60:0/2271719017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a41054e0 msgr2=0x7fe1a410d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.706+0000 7fe1aa644640 1 --2- 172.21.15.60:0/2271719017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a41054e0 0x7fe1a410d0b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe19c009ca0 tx=0x7fe19c02c0f0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.706+0000 7fe1aa644640 1 -- 172.21.15.60:0/2271719017 shutdown_connections 2024-03-04T12:26:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.706+0000 7fe1aa644640 1 --2- 172.21.15.60:0/2271719017 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a41054e0 0x7fe1a410d0b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.706+0000 7fe1aa644640 1 --2- 172.21.15.60:0/2271719017 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1a4100dc0 0x7fe1a4104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.706+0000 7fe1aa644640 1 --2- 172.21.15.60:0/2271719017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1a4100410 0x7fe1a41007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.706+0000 7fe1aa644640 1 -- 172.21.15.60:0/2271719017 >> 172.21.15.60:0/2271719017 conn(0x7fe1a4077330 msgr2=0x7fe1a4077740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:50.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.706+0000 7fe1aa644640 1 -- 172.21.15.60:0/2271719017 shutdown_connections 2024-03-04T12:26:50.714 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.706+0000 7fe1aa644640 1 -- 172.21.15.60:0/2271719017 wait complete. 2024-03-04T12:26:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.707+0000 7fe1aa644640 1 Processor -- start 2024-03-04T12:26:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.707+0000 7fe1aa644640 1 -- start start 2024-03-04T12:26:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.707+0000 7fe1aa644640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1a4100410 0x7fe1a419d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.707+0000 7fe1aa644640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1a41054e0 0x7fe1a419dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.707+0000 7fe1aa644640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a4197970 0x7fe1a4197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.707+0000 7fe1aa644640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a41116b0 con 0x7fe1a4197970 2024-03-04T12:26:50.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.707+0000 7fe1aa644640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a4111530 con 0x7fe1a41054e0 2024-03-04T12:26:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.707+0000 7fe1aa644640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a4111830 con 0x7fe1a4100410 2024-03-04T12:26:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1a41054e0 0x7fe1a419dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a8be1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a4197970 0x7fe1a4197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1a41054e0 0x7fe1a419dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:40166/0 (socket says 172.21.15.60:40166) 2024-03-04T12:26:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a37fe640 1 -- 172.21.15.60:0/950882861 learned_addr learned my addr 172.21.15.60:0/950882861 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a3fff640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1a4100410 0x7fe1a419d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a37fe640 1 -- 172.21.15.60:0/950882861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1a4100410 msgr2=0x7fe1a419d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1a4100410 0x7fe1a419d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a37fe640 1 -- 172.21.15.60:0/950882861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a4197970 msgr2=0x7fe1a4197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a4197970 0x7fe1a4197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a37fe640 1 -- 172.21.15.60:0/950882861 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a4198490 con 0x7fe1a41054e0 2024-03-04T12:26:50.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a3fff640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1a4100410 0x7fe1a419d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:50.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1a41054e0 0x7fe1a419dd00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe198005fd0 tx=0x7fe19800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:50.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.708+0000 7fe1a17fa640 1 -- 172.21.15.60:0/950882861 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198017070 con 0x7fe1a41054e0 2024-03-04T12:26:50.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.709+0000 7fe1a17fa640 1 -- 172.21.15.60:0/950882861 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe198009c30 con 0x7fe1a41054e0 2024-03-04T12:26:50.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.709+0000 7fe1a17fa640 1 -- 172.21.15.60:0/950882861 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198010460 con 0x7fe1a41054e0 2024-03-04T12:26:50.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.709+0000 7fe1aa644640 1 -- 172.21.15.60:0/950882861 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a4198780 con 0x7fe1a41054e0 2024-03-04T12:26:50.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.709+0000 7fe1aa644640 1 -- 172.21.15.60:0/950882861 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a410dc70 con 0x7fe1a41054e0 2024-03-04T12:26:50.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.711+0000 7fe1ad223740 1 Processor -- start 2024-03-04T12:26:50.726 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1ad223740 1 -- start start 2024-03-04T12:26:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1a17fa640 1 -- 172.21.15.60:0/950882861 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7fe198010600 con 0x7fe1a41054e0 2024-03-04T12:26:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1ad223740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 0x55e61e8c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1ad223740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1ad223740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8d2c90 0x55e61e8d5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1ad223740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61e82ce00 con 0x55e61e8c9790 2024-03-04T12:26:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1ad223740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61e7d5b10 con 0x55e61e8d2c90 2024-03-04T12:26:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1ad223740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61e828ea0 con 0x55e61e8ca0b0 2024-03-04T12:26:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1a8be1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 0x55e61e8c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1a8be1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 0x55e61e8c9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59276/0 (socket says 172.21.15.60:59276) 2024-03-04T12:26:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1a8be1640 1 -- 172.21.15.60:0/1030094970 learned_addr learned my addr 172.21.15.60:0/1030094970 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/1030094970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8d2c90 0x55e61e8d5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.727 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1a3fff640 1 --2- 172.21.15.60:0/1030094970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1a8be1640 1 -- 172.21.15.60:0/1030094970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8ca0b0 msgr2=0x55e61e8d2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.712+0000 7fe1a8be1640 1 --2- 172.21.15.60:0/1030094970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.713+0000 7fe1a8be1640 1 -- 172.21.15.60:0/1030094970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8d2c90 msgr2=0x55e61e8d5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.713+0000 7fe1a8be1640 1 --2- 172.21.15.60:0/1030094970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8d2c90 0x55e61e8d5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.713+0000 7fe1a8be1640 1 -- 172.21.15.60:0/1030094970 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e61e60b7d0 con 0x55e61e8c9790 2024-03-04T12:26:50.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.713+0000 7fe1a17fa640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fe18c0a2010 0x7fe18c0a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.713+0000 7fe1a3fff640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fe18c0a2010 0x7fe18c0a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.713+0000 7fe1a8be1640 1 --2- 172.21.15.60:0/1030094970 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 0x55e61e8c9b70 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe19c009ca0 tx=0x7fe19c005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3346aff7ffe619 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:50.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.713+0000 7fe1a17fa640 1 -- 172.21.15.60:0/950882861 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fe1980ce3c0 con 0x7fe1a41054e0 2024-03-04T12:26:50.728 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.713+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/1030094970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8d2c90 0x55e61e8d5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:26:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.713+0000 7fe18a7fc640 1 -- 172.21.15.60:0/1030094970 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c04b070 con 0x55e61e8c9790 2024-03-04T12:26:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.713+0000 7fe18a7fc640 1 -- 172.21.15.60:0/1030094970 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe19c032c90 con 0x55e61e8c9790 2024-03-04T12:26:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.713+0000 7fe18a7fc640 1 -- 172.21.15.60:0/1030094970 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c004040 con 0x55e61e8c9790 2024-03-04T12:26:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.714+0000 7fe1ad223740 1 -- 172.21.15.60:0/1030094970 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 msgr2=0x55e61e8c9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.714+0000 7fe1ad223740 1 --2- 172.21.15.60:0/1030094970 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 0x55e61e8c9b70 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fe19c009ca0 tx=0x7fe19c005c50 comp rx=0 tx=0).stop 2024-03-04T12:26:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.714+0000 7fe1a3fff640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fe18c0a2010 0x7fe18c0a44d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55e61e8c9260 tx=0x7fe19400b410 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.714+0000 7fe1ad223740 1 -- 172.21.15.60:0/1030094970 shutdown_connections 2024-03-04T12:26:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.714+0000 7fe1ad223740 1 --2- 172.21.15.60:0/1030094970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8d2c90 0x55e61e8d5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.714+0000 7fe1ad223740 1 --2- 172.21.15.60:0/1030094970 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.714+0000 7fe1ad223740 1 --2- 172.21.15.60:0/1030094970 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 0x55e61e8c9b70 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.729 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.714+0000 7fe1ad223740 1 -- 172.21.15.60:0/1030094970 >> 172.21.15.60:0/1030094970 conn(0x55e61e8c3ec0 msgr2=0x55e61e8c42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.714+0000 7fe1ad223740 1 -- 172.21.15.60:0/1030094970 shutdown_connections 2024-03-04T12:26:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.714+0000 7fe1ad223740 1 -- 172.21.15.60:0/1030094970 wait complete. 2024-03-04T12:26:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.714+0000 7fe1ad223740 1 -- start start 2024-03-04T12:26:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1ad223740 1 Processor -- start 2024-03-04T12:26:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1ad223740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8c9790 0x55e61e8d2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1ad223740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8ca0b0 0x55e61e8cf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1ad223740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8d2c90 0x55e61e8cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1ad223740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61e8d5a20 con 0x55e61e8ca0b0 2024-03-04T12:26:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1ad223740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61e7e90a0 con 0x55e61e8d2c90 2024-03-04T12:26:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1ad223740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61e8d5ba0 con 0x55e61e8c9790 2024-03-04T12:26:50.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1ad223740 10 client.0 _ll_register_callbacks cb 0x55e61e7e8a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1a8be1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8d2c90 0x55e61e8cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1a3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8c9790 0x55e61e8d2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1a8be1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8d2c90 0x55e61e8cfcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:40192/0 (socket says 172.21.15.60:40192) 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1a3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8c9790 0x55e61e8d2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:44726/0 (socket says 172.21.15.60:44726) 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1a8be1640 1 -- 172.21.15.60:0/926675963 learned_addr learned my addr 172.21.15.60:0/926675963 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.715+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/926675963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8ca0b0 0x55e61e8cf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1a3fff640 1 -- 172.21.15.60:0/926675963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8d2c90 msgr2=0x55e61e8cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1a3fff640 1 --2- 172.21.15.60:0/926675963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8d2c90 0x55e61e8cfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1a3fff640 1 -- 172.21.15.60:0/926675963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8ca0b0 msgr2=0x55e61e8cf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1a3fff640 1 --2- 172.21.15.60:0/926675963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8ca0b0 0x55e61e8cf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1a3fff640 1 -- 172.21.15.60:0/926675963 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e61e5413f0 con 0x55e61e8c9790 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1a3fff640 1 --2- 172.21.15.60:0/926675963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8c9790 0x55e61e8d2400 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4198ab0 tx=0x7fe1940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab9463d688012ee server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1837fe640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1837fe640 1 -- 172.21.15.60:0/926675963 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19401fa90 con 0x55e61e8c9790 2024-03-04T12:26:50.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1837fe640 1 -- 172.21.15.60:0/926675963 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fe194017030 con 0x55e61e8c9790 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1ad223740 1 -- 172.21.15.60:0/926675963 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe19c032c90 con 0x55e61e8c9790 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1837fe640 1 -- 172.21.15.60:0/926675963 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe19401fc30 con 0x55e61e8c9790 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1ad223740 10 client.25711 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.716+0000 7fe1ad223740 1 -- 172.21.15.60:0/926675963 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e61e5010f0 con 0x55e61e8c9790 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.717+0000 7fe1837fe640 1 -- 172.21.15.60:0/926675963 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe19c032c90 con 0x55e61e8c9790 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.717+0000 7fe1837fe640 1 -- 172.21.15.60:0/926675963 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe19401c3f0 con 0x55e61e8c9790 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.717+0000 7fe1ad223740 10 client.25711 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.717+0000 7fe1ad223740 10 client.25711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.717+0000 7fe1ad223740 20 client.25711 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.717+0000 7fe1ad223740 1 --2- 172.21.15.60:0/926675963 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55e61e8e27a0 0x55e61e8e4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.717+0000 7fe1ad223740 4 client.25711 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.717+0000 7fe1ad223740 1 -- 172.21.15.60:0/926675963 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e61e83a340 con 0x55e61e8e27a0 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.717+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/926675963 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55e61e8e27a0 0x55e61e8e4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.732 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.718+0000 7fe1837fe640 10 client.25711 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.718+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/926675963 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55e61e8e27a0 0x55e61e8e4bb0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7d930cb28c3bba2 server_cookie=29715fa6feb8c915 in_seq=0 out_seq=0 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.718+0000 7fe1837fe640 1 -- 172.21.15.60:0/926675963 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e61e83a340 con 0x55e61e8e27a0 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.718+0000 7fe1837fe640 10 client.25711 handle_command_reply: tid=0 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.718+0000 7fe1ad223740 1 client.25711 shutdown 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.718+0000 7fe1ad223740 1 -- 172.21.15.60:0/926675963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8c9790 msgr2=0x55e61e8d2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.718+0000 7fe1ad223740 1 --2- 172.21.15.60:0/926675963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8c9790 0x55e61e8d2400 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4198ab0 tx=0x7fe1940046c0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.718+0000 7fe1ad223740 1 -- 172.21.15.60:0/926675963 shutdown_connections 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.718+0000 7fe1ad223740 1 --2- 172.21.15.60:0/926675963 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55e61e8e27a0 0x55e61e8e4bb0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.718+0000 7fe1837fe640 0 client.25711 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.718+0000 7fe1ad223740 1 --2- 172.21.15.60:0/926675963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8d2c90 0x55e61e8cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.718+0000 7fe1ad223740 1 --2- 172.21.15.60:0/926675963 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8ca0b0 0x55e61e8cf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.719+0000 7fe1ad223740 1 --2- 172.21.15.60:0/926675963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8c9790 0x55e61e8d2400 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.719+0000 7fe1ad223740 1 -- 172.21.15.60:0/926675963 >> 172.21.15.60:0/926675963 conn(0x55e61e8c3ec0 msgr2=0x55e61e8c5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:50.733 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.719+0000 7fe1ad223740 1 -- 172.21.15.60:0/926675963 shutdown_connections 2024-03-04T12:26:50.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.719+0000 7fe1ad223740 1 -- 172.21.15.60:0/926675963 wait complete. 2024-03-04T12:26:50.734 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.719+0000 7fe1ad223740 20 client.25711 trim_cache size 0 max 16384 2024-03-04T12:26:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1ad223740 1 Processor -- start 2024-03-04T12:26:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1ad223740 1 -- start start 2024-03-04T12:26:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1ad223740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 0x55e61e8d1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1ad223740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1ad223740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8d2c90 0x55e61e8d9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1ad223740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a41ac2f0 con 0x55e61e8c9790 2024-03-04T12:26:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1ad223740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a41adc70 con 0x55e61e8ca0b0 2024-03-04T12:26:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1ad223740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a41ac950 con 0x55e61e8d2c90 2024-03-04T12:26:50.757 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1a3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 0x55e61e8d1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1a37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1a37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:40204/0 (socket says 172.21.15.60:40204) 2024-03-04T12:26:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1a37fe640 1 -- 172.21.15.60:0/1750158545 learned_addr learned my addr 172.21.15.60:0/1750158545 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.749+0000 7fe1a3fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 0x55e61e8d1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59304/0 (socket says 172.21.15.60:59304) 2024-03-04T12:26:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.750+0000 7fe1a8be1640 1 --2- 172.21.15.60:0/1750158545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8d2c90 0x55e61e8d9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.750+0000 7fe1a37fe640 1 -- 172.21.15.60:0/1750158545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8d2c90 msgr2=0x55e61e8d9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.750+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/1750158545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8d2c90 0x55e61e8d9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.750+0000 7fe1a37fe640 1 -- 172.21.15.60:0/1750158545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 msgr2=0x55e61e8d1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.750+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/1750158545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 0x55e61e8d1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.750+0000 7fe1a37fe640 1 -- 172.21.15.60:0/1750158545 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e61e8e9da0 con 0x55e61e8ca0b0 2024-03-04T12:26:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.750+0000 7fe1a8be1640 1 --2- 172.21.15.60:0/1750158545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8d2c90 0x55e61e8d9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:26:50.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.750+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/1750158545 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d2400 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe19809a750 tx=0x7fe1980d0940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2addf2297713ebc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.750+0000 7fe189ffb640 1 -- 172.21.15.60:0/1750158545 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198009de0 con 0x55e61e8ca0b0 2024-03-04T12:26:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.750+0000 7fe189ffb640 1 -- 172.21.15.60:0/1750158545 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe19809a7c0 con 0x55e61e8ca0b0 2024-03-04T12:26:50.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.750+0000 7fe189ffb640 1 -- 172.21.15.60:0/1750158545 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1980ce120 con 0x55e61e8ca0b0 2024-03-04T12:26:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.751+0000 7fe1ad223740 1 -- 172.21.15.60:0/1750158545 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8ca0b0 msgr2=0x55e61e8d2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.751+0000 7fe1ad223740 1 --2- 172.21.15.60:0/1750158545 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d2400 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe19809a750 tx=0x7fe1980d0940 comp rx=0 tx=0).stop 2024-03-04T12:26:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.751+0000 7fe1ad223740 1 -- 172.21.15.60:0/1750158545 shutdown_connections 2024-03-04T12:26:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.751+0000 7fe1ad223740 1 --2- 172.21.15.60:0/1750158545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8d2c90 0x55e61e8d9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.751+0000 7fe1ad223740 1 --2- 172.21.15.60:0/1750158545 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d2400 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.751+0000 7fe1ad223740 1 --2- 172.21.15.60:0/1750158545 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8c9790 0x55e61e8d1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.751+0000 7fe1ad223740 1 -- 172.21.15.60:0/1750158545 >> 172.21.15.60:0/1750158545 conn(0x55e61e8c3ec0 msgr2=0x55e61e8cf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:50.763 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.751+0000 7fe1ad223740 1 -- 172.21.15.60:0/1750158545 shutdown_connections 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.752+0000 7fe1ad223740 1 -- 172.21.15.60:0/1750158545 wait complete. 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.752+0000 7fe1ad223740 1 -- start start 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.752+0000 7fe1ad223740 1 Processor -- start 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1ad223740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8c9790 0x55e61e8d60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1ad223740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1ad223740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8d2c90 0x55e61e931cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1ad223740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a41ad610 con 0x55e61e8ca0b0 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1ad223740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a41acfb0 con 0x55e61e8c9790 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1ad223740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a41aa590 con 0x55e61e8d2c90 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1a37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1a37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59320/0 (socket says 172.21.15.60:59320) 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1a37fe640 1 -- 172.21.15.60:0/3765577420 learned_addr learned my addr 172.21.15.60:0/3765577420 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1a3fff640 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8c9790 0x55e61e8d60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1a8be1640 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8d2c90 0x55e61e931cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1a37fe640 1 -- 172.21.15.60:0/3765577420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8d2c90 msgr2=0x55e61e931cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8d2c90 0x55e61e931cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1a37fe640 1 -- 172.21.15.60:0/3765577420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8c9790 msgr2=0x55e61e8d60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:26:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1ad223740 10 client.0 _ll_register_callbacks cb 0x7fe194010800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:26:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8c9790 0x55e61e8d60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1a37fe640 1 -- 172.21.15.60:0/3765577420 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e61e932300 con 0x55e61e8ca0b0 2024-03-04T12:26:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1ad223740 1 -- 172.21.15.60:0/3765577420 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe19809a7c0 con 0x55e61e8ca0b0 2024-03-04T12:26:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1a37fe640 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d6610 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe19809a4c0 tx=0x7fe1980cb5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ff3ad29ee46a18 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:26:50.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.753+0000 7fe1837fe640 10 client.16191 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.754+0000 7fe1837fe640 1 -- 172.21.15.60:0/3765577420 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1980035f0 con 0x55e61e8ca0b0 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.754+0000 7fe1837fe640 1 -- 172.21.15.60:0/3765577420 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fe1980b3220 con 0x55e61e8ca0b0 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.754+0000 7fe1837fe640 1 -- 172.21.15.60:0/3765577420 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe198009de0 con 0x55e61e8ca0b0 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.754+0000 7fe1ad223740 10 client.16191 fetch_fsmap learned FSMap version 29 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.754+0000 7fe1ad223740 1 -- 172.21.15.60:0/3765577420 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({fsmap=29}) v3 -- 0x55e61e9395e0 con 0x55e61e8ca0b0 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.754+0000 7fe1837fe640 1 -- 172.21.15.60:0/3765577420 <== mon.0 v2:172.21.15.60:3300/0 4 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe198096e30 con 0x55e61e8ca0b0 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.754+0000 7fe1ad223740 10 client.16191 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.754+0000 7fe1ad223740 10 client.16191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.754+0000 7fe1ad223740 20 client.16191 populate_metadata read hostname 'smithi060' 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.754+0000 7fe1ad223740 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55e61e8e27a0 0x55e61e93bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.754+0000 7fe1ad223740 4 client.16191 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.754+0000 7fe1ad223740 1 -- 172.21.15.60:0/3765577420 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e61e834690 con 0x55e61e8e27a0 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.755+0000 7fe1a3fff640 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55e61e8e27a0 0x55e61e93bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:26:50.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.755+0000 7fe1837fe640 10 client.16191 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.755+0000 7fe1a3fff640 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55e61e8e27a0 0x55e61e93bae0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a859f1a0b0baef7 server_cookie=cdc7752d0ca14993 in_seq=0 out_seq=0 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.755+0000 7fe1837fe640 1 -- 172.21.15.60:0/3765577420 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e61e834690 con 0x55e61e8e27a0 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.755+0000 7fe1837fe640 10 client.16191 handle_command_reply: tid=0 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.755+0000 7fe1ad223740 1 client.16191 shutdown 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.755+0000 7fe1ad223740 1 -- 172.21.15.60:0/3765577420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8ca0b0 msgr2=0x55e61e8d6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.755+0000 7fe1ad223740 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d6610 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe19809a4c0 tx=0x7fe1980cb5d0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1ad223740 1 -- 172.21.15.60:0/3765577420 shutdown_connections 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1ad223740 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55e61e8e27a0 0x55e61e93bae0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1ad223740 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e61e8d2c90 0x55e61e931cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1ad223740 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e61e8ca0b0 0x55e61e8d6610 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1ad223740 1 --2- 172.21.15.60:0/3765577420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e61e8c9790 0x55e61e8d60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1837fe640 0 client.16191 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1ad223740 1 -- 172.21.15.60:0/3765577420 >> 172.21.15.60:0/3765577420 conn(0x55e61e8c3ec0 msgr2=0x55e61e8ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1ad223740 1 -- 172.21.15.60:0/3765577420 shutdown_connections 2024-03-04T12:26:50.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1ad223740 1 -- 172.21.15.60:0/3765577420 wait complete. 2024-03-04T12:26:50.768 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1ad223740 20 client.16191 trim_cache size 0 max 16384 2024-03-04T12:26:50.768 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:26:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T12:26:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "scrubs": { 2024-03-04T12:26:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "b8a23e46-1261-4201-bbac-188a95ded3fe": { 2024-03-04T12:26:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/", 2024-03-04T12:26:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "b8a23e46-1261-4201-bbac-188a95ded3fe", 2024-03-04T12:26:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: "options": "recursive,force" 2024-03-04T12:26:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:26:50.768 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:26:50.769 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:26:50.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1aa644640 1 -- 172.21.15.60:0/950882861 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fe18c0a2010 msgr2=0x7fe18c0a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1aa644640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fe18c0a2010 0x7fe18c0a44d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55e61e8c9260 tx=0x7fe19400b410 comp rx=0 tx=0).stop 2024-03-04T12:26:50.769 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1aa644640 1 -- 172.21.15.60:0/950882861 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1a41054e0 msgr2=0x7fe1a419dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:26:50.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1aa644640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1a41054e0 0x7fe1a419dd00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe198005fd0 tx=0x7fe19800dde0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1aa644640 1 -- 172.21.15.60:0/950882861 shutdown_connections 2024-03-04T12:26:50.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1aa644640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fe18c0a2010 0x7fe18c0a44d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1aa644640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe1a4197970 0x7fe1a4197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1aa644640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1a41054e0 0x7fe1a419dd00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1aa644640 1 --2- 172.21.15.60:0/950882861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1a4100410 0x7fe1a419d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:26:50.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.756+0000 7fe1aa644640 1 -- 172.21.15.60:0/950882861 >> 172.21.15.60:0/950882861 conn(0x7fe1a4077330 msgr2=0x7fe1a40fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:26:50.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.757+0000 7fe1aa644640 1 -- 172.21.15.60:0/950882861 shutdown_connections 2024-03-04T12:26:50.770 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:26:50.757+0000 7fe1aa644640 1 -- 172.21.15.60:0/950882861 wait complete. 2024-03-04T12:26:50.777 INFO:tasks.cephfs.filesystem:scrub status for tag:b8a23e46-1261-4201-bbac-188a95ded3fe - {'path': '/', 'tag': 'b8a23e46-1261-4201-bbac-188a95ded3fe', 'options': 'recursive,force'} 2024-03-04T12:26:51.104 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:51 smithi060 ceph-mon[32745]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.4 KiB/s wr, 14 op/s 2024-03-04T12:26:51.104 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:51 smithi060 ceph-mon[32745]: scrub queued for path: / 2024-03-04T12:26:51.104 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:51 smithi060 ceph-mon[32745]: scrub summary: idle+waiting paths [/] 2024-03-04T12:26:51.104 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:51 smithi060 ceph-mon[32745]: scrub summary: active paths [/] 2024-03-04T12:26:51.104 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:51 smithi060 ceph-mon[32745]: scrub summary: idle+waiting paths [/] 2024-03-04T12:26:51.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:51 smithi086 ceph-mon[36851]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.4 KiB/s wr, 14 op/s 2024-03-04T12:26:51.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:51 smithi086 ceph-mon[36851]: scrub queued for path: / 2024-03-04T12:26:51.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:51 smithi086 ceph-mon[36851]: scrub summary: idle+waiting paths [/] 2024-03-04T12:26:51.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:51 smithi086 ceph-mon[36851]: scrub summary: active paths [/] 2024-03-04T12:26:51.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:51 smithi086 ceph-mon[36851]: scrub summary: idle+waiting paths [/] 2024-03-04T12:26:51.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:51 smithi063 ceph-mon[36717]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.4 KiB/s wr, 14 op/s 2024-03-04T12:26:51.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:51 smithi063 ceph-mon[36717]: scrub queued for path: / 2024-03-04T12:26:51.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:51 smithi063 ceph-mon[36717]: scrub summary: idle+waiting paths [/] 2024-03-04T12:26:51.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:51 smithi063 ceph-mon[36717]: scrub summary: active paths [/] 2024-03-04T12:26:51.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:51 smithi063 ceph-mon[36717]: scrub summary: idle+waiting paths [/] 2024-03-04T12:26:52.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:52 smithi060 ceph-mon[32745]: scrub summary: idle 2024-03-04T12:26:52.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:52 smithi086 ceph-mon[36851]: scrub summary: idle 2024-03-04T12:26:52.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:52 smithi063 ceph-mon[36717]: scrub summary: idle 2024-03-04T12:26:53.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:53 smithi060 ceph-mon[32745]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-04T12:26:53.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:53 smithi086 ceph-mon[36851]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-04T12:26:53.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:53 smithi063 ceph-mon[36717]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-04T12:26:55.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:55 smithi060 ceph-mon[32745]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-04T12:26:55.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:55 smithi086 ceph-mon[36851]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-04T12:26:55.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:55 smithi063 ceph-mon[36717]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-04T12:26:57.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:57 smithi060 ceph-mon[32745]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-04T12:26:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:57 smithi086 ceph-mon[36851]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-04T12:26:57.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:57 smithi063 ceph-mon[36717]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-04T12:26:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:59 smithi060 ceph-mon[32745]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-04T12:26:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:26:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:26:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:26:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:26:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:26:59 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:26:59.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:59 smithi086 ceph-mon[36851]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-04T12:26:59.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:26:59.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:26:59.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:26:59.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:26:59.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:26:59 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:26:59.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:59 smithi063 ceph-mon[36717]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-04T12:26:59.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:26:59.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:26:59.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:26:59.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:26:59.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:26:59 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:27:00.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:00 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:27:00.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:00 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:27:00.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:00 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:27:00.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:00 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:27:00.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:00 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:27:00.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:00 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:27:00.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:00 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:27:00.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:00 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:27:00.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:00 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:27:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.084+0000 7ffb5679f640 1 -- 172.21.15.60:0/1341002674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb50100520 msgr2=0x7ffb50100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.084+0000 7ffb5679f640 1 --2- 172.21.15.60:0/1341002674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb50100520 0x7ffb50100900 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009930 tx=0x7ffb4c02bc80 comp rx=0 tx=0).stop 2024-03-04T12:27:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.084+0000 7ffb5679f640 1 -- 172.21.15.60:0/1341002674 shutdown_connections 2024-03-04T12:27:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.084+0000 7ffb5679f640 1 --2- 172.21.15.60:0/1341002674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb501053f0 0x7ffb5010cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.084+0000 7ffb5679f640 1 --2- 172.21.15.60:0/1341002674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb50100ed0 0x7ffb50104cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.084+0000 7ffb5679f640 1 --2- 172.21.15.60:0/1341002674 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb50100520 0x7ffb50100900 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.084+0000 7ffb5679f640 1 -- 172.21.15.60:0/1341002674 >> 172.21.15.60:0/1341002674 conn(0x7ffb50075b90 msgr2=0x7ffb50075fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.084+0000 7ffb5679f640 1 -- 172.21.15.60:0/1341002674 shutdown_connections 2024-03-04T12:27:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.084+0000 7ffb5679f640 1 -- 172.21.15.60:0/1341002674 wait complete. 2024-03-04T12:27:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5679f640 1 Processor -- start 2024-03-04T12:27:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5679f640 1 -- start start 2024-03-04T12:27:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5679f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb50100520 0x7ffb5010d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5679f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb501053f0 0x7ffb5010de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5679f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb5010e350 0x7ffb501a5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5679f640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb50111540 con 0x7ffb50100520 2024-03-04T12:27:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5679f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb501113c0 con 0x7ffb501053f0 2024-03-04T12:27:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5679f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb501116c0 con 0x7ffb5010e350 2024-03-04T12:27:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5579d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb50100520 0x7ffb5010d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5579d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb50100520 0x7ffb5010d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48314/0 (socket says 172.21.15.60:48314) 2024-03-04T12:27:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5579d640 1 -- 172.21.15.60:0/2283500282 learned_addr learned my addr 172.21.15.60:0/2283500282 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb55f9e640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb5010e350 0x7ffb501a5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb501053f0 0x7ffb5010de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5579d640 1 -- 172.21.15.60:0/2283500282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb5010e350 msgr2=0x7ffb501a5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5579d640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb5010e350 0x7ffb501a5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5579d640 1 -- 172.21.15.60:0/2283500282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb501053f0 msgr2=0x7ffb5010de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.085+0000 7ffb5579d640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb501053f0 0x7ffb5010de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.086+0000 7ffb5579d640 1 -- 172.21.15.60:0/2283500282 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb501a62f0 con 0x7ffb50100520 2024-03-04T12:27:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.086+0000 7ffb55f9e640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb5010e350 0x7ffb501a5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.086+0000 7ffb5579d640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb50100520 0x7ffb5010d8d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009430 tx=0x7ffb4c02cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.086+0000 7ffb467fc640 1 -- 172.21.15.60:0/2283500282 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c041070 con 0x7ffb50100520 2024-03-04T12:27:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.086+0000 7ffb5679f640 1 -- 172.21.15.60:0/2283500282 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb501a6580 con 0x7ffb50100520 2024-03-04T12:27:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.086+0000 7ffb467fc640 1 -- 172.21.15.60:0/2283500282 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb4c004450 con 0x7ffb50100520 2024-03-04T12:27:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.086+0000 7ffb467fc640 1 -- 172.21.15.60:0/2283500282 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c003df0 con 0x7ffb50100520 2024-03-04T12:27:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.086+0000 7ffb5679f640 1 -- 172.21.15.60:0/2283500282 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb501a6ac0 con 0x7ffb50100520 2024-03-04T12:27:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.089+0000 7ffb5a5db740 1 Processor -- start 2024-03-04T12:27:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.089+0000 7ffb5a5db740 1 -- start start 2024-03-04T12:27:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.089+0000 7ffb5a5db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 0x55b21a586ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.089+0000 7ffb5a5db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a587530 0x55b21a5879b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.089+0000 7ffb5a5db740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a590ab0 0x55b21a592ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.089+0000 7ffb5a5db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21a4eb5f0 con 0x55b21a5882a0 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.089+0000 7ffb5a5db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21a494930 con 0x55b21a587530 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.089+0000 7ffb5a5db740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21a4e7690 con 0x55b21a590ab0 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.090+0000 7ffb55f9e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 0x55b21a586ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.090+0000 7ffb55f9e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 0x55b21a586ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48326/0 (socket says 172.21.15.60:48326) 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.090+0000 7ffb55f9e640 1 -- 172.21.15.60:0/4287250480 learned_addr learned my addr 172.21.15.60:0/4287250480 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.090+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/4287250480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a590ab0 0x55b21a592ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.090+0000 7ffb55f9e640 1 -- 172.21.15.60:0/4287250480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a590ab0 msgr2=0x55b21a592ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.090+0000 7ffb55f9e640 1 --2- 172.21.15.60:0/4287250480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a590ab0 0x55b21a592ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.090+0000 7ffb55f9e640 1 -- 172.21.15.60:0/4287250480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a587530 msgr2=0x55b21a5879b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.090+0000 7ffb55f9e640 1 --2- 172.21.15.60:0/4287250480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a587530 0x55b21a5879b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.090+0000 7ffb55f9e640 1 -- 172.21.15.60:0/4287250480 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b21a200210 con 0x55b21a5882a0 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.090+0000 7ffb467fc640 1 -- 172.21.15.60:0/2283500282 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7ffb4c044070 con 0x7ffb50100520 2024-03-04T12:27:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.090+0000 7ffb55f9e640 1 --2- 172.21.15.60:0/4287250480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 0x55b21a586ff0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ffb480107a0 tx=0x7ffb48010c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c78165cae2a79ed server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.090+0000 7ffb3b7fe640 1 -- 172.21.15.60:0/4287250480 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb48003b70 con 0x55b21a5882a0 2024-03-04T12:27:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.091+0000 7ffb3b7fe640 1 -- 172.21.15.60:0/4287250480 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb48010e50 con 0x55b21a5882a0 2024-03-04T12:27:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.091+0000 7ffb3b7fe640 1 -- 172.21.15.60:0/4287250480 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb48004680 con 0x55b21a5882a0 2024-03-04T12:27:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.091+0000 7ffb467fc640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ffb3c0a2020 0x7ffb3c0a44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.091+0000 7ffb5579d640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ffb3c0a2020 0x7ffb3c0a44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.091+0000 7ffb467fc640 1 -- 172.21.15.60:0/2283500282 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7ffb4c0f0220 con 0x7ffb50100520 2024-03-04T12:27:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.091+0000 7ffb5a5db740 1 -- 172.21.15.60:0/4287250480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 msgr2=0x55b21a586ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.091+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/4287250480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 0x55b21a586ff0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7ffb480107a0 tx=0x7ffb48010c60 comp rx=0 tx=0).stop 2024-03-04T12:27:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.092+0000 7ffb5a5db740 1 -- 172.21.15.60:0/4287250480 shutdown_connections 2024-03-04T12:27:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.092+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/4287250480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a590ab0 0x55b21a592ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.092+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/4287250480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a587530 0x55b21a5879b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.092+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/4287250480 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 0x55b21a586ff0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.092+0000 7ffb5a5db740 1 -- 172.21.15.60:0/4287250480 >> 172.21.15.60:0/4287250480 conn(0x55b21a54eb90 msgr2=0x55b21a582220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.092+0000 7ffb5a5db740 1 -- 172.21.15.60:0/4287250480 shutdown_connections 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.092+0000 7ffb5579d640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ffb3c0a2020 0x7ffb3c0a44e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0ed5c0 tx=0x7ffb4c0b83d0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.092+0000 7ffb5a5db740 1 -- 172.21.15.60:0/4287250480 wait complete. 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.092+0000 7ffb5a5db740 1 -- start start 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.092+0000 7ffb5a5db740 1 Processor -- start 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.093+0000 7ffb5a5db740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a587530 0x55b21a58db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.093+0000 7ffb5a5db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a5882a0 0x55b21a58e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.093+0000 7ffb5a5db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a590ab0 0x55b21a58e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.093+0000 7ffb55f9e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a587530 0x55b21a58db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.093+0000 7ffb55f9e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a587530 0x55b21a58db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:35768/0 (socket says 172.21.15.60:35768) 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.093+0000 7ffb55f9e640 1 -- 172.21.15.60:0/2108772261 learned_addr learned my addr 172.21.15.60:0/2108772261 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.093+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2108772261 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21a5937e0 con 0x55b21a590ab0 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.093+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a5882a0 0x55b21a58e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.093+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2108772261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21a4a7d90 con 0x55b21a5882a0 2024-03-04T12:27:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.093+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2108772261 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21a593960 con 0x55b21a587530 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.093+0000 7ffb55f9e640 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a590ab0 0x55b21a58e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb54f9c640 1 -- 172.21.15.60:0/2108772261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a587530 msgr2=0x55b21a58db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a587530 0x55b21a58db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb54f9c640 1 -- 172.21.15.60:0/2108772261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a590ab0 msgr2=0x55b21a58e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a590ab0 0x55b21a58e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb54f9c640 1 -- 172.21.15.60:0/2108772261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b21a2cc620 con 0x55b21a5882a0 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a5882a0 0x55b21a58e0d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55b21a587fd0 tx=0x7ffb40012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7e04f2dcf5eeba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb5a5db740 10 client.0 _ll_register_callbacks cb 0x55b21a1ffa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2108772261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb48010e50 con 0x55b21a5882a0 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb38ff9640 1 -- 172.21.15.60:0/2108772261 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4000f8f0 con 0x55b21a5882a0 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb38ff9640 1 -- 172.21.15.60:0/2108772261 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7ffb4000fa90 con 0x55b21a5882a0 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb38ff9640 1 -- 172.21.15.60:0/2108772261 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb4000fc90 con 0x55b21a5882a0 2024-03-04T12:27:01.106 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb5a5db740 10 client.25655 fetch_fsmap learned FSMap version 29 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.094+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2108772261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({fsmap=29}) v3 -- 0x55b21a59fa20 con 0x55b21a5882a0 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.095+0000 7ffb38ff9640 1 -- 172.21.15.60:0/2108772261 <== mon.2 v2:172.21.15.63:3300/0 4 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ffb40005020 con 0x55b21a5882a0 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.095+0000 7ffb5a5db740 10 client.25655 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.095+0000 7ffb5a5db740 10 client.25655 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.095+0000 7ffb5a5db740 20 client.25655 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.095+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55b21a5a0e90 0x55b21a5a3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.095+0000 7ffb5a5db740 4 client.25655 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.095+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2108772261 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b21a4f8b30 con 0x55b21a5a0e90 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.095+0000 7ffb55f9e640 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55b21a5a0e90 0x55b21a5a3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.095+0000 7ffb38ff9640 10 client.25655 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.095+0000 7ffb55f9e640 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55b21a5a0e90 0x55b21a5a3300 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=526d07891e60a3bf server_cookie=930ad04e89bac490 in_seq=0 out_seq=0 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.096+0000 7ffb38ff9640 1 -- 172.21.15.60:0/2108772261 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b21a4f8b30 con 0x55b21a5a0e90 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.096+0000 7ffb38ff9640 10 client.25655 handle_command_reply: tid=0 2024-03-04T12:27:01.107 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.096+0000 7ffb5a5db740 1 client.25655 shutdown 2024-03-04T12:27:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.096+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2108772261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a5882a0 msgr2=0x55b21a58e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.096+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a5882a0 0x55b21a58e0d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55b21a587fd0 tx=0x7ffb40012b60 comp rx=0 tx=0).stop 2024-03-04T12:27:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.096+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2108772261 shutdown_connections 2024-03-04T12:27:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.096+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55b21a5a0e90 0x55b21a5a3300 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.096+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a590ab0 0x55b21a58e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.096+0000 7ffb38ff9640 0 client.25655 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.096+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a5882a0 0x55b21a58e0d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.096+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/2108772261 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a587530 0x55b21a58db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.096+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2108772261 >> 172.21.15.60:0/2108772261 conn(0x55b21a54eb90 msgr2=0x55b21a583f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.097+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2108772261 shutdown_connections 2024-03-04T12:27:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.097+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2108772261 wait complete. 2024-03-04T12:27:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.097+0000 7ffb5a5db740 20 client.25655 trim_cache size 0 max 16384 2024-03-04T12:27:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.123+0000 7ffb5a5db740 1 Processor -- start 2024-03-04T12:27:01.131 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.123+0000 7ffb5a5db740 1 -- start start 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb5a5db740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a587530 0x55b21a583d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb5a5db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 0x55b21a597c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb5a5db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a590ab0 0x55b21a598150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb54f9c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 0x55b21a597c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb55f9e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a587530 0x55b21a583d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb54f9c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 0x55b21a597c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48340/0 (socket says 172.21.15.60:48340) 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb54f9c640 1 -- 172.21.15.60:0/1066679835 learned_addr learned my addr 172.21.15.60:0/1066679835 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb5a5db740 1 -- 172.21.15.60:0/1066679835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb501b03f0 con 0x55b21a5882a0 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb5a5db740 1 -- 172.21.15.60:0/1066679835 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb501b1d70 con 0x55b21a590ab0 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb5a5db740 1 -- 172.21.15.60:0/1066679835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb501b0a50 con 0x55b21a587530 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb55f9e640 1 --2- 172.21.15.60:0/1066679835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a590ab0 0x55b21a598150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb54f9c640 1 -- 172.21.15.60:0/1066679835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a587530 msgr2=0x55b21a583d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/1066679835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a587530 0x55b21a583d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.132 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb54f9c640 1 -- 172.21.15.60:0/1066679835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a590ab0 msgr2=0x55b21a598150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:01.133 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/1066679835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a590ab0 0x55b21a598150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.133 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.124+0000 7ffb54f9c640 1 -- 172.21.15.60:0/1066679835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b21a59d5f0 con 0x55b21a5882a0 2024-03-04T12:27:01.133 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.125+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/1066679835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 0x55b21a597c10 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ffb4000a010 tx=0x7ffb40014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6ca0301963c7f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:01.133 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.125+0000 7ffb3affd640 1 -- 172.21.15.60:0/1066679835 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4000fe30 con 0x55b21a5882a0 2024-03-04T12:27:01.133 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.125+0000 7ffb3affd640 1 -- 172.21.15.60:0/1066679835 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb48010e50 con 0x55b21a5882a0 2024-03-04T12:27:01.133 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.125+0000 7ffb3affd640 1 -- 172.21.15.60:0/1066679835 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb40003970 con 0x55b21a5882a0 2024-03-04T12:27:01.136 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.126+0000 7ffb5a5db740 1 -- 172.21.15.60:0/1066679835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 msgr2=0x55b21a597c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.126+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/1066679835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 0x55b21a597c10 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7ffb4000a010 tx=0x7ffb40014000 comp rx=0 tx=0).stop 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.126+0000 7ffb5a5db740 1 -- 172.21.15.60:0/1066679835 shutdown_connections 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.126+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/1066679835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a590ab0 0x55b21a598150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.126+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/1066679835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a5882a0 0x55b21a597c10 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.126+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/1066679835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a587530 0x55b21a583d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.126+0000 7ffb5a5db740 1 -- 172.21.15.60:0/1066679835 >> 172.21.15.60:0/1066679835 conn(0x55b21a54eb90 msgr2=0x55b21a583670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.126+0000 7ffb5a5db740 1 -- 172.21.15.60:0/1066679835 shutdown_connections 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.126+0000 7ffb5a5db740 1 -- 172.21.15.60:0/1066679835 wait complete. 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.126+0000 7ffb5a5db740 1 -- start start 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.126+0000 7ffb5a5db740 1 Processor -- start 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb5a5db740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a587530 0x55b21a596980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb5a5db740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a5882a0 0x55b21a596ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.137 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb5a5db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a590ab0 0x55b21a5f7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb5a5db740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb501b1710 con 0x55b21a587530 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb5a5db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb501b10b0 con 0x55b21a590ab0 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb55f9e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a587530 0x55b21a596980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb5a5db740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb501ae690 con 0x55b21a5882a0 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb55f9e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a587530 0x55b21a596980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:48352/0 (socket says 172.21.15.60:48352) 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb55f9e640 1 -- 172.21.15.60:0/2394192213 learned_addr learned my addr 172.21.15.60:0/2394192213 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a5882a0 0x55b21a596ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb5a5db740 10 client.0 _ll_register_callbacks cb 0x7ffb40009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb54f9c640 1 -- 172.21.15.60:0/2394192213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a590ab0 msgr2=0x55b21a5f7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.127+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a590ab0 0x55b21a5f7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.128+0000 7ffb54f9c640 1 -- 172.21.15.60:0/2394192213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a587530 msgr2=0x55b21a596980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.128+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a587530 0x55b21a596980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.128+0000 7ffb54f9c640 1 -- 172.21.15.60:0/2394192213 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b21a597780 con 0x55b21a5882a0 2024-03-04T12:27:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.128+0000 7ffb55f9e640 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a587530 0x55b21a596980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.128+0000 7ffb54f9c640 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a5882a0 0x55b21a596ec0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55b21a58ed20 tx=0x7ffb40009e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2ef75dc428a6a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.128+0000 7ffb38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.128+0000 7ffb38ff9640 1 -- 172.21.15.60:0/2394192213 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4000f8f0 con 0x55b21a5882a0 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.128+0000 7ffb38ff9640 1 -- 172.21.15.60:0/2394192213 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9451+0+0 (secure 0 0 0) 0x7ffb400035f0 con 0x55b21a5882a0 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.128+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2394192213 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb48010e50 con 0x55b21a5882a0 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.128+0000 7ffb38ff9640 1 -- 172.21.15.60:0/2394192213 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb4003c050 con 0x55b21a5882a0 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.128+0000 7ffb5a5db740 10 client.25735 fetch_fsmap learned FSMap version 29 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.128+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2394192213 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b21a5f8a10 con 0x55b21a5882a0 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.129+0000 7ffb38ff9640 1 -- 172.21.15.60:0/2394192213 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb40003cf0 con 0x55b21a5882a0 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.130+0000 7ffb38ff9640 1 -- 172.21.15.60:0/2394192213 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 29) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ffb4001c940 con 0x55b21a5882a0 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.130+0000 7ffb5a5db740 10 client.25735 fetch_fsmap finished waiting for FSMap version 29 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.130+0000 7ffb5a5db740 10 client.25735 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.130+0000 7ffb5a5db740 20 client.25735 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.130+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55b21a5a0e90 0x55b21a5fafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.130+0000 7ffb5a5db740 4 client.25735 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T12:27:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.130+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2394192213 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7ffb48010e50 con 0x55b21a5a0e90 2024-03-04T12:27:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.130+0000 7ffb55f9e640 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55b21a5a0e90 0x55b21a5fafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.131+0000 7ffb38ff9640 10 client.25735 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.131+0000 7ffb55f9e640 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55b21a5a0e90 0x55b21a5fafd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=541c4d814bd908cf server_cookie=1c714341ae2895e8 in_seq=0 out_seq=0 2024-03-04T12:27:01.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:01 smithi060 ceph-mon[32745]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-04T12:27:01.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:01 smithi086 ceph-mon[36851]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-04T12:27:01.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:01 smithi063 ceph-mon[36717]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 255 B/s wr, 13 op/s 2024-03-04T12:27:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.613+0000 7ffb38ff9640 1 -- 172.21.15.60:0/2394192213 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7ffb48010e50 con 0x55b21a5a0e90 2024-03-04T12:27:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.613+0000 7ffb38ff9640 10 client.25735 handle_command_reply: tid=0 2024-03-04T12:27:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.613+0000 7ffb5a5db740 1 client.25735 shutdown 2024-03-04T12:27:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.614+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2394192213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a5882a0 msgr2=0x55b21a596ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.614+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a5882a0 0x55b21a596ec0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55b21a58ed20 tx=0x7ffb40009e80 comp rx=0 tx=0).stop 2024-03-04T12:27:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.614+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2394192213 shutdown_connections 2024-03-04T12:27:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.614+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55b21a5a0e90 0x55b21a5fafd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.614+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b21a590ab0 0x55b21a5f7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.614+0000 7ffb38ff9640 0 client.25735 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.614+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b21a5882a0 0x55b21a596ec0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.614+0000 7ffb5a5db740 1 --2- 172.21.15.60:0/2394192213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b21a587530 0x55b21a596980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.614+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2394192213 >> 172.21.15.60:0/2394192213 conn(0x55b21a54eb90 msgr2=0x55b21a583e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.614+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2394192213 shutdown_connections 2024-03-04T12:27:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.614+0000 7ffb5a5db740 1 -- 172.21.15.60:0/2394192213 wait complete. 2024-03-04T12:27:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.614+0000 7ffb5a5db740 20 client.25735 trim_cache size 0 max 16384 2024-03-04T12:27:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:{ 2024-03-04T12:27:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "message": "", 2024-03-04T12:27:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "return_code": 0 2024-03-04T12:27:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:} 2024-03-04T12:27:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.615+0000 7ffb5679f640 1 -- 172.21.15.60:0/2283500282 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ffb3c0a2020 msgr2=0x7ffb3c0a44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.615+0000 7ffb5679f640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ffb3c0a2020 0x7ffb3c0a44e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0ed5c0 tx=0x7ffb4c0b83d0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.615+0000 7ffb5679f640 1 -- 172.21.15.60:0/2283500282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb50100520 msgr2=0x7ffb5010d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.615+0000 7ffb5679f640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb50100520 0x7ffb5010d8d0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009430 tx=0x7ffb4c02cbe0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.615+0000 7ffb5679f640 1 -- 172.21.15.60:0/2283500282 shutdown_connections 2024-03-04T12:27:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.615+0000 7ffb5679f640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb5010e350 0x7ffb501a5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.615+0000 7ffb5679f640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb501053f0 0x7ffb5010de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.615+0000 7ffb5679f640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ffb3c0a2020 0x7ffb3c0a44e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.615+0000 7ffb5679f640 1 --2- 172.21.15.60:0/2283500282 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffb50100520 0x7ffb5010d8d0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.615+0000 7ffb5679f640 1 -- 172.21.15.60:0/2283500282 >> 172.21.15.60:0/2283500282 conn(0x7ffb50075b90 msgr2=0x7ffb500fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.615+0000 7ffb5679f640 1 -- 172.21.15.60:0/2283500282 shutdown_connections 2024-03-04T12:27:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:01.615+0000 7ffb5679f640 1 -- 172.21.15.60:0/2283500282 wait complete. 2024-03-04T12:27:02.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:02 smithi060 ceph-mon[32745]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-04T12:27:02.581 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-mon[36851]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-04T12:27:02.581 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: *** Caught signal (Segmentation fault) ** 2024-03-04T12:27:02.582 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: in thread 7f4be6c58700 thread_name:md_log_replay 2024-03-04T12:27:02.582 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T12:27:02.582 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4bf8019d20] 2024-03-04T12:27:02.582 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 2: /usr/bin/ceph-mds(+0x503e0f) [0x55ab949fce0f] 2024-03-04T12:27:02.582 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55ab94a00683] 2024-03-04T12:27:02.582 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 4: (MDLog::_replay_thread()+0x6a9) [0x55ab94a5c049] 2024-03-04T12:27:02.582 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55ab94a65481] 2024-03-04T12:27:02.582 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f4bf800f1ca] 2024-03-04T12:27:02.582 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 7: clone() 2024-03-04T12:27:02.582 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 2024-03-04T12:27:02.193+0000 7f4be6c58700 -1 *** Caught signal (Segmentation fault) ** 2024-03-04T12:27:02.582 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: in thread 7f4be6c58700 thread_name:md_log_replay 2024-03-04T12:27:02.582 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4bf8019d20] 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 2: /usr/bin/ceph-mds(+0x503e0f) [0x55ab949fce0f] 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55ab94a00683] 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 4: (MDLog::_replay_thread()+0x6a9) [0x55ab94a5c049] 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55ab94a65481] 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f4bf800f1ca] 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 7: clone() 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 0> 2024-03-04T12:27:02.193+0000 7f4be6c58700 -1 *** Caught signal (Segmentation fault) ** 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: in thread 7f4be6c58700 thread_name:md_log_replay 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T12:27:02.583 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4bf8019d20] 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 2: /usr/bin/ceph-mds(+0x503e0f) [0x55ab949fce0f] 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55ab94a00683] 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 4: (MDLog::_replay_thread()+0x6a9) [0x55ab94a5c049] 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55ab94a65481] 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f4bf800f1ca] 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 7: clone() 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 0> 2024-03-04T12:27:02.193+0000 7f4be6c58700 -1 *** Caught signal (Segmentation fault) ** 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: in thread 7f4be6c58700 thread_name:md_log_replay 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: ceph version 19.0.0-1845-g717ce3cc (717ce3cce8d0166e4e577faa8004e6f9cb4128c0) squid (dev) 2024-03-04T12:27:02.584 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4bf8019d20] 2024-03-04T12:27:02.585 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 2: /usr/bin/ceph-mds(+0x503e0f) [0x55ab949fce0f] 2024-03-04T12:27:02.585 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x55ab94a00683] 2024-03-04T12:27:02.585 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 4: (MDLog::_replay_thread()+0x6a9) [0x55ab94a5c049] 2024-03-04T12:27:02.585 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 5: (MDLog::ReplayThread::entry()+0x11) [0x55ab94a65481] 2024-03-04T12:27:02.585 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f4bf800f1ca] 2024-03-04T12:27:02.585 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 7: clone() 2024-03-04T12:27:02.585 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T12:27:02.585 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[67841]: 2024-03-04T12:27:02.585 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 podman[71377]: 2024-03-04 12:27:02.255919139 +0000 UTC m=+0.022190853 container died b34586918c54540155abd2ad19e3955c071007f655113f2e1e8841a7108b8b04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-04T12:27:02.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:02 smithi063 ceph-mon[36717]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-04T12:27:03.349 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 podman[71377]: 2024-03-04 12:27:02.958865077 +0000 UTC m=+0.725136853 container remove b34586918c54540155abd2ad19e3955c071007f655113f2e1e8841a7108b8b04 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:27:03.349 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:02 smithi086 systemd[1]: ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.c.service: Main process exited, code=exited, status=139/n/a 2024-03-04T12:27:03.349 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:03 smithi086 systemd[1]: ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.c.service: Failed with result 'exit-code'. 2024-03-04T12:27:04.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:04 smithi086 ceph-mon[36851]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-04T12:27:04.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:04 smithi063 ceph-mon[36717]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-04T12:27:04.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:04 smithi060 ceph-mon[32745]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-04T12:27:06.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:06 smithi086 ceph-mon[36851]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-04T12:27:06.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:06 smithi063 ceph-mon[36717]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-04T12:27:06.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:06 smithi060 ceph-mon[32745]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-04T12:27:08.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:08 smithi086 ceph-mon[36851]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-04T12:27:08.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:08 smithi063 ceph-mon[36717]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-04T12:27:08.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:08 smithi060 ceph-mon[32745]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-04T12:27:10.302 INFO:tasks.workunit.client.0.smithi060.stderr:Updating files: 97% (11906/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-04T12:27:10.657 INFO:tasks.workunit.client.0.smithi060.stderr:Note: switching to '717ce3cce8d0166e4e577faa8004e6f9cb4128c0'. 2024-03-04T12:27:10.657 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:10.657 INFO:tasks.workunit.client.0.smithi060.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-04T12:27:10.657 INFO:tasks.workunit.client.0.smithi060.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr:state without impacting any branches by switching back to a branch. 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr: git switch -c 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr:Or undo this operation with: 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr: git switch - 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:10.658 INFO:tasks.workunit.client.0.smithi060.stderr:HEAD is now at 717ce3cce8d Merge branch 'wip-64616' into wip-vshankar-testing-20240304.042522 2024-03-04T12:27:10.661 DEBUG:teuthology.orchestra.run.smithi060:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-04T12:27:10.696 INFO:tasks.workunit.client.0.smithi060.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-04T12:27:10.698 INFO:tasks.workunit.client.0.smithi060.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-04T12:27:10.699 INFO:tasks.workunit.client.0.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-04T12:27:11.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:10 smithi086 ceph-mon[36851]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-04T12:27:11.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:10 smithi063 ceph-mon[36717]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-04T12:27:11.382 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:10 smithi060 ceph-mon[32745]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-03-04T12:27:11.455 INFO:tasks.workunit.client.0.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-04T12:27:11.535 INFO:tasks.workunit.client.0.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-04T12:27:11.576 INFO:tasks.workunit.client.0.smithi060.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-04T12:27:11.577 INFO:tasks.workunit.client.0.smithi060.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-04T12:27:11.577 INFO:tasks.workunit.client.0.smithi060.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-04T12:27:11.617 INFO:tasks.workunit.client.0.smithi060.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-04T12:27:11.620 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:27:11.620 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-04T12:27:11.675 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-03-04T12:27:11.676 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-03-04T12:27:11.677 DEBUG:teuthology.orchestra.run.smithi060:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-03-04T12:27:11.735 INFO:tasks.workunit.client.0.smithi060.stderr:+ set -e 2024-03-04T12:27:11.736 INFO:tasks.workunit.client.0.smithi060.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-03-04T12:27:11.738 INFO:tasks.workunit.client.0.smithi060.stderr:Cloning into 'xfstests-dev'... 2024-03-04T12:27:13.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:12 smithi086 ceph-mon[36851]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-04T12:27:13.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:12 smithi063 ceph-mon[36717]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-04T12:27:13.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:12 smithi060 ceph-mon[32745]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-04T12:27:13.724 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:13 smithi086 systemd[1]: ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-03-04T12:27:13.727 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:13 smithi086 systemd[1]: Stopped Ceph mds.c for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:27:13.727 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:13 smithi086 systemd[1]: Starting Ceph mds.c for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:27:13.727 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:13 smithi086 podman[71472]: 2024-03-04 12:27:13.479319295 +0000 UTC m=+0.018053414 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:27:13.727 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:13 smithi086 podman[71472]: 2024-03-04 12:27:13.593018713 +0000 UTC m=+0.131752868 container create 5bb45a38feef9804a6b74f7cb5a8bf4421818c7d63e8291be68a3129c32eb859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:27:14.099 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:13 smithi086 ceph-mon[36851]: daemon mds.c restarted 2024-03-04T12:27:14.099 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:13 smithi086 podman[71472]: 2024-03-04 12:27:13.829116914 +0000 UTC m=+0.367851045 container init 5bb45a38feef9804a6b74f7cb5a8bf4421818c7d63e8291be68a3129c32eb859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-04T12:27:14.099 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:13 smithi086 podman[71472]: 2024-03-04 12:27:13.831622373 +0000 UTC m=+0.370356495 container start 5bb45a38feef9804a6b74f7cb5a8bf4421818c7d63e8291be68a3129c32eb859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-04T12:27:14.100 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:13 smithi086 ceph-mds[71492]: -- 172.21.15.86:0/3627865039 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cb9f5ada40 con 0x55cb9f580800 2024-03-04T12:27:14.100 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:13 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[71488]: starting mds.c at 2024-03-04T12:27:14.100 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:13 smithi086 bash[71472]: 5bb45a38feef9804a6b74f7cb5a8bf4421818c7d63e8291be68a3129c32eb859 2024-03-04T12:27:14.100 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:27:13 smithi086 systemd[1]: Started Ceph mds.c for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:27:14.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:13 smithi063 ceph-mon[36717]: daemon mds.c restarted 2024-03-04T12:27:14.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:13 smithi060 ceph-mon[32745]: daemon mds.c restarted 2024-03-04T12:27:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.716+0000 7f7b41080640 1 -- 172.21.15.60:0/677936650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b3c101570 msgr2=0x7f7b3c101950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.716+0000 7f7b41080640 1 --2- 172.21.15.60:0/677936650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b3c101570 0x7f7b3c101950 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009900 tx=0x7f7b2c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.716+0000 7f7b41080640 1 -- 172.21.15.60:0/677936650 shutdown_connections 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.716+0000 7f7b41080640 1 --2- 172.21.15.60:0/677936650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b3c106640 0x7f7b3c10e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.716+0000 7f7b41080640 1 --2- 172.21.15.60:0/677936650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b3c101f20 0x7f7b3c105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.716+0000 7f7b41080640 1 --2- 172.21.15.60:0/677936650 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b3c101570 0x7f7b3c101950 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.716+0000 7f7b41080640 1 -- 172.21.15.60:0/677936650 >> 172.21.15.60:0/677936650 conn(0x7f7b3c0fce50 msgr2=0x7f7b3c0ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.716+0000 7f7b41080640 1 -- 172.21.15.60:0/677936650 shutdown_connections 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.716+0000 7f7b41080640 1 -- 172.21.15.60:0/677936650 wait complete. 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.716+0000 7f7b41080640 1 Processor -- start 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.716+0000 7f7b41080640 1 -- start start 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b41080640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b3c101570 0x7f7b3c073630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b41080640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b3c106640 0x7f7b3c073b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b41080640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b3c0740b0 0x7f7b3c1a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b41080640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c111740 con 0x7f7b3c106640 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b41080640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1115c0 con 0x7f7b3c101570 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b41080640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1118c0 con 0x7f7b3c0740b0 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b3c106640 0x7f7b3c073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b3c106640 0x7f7b3c073b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49622/0 (socket says 172.21.15.60:49622) 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b33fff640 1 -- 172.21.15.60:0/2829179878 learned_addr learned my addr 172.21.15.60:0/2829179878 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b3b59e640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b3c0740b0 0x7f7b3c1a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b3ad9d640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b3c101570 0x7f7b3c073630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b33fff640 1 -- 172.21.15.60:0/2829179878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b3c0740b0 msgr2=0x7f7b3c1a5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b33fff640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b3c0740b0 0x7f7b3c1a5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b33fff640 1 -- 172.21.15.60:0/2829179878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b3c101570 msgr2=0x7f7b3c073630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b33fff640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b3c101570 0x7f7b3c073630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b33fff640 1 -- 172.21.15.60:0/2829179878 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3c1a64f0 con 0x7f7b3c106640 2024-03-04T12:27:14.727 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.717+0000 7f7b33fff640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b3c106640 0x7f7b3c073b70 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7b2400e9f0 tx=0x7f7b2400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.718+0000 7f7b38d99640 1 -- 172.21.15.60:0/2829179878 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2400ccc0 con 0x7f7b3c106640 2024-03-04T12:27:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.718+0000 7f7b41080640 1 -- 172.21.15.60:0/2829179878 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3c1a67e0 con 0x7f7b3c106640 2024-03-04T12:27:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.718+0000 7f7b38d99640 1 -- 172.21.15.60:0/2829179878 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b24005500 con 0x7f7b3c106640 2024-03-04T12:27:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.718+0000 7f7b38d99640 1 -- 172.21.15.60:0/2829179878 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b24010690 con 0x7f7b3c106640 2024-03-04T12:27:14.728 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.718+0000 7f7b41080640 1 -- 172.21.15.60:0/2829179878 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3c1a6d30 con 0x7f7b3c106640 2024-03-04T12:27:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.720+0000 7f7b38d99640 1 -- 172.21.15.60:0/2829179878 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f7b240046c0 con 0x7f7b3c106640 2024-03-04T12:27:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.720+0000 7f7b43c5b740 1 Processor -- start 2024-03-04T12:27:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.720+0000 7f7b43c5b740 1 -- start start 2024-03-04T12:27:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b38d99640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f7b280a2000 0x7f7b280a44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b43c5b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46310d0 0x55a8d462fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b43c5b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 0x55a8d46307e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b43c5b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46398e0 0x55a8d463bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b43c5b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8d45945f0 con 0x55a8d4630360 2024-03-04T12:27:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b43c5b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8d453d930 con 0x55a8d46398e0 2024-03-04T12:27:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b43c5b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8d4590690 con 0x55a8d46310d0 2024-03-04T12:27:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b3b59e640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f7b280a2000 0x7f7b280a44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.735 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 0x55a8d46307e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 0x55a8d46307e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49624/0 (socket says 172.21.15.60:49624) 2024-03-04T12:27:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b33fff640 1 -- 172.21.15.60:0/3350338046 learned_addr learned my addr 172.21.15.60:0/3350338046 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b38d99640 1 -- 172.21.15.60:0/2829179878 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f7b240cec40 con 0x7f7b3c106640 2024-03-04T12:27:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b3ad9d640 1 --2- 172.21.15.60:0/3350338046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46310d0 0x55a8d462fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b3b59e640 1 --2- 172.21.15.60:0/3350338046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46398e0 0x55a8d463bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b33fff640 1 -- 172.21.15.60:0/3350338046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46310d0 msgr2=0x55a8d462fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b33fff640 1 --2- 172.21.15.60:0/3350338046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46310d0 0x55a8d462fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b33fff640 1 -- 172.21.15.60:0/3350338046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46398e0 msgr2=0x55a8d463bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b33fff640 1 --2- 172.21.15.60:0/3350338046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46398e0 0x55a8d463bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.736 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b33fff640 1 -- 172.21.15.60:0/3350338046 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8d42a9210 con 0x55a8d4630360 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b3ad9d640 1 --2- 172.21.15.60:0/3350338046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46310d0 0x55a8d462fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.721+0000 7f7b3b59e640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f7b280a2000 0x7f7b280a44c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7b34005fd0 tx=0x7f7b34009e60 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.722+0000 7f7b33fff640 1 --2- 172.21.15.60:0/3350338046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 0x55a8d46307e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7b2409ad40 tx=0x7f7b2409ae10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f88682c6adac1f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.722+0000 7f7b317fa640 1 -- 172.21.15.60:0/3350338046 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2409b740 con 0x55a8d4630360 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.722+0000 7f7b317fa640 1 -- 172.21.15.60:0/3350338046 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b2409b040 con 0x55a8d4630360 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.722+0000 7f7b317fa640 1 -- 172.21.15.60:0/3350338046 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b240c9f70 con 0x55a8d4630360 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.722+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3350338046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 msgr2=0x55a8d46307e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.722+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/3350338046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 0x55a8d46307e0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7b2409ad40 tx=0x7f7b2409ae10 comp rx=0 tx=0).stop 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3350338046 shutdown_connections 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/3350338046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46398e0 0x55a8d463bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/3350338046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 0x55a8d46307e0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/3350338046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46310d0 0x55a8d462fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3350338046 >> 172.21.15.60:0/3350338046 conn(0x55a8d45f7b90 msgr2=0x55a8d462b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3350338046 shutdown_connections 2024-03-04T12:27:14.737 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3350338046 wait complete. 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 -- start start 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 Processor -- start 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d4630360 0x55a8d4636860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46310d0 0x55a8d4636da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46398e0 0x55a8d46372e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8d463c5b0 con 0x55a8d46398e0 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8d4550d90 con 0x55a8d4630360 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8d463c730 con 0x55a8d46310d0 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b43c5b740 10 client.0 _ll_register_callbacks cb 0x55a8d42a8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b3b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46398e0 0x55a8d46372e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.723+0000 7f7b3b59e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46398e0 0x55a8d46372e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49632/0 (socket says 172.21.15.60:49632) 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b3b59e640 1 -- 172.21.15.60:0/807514455 learned_addr learned my addr 172.21.15.60:0/807514455 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b3ad9d640 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d4630360 0x55a8d4636860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.738 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b33fff640 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46310d0 0x55a8d4636da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b3b59e640 1 -- 172.21.15.60:0/807514455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46310d0 msgr2=0x55a8d4636da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b3b59e640 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46310d0 0x55a8d4636da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b3b59e640 1 -- 172.21.15.60:0/807514455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d4630360 msgr2=0x55a8d4636860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b3b59e640 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d4630360 0x55a8d4636860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b3b59e640 1 -- 172.21.15.60:0/807514455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a8d4375620 con 0x55a8d46398e0 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b3b59e640 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46398e0 0x55a8d46372e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7b34003bd0 tx=0x7f7b34004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81eb1350d90b615 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b227fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b227fc640 1 -- 172.21.15.60:0/807514455 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34026020 con 0x55a8d46398e0 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b227fc640 1 -- 172.21.15.60:0/807514455 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f7b34004780 con 0x55a8d46398e0 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b43c5b740 1 -- 172.21.15.60:0/807514455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b2409b040 con 0x55a8d46398e0 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b227fc640 1 -- 172.21.15.60:0/807514455 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b34031050 con 0x55a8d46398e0 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b43c5b740 10 client.16227 fetch_fsmap learned FSMap version 31 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.724+0000 7f7b43c5b740 1 -- 172.21.15.60:0/807514455 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55a8d46487d0 con 0x55a8d46398e0 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.725+0000 7f7b227fc640 1 -- 172.21.15.60:0/807514455 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b2409b040 con 0x55a8d46398e0 2024-03-04T12:27:14.739 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.725+0000 7f7b227fc640 1 -- 172.21.15.60:0/807514455 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7b3400f040 con 0x55a8d46398e0 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.725+0000 7f7b43c5b740 10 client.16227 fetch_fsmap finished waiting for FSMap version 31 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.725+0000 7f7b43c5b740 10 client.16227 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.725+0000 7f7b43c5b740 20 client.16227 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.725+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55a8d4649c90 0x55a8d464c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.725+0000 7f7b43c5b740 4 client.16227 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.725+0000 7f7b43c5b740 1 -- 172.21.15.60:0/807514455 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8d45a1b30 con 0x55a8d4649c90 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.725+0000 7f7b3ad9d640 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55a8d4649c90 0x55a8d464c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.725+0000 7f7b227fc640 10 client.16227 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.725+0000 7f7b3ad9d640 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55a8d4649c90 0x55a8d464c080 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e42ebe9be11b2ce server_cookie=4f39f4248eb097ea in_seq=0 out_seq=0 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b227fc640 1 -- 172.21.15.60:0/807514455 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a8d45a1b30 con 0x55a8d4649c90 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b227fc640 10 client.16227 handle_command_reply: tid=0 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b43c5b740 1 client.16227 shutdown 2024-03-04T12:27:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b43c5b740 1 -- 172.21.15.60:0/807514455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46398e0 msgr2=0x55a8d46372e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46398e0 0x55a8d46372e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7b34003bd0 tx=0x7f7b34004ab0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b43c5b740 1 -- 172.21.15.60:0/807514455 shutdown_connections 2024-03-04T12:27:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55a8d4649c90 0x55a8d464c080 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46398e0 0x55a8d46372e0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46310d0 0x55a8d4636da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/807514455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d4630360 0x55a8d4636860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b227fc640 0 client.16227 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b43c5b740 1 -- 172.21.15.60:0/807514455 >> 172.21.15.60:0/807514455 conn(0x55a8d45f7b90 msgr2=0x55a8d462cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b43c5b740 1 -- 172.21.15.60:0/807514455 shutdown_connections 2024-03-04T12:27:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.726+0000 7f7b43c5b740 1 -- 172.21.15.60:0/807514455 wait complete. 2024-03-04T12:27:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.727+0000 7f7b43c5b740 20 client.16227 trim_cache size 0 max 16384 2024-03-04T12:27:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b43c5b740 1 Processor -- start 2024-03-04T12:27:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b43c5b740 1 -- start start 2024-03-04T12:27:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b43c5b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d4630360 0x55a8d4638f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b43c5b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46310d0 0x55a8d4635130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b43c5b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46398e0 0x55a8d4635670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b43c5b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1b0710 con 0x55a8d46310d0 2024-03-04T12:27:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b43c5b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1b2090 con 0x55a8d46398e0 2024-03-04T12:27:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b43c5b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1b0d70 con 0x55a8d4630360 2024-03-04T12:27:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46310d0 0x55a8d4635130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b33fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46310d0 0x55a8d4635130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49642/0 (socket says 172.21.15.60:49642) 2024-03-04T12:27:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b33fff640 1 -- 172.21.15.60:0/2559406106 learned_addr learned my addr 172.21.15.60:0/2559406106 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:14.751 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b3ad9d640 1 --2- 172.21.15.60:0/2559406106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d4630360 0x55a8d4638f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b3b59e640 1 --2- 172.21.15.60:0/2559406106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46398e0 0x55a8d4635670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b33fff640 1 -- 172.21.15.60:0/2559406106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d4630360 msgr2=0x55a8d4638f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b33fff640 1 --2- 172.21.15.60:0/2559406106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d4630360 0x55a8d4638f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b33fff640 1 -- 172.21.15.60:0/2559406106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46398e0 msgr2=0x55a8d4635670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.740+0000 7f7b33fff640 1 --2- 172.21.15.60:0/2559406106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46398e0 0x55a8d4635670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.741+0000 7f7b33fff640 1 -- 172.21.15.60:0/2559406106 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8d462f630 con 0x55a8d46310d0 2024-03-04T12:27:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.741+0000 7f7b33fff640 1 --2- 172.21.15.60:0/2559406106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46310d0 0x55a8d4635130 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009380 tx=0x7f7b24004ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a61a1ce160cba7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.741+0000 7f7b30ff9640 1 -- 172.21.15.60:0/2559406106 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2409b740 con 0x55a8d46310d0 2024-03-04T12:27:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.741+0000 7f7b30ff9640 1 -- 172.21.15.60:0/2559406106 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b240a37e0 con 0x55a8d46310d0 2024-03-04T12:27:14.752 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.741+0000 7f7b30ff9640 1 -- 172.21.15.60:0/2559406106 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b24095bb0 con 0x55a8d46310d0 2024-03-04T12:27:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.741+0000 7f7b43c5b740 1 -- 172.21.15.60:0/2559406106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46310d0 msgr2=0x55a8d4635130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.741+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/2559406106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46310d0 0x55a8d4635130 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009380 tx=0x7f7b24004ba0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 -- 172.21.15.60:0/2559406106 shutdown_connections 2024-03-04T12:27:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/2559406106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46398e0 0x55a8d4635670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/2559406106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d46310d0 0x55a8d4635130 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/2559406106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d4630360 0x55a8d4638f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 -- 172.21.15.60:0/2559406106 >> 172.21.15.60:0/2559406106 conn(0x55a8d45f7b90 msgr2=0x55a8d46509d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 -- 172.21.15.60:0/2559406106 shutdown_connections 2024-03-04T12:27:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 -- 172.21.15.60:0/2559406106 wait complete. 2024-03-04T12:27:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 -- start start 2024-03-04T12:27:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 Processor -- start 2024-03-04T12:27:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 0x55a8d463f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46310d0 0x55a8d463f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46398e0 0x55a8d463fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1b1a30 con 0x55a8d4630360 2024-03-04T12:27:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1b13d0 con 0x55a8d46310d0 2024-03-04T12:27:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3c1ae9b0 con 0x55a8d46398e0 2024-03-04T12:27:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b43c5b740 10 client.0 _ll_register_callbacks cb 0x7f7b34012f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b3ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 0x55a8d463f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b3ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 0x55a8d463f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:49658/0 (socket says 172.21.15.60:49658) 2024-03-04T12:27:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b3ad9d640 1 -- 172.21.15.60:0/3856906710 learned_addr learned my addr 172.21.15.60:0/3856906710 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.742+0000 7f7b3b59e640 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46398e0 0x55a8d463fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b33fff640 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46310d0 0x55a8d463f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b3ad9d640 1 -- 172.21.15.60:0/3856906710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46398e0 msgr2=0x55a8d463fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b3ad9d640 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46398e0 0x55a8d463fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b3ad9d640 1 -- 172.21.15.60:0/3856906710 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46310d0 msgr2=0x55a8d463f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b3ad9d640 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46310d0 0x55a8d463f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b3ad9d640 1 -- 172.21.15.60:0/3856906710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a8d46a0160 con 0x55a8d4630360 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b3ad9d640 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 0x55a8d463f2c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c005c40 tx=0x7f7b2c004310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce2378d7bd3d375 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b227fc640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b227fc640 1 -- 172.21.15.60:0/3856906710 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c04f030 con 0x55a8d4630360 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b227fc640 1 -- 172.21.15.60:0/3856906710 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f7b2c02c880 con 0x55a8d4630360 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3856906710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7b240a37e0 con 0x55a8d4630360 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b227fc640 1 -- 172.21.15.60:0/3856906710 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b2c059050 con 0x55a8d4630360 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b43c5b740 10 client.16233 fetch_fsmap learned FSMap version 31 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3856906710 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55a8d46a1460 con 0x55a8d4630360 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b227fc640 1 -- 172.21.15.60:0/3856906710 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b240a37e0 con 0x55a8d4630360 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b227fc640 1 -- 172.21.15.60:0/3856906710 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7b2c004920 con 0x55a8d4630360 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b43c5b740 10 client.16233 fetch_fsmap finished waiting for FSMap version 31 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b43c5b740 10 client.16233 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:14.759 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b43c5b740 20 client.16233 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.743+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55a8d46a1bc0 0x55a8d46a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.744+0000 7f7b43c5b740 4 client.16233 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T12:27:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.744+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3856906710 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a8d459be80 con 0x55a8d46a1bc0 2024-03-04T12:27:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.744+0000 7f7b33fff640 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55a8d46a1bc0 0x55a8d46a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.744+0000 7f7b227fc640 10 client.16233 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:14.760 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:14.744+0000 7f7b33fff640 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55a8d46a1bc0 0x55a8d46a3fe0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ae10ee76faf22a4 server_cookie=b8c7996b8d37680 in_seq=0 out_seq=0 2024-03-04T12:27:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b227fc640 1 -- 172.21.15.60:0/3856906710 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a8d459be80 con 0x55a8d46a1bc0 2024-03-04T12:27:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b227fc640 10 client.16233 handle_command_reply: tid=0 2024-03-04T12:27:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b43c5b740 1 client.16233 shutdown 2024-03-04T12:27:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3856906710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 msgr2=0x55a8d463f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 0x55a8d463f2c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c005c40 tx=0x7f7b2c004310 comp rx=0 tx=0).stop 2024-03-04T12:27:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3856906710 shutdown_connections 2024-03-04T12:27:15.049 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55a8d46a1bc0 0x55a8d46a3fe0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:15.050 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a8d46398e0 0x55a8d463fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:15.050 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8d46310d0 0x55a8d463f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:15.050 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b43c5b740 1 --2- 172.21.15.60:0/3856906710 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55a8d4630360 0x55a8d463f2c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:15.050 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b227fc640 0 client.16233 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:15.050 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3856906710 >> 172.21.15.60:0/3856906710 conn(0x55a8d45f7b90 msgr2=0x55a8d46350f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:15.050 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3856906710 shutdown_connections 2024-03-04T12:27:15.050 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b43c5b740 1 -- 172.21.15.60:0/3856906710 wait complete. 2024-03-04T12:27:15.050 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.041+0000 7f7b43c5b740 20 client.16233 trim_cache size 0 max 16384 2024-03-04T12:27:15.050 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:{ 2024-03-04T12:27:15.050 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "message": "", 2024-03-04T12:27:15.050 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "return_code": 0 2024-03-04T12:27:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:} 2024-03-04T12:27:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.042+0000 7f7b41080640 1 -- 172.21.15.60:0/2829179878 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f7b280a2000 msgr2=0x7f7b280a44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.042+0000 7f7b41080640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f7b280a2000 0x7f7b280a44c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7b34005fd0 tx=0x7f7b34009e60 comp rx=0 tx=0).stop 2024-03-04T12:27:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.042+0000 7f7b41080640 1 -- 172.21.15.60:0/2829179878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b3c106640 msgr2=0x7f7b3c073b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.042+0000 7f7b41080640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b3c106640 0x7f7b3c073b70 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f7b2400e9f0 tx=0x7f7b2400eeb0 comp rx=0 tx=0).stop 2024-03-04T12:27:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.042+0000 7f7b41080640 1 -- 172.21.15.60:0/2829179878 shutdown_connections 2024-03-04T12:27:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.042+0000 7f7b41080640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b3c0740b0 0x7f7b3c1a5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.042+0000 7f7b41080640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f7b280a2000 0x7f7b280a44c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.042+0000 7f7b41080640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f7b3c106640 0x7f7b3c073b70 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.042+0000 7f7b41080640 1 --2- 172.21.15.60:0/2829179878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7b3c101570 0x7f7b3c073630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.042+0000 7f7b41080640 1 -- 172.21.15.60:0/2829179878 >> 172.21.15.60:0/2829179878 conn(0x7f7b3c0fce50 msgr2=0x7f7b3c0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.042+0000 7f7b41080640 1 -- 172.21.15.60:0/2829179878 shutdown_connections 2024-03-04T12:27:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:15.042+0000 7f7b41080640 1 -- 172.21.15.60:0/2829179878 wait complete. 2024-03-04T12:27:15.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:14 smithi086 ceph-mon[36851]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-04T12:27:15.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:14 smithi086 ceph-mon[36851]: osdmap e95: 12 total, 12 up, 12 in 2024-03-04T12:27:15.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:14 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.86:6836/2924944116,v1:172.21.15.86:6837/2924944116] up:boot 2024-03-04T12:27:15.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:14 smithi086 ceph-mon[36851]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-04T12:27:15.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:14 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T12:27:15.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:14 smithi086 ceph-mon[36851]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:15.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:14 smithi063 ceph-mon[36717]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-04T12:27:15.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:14 smithi063 ceph-mon[36717]: osdmap e95: 12 total, 12 up, 12 in 2024-03-04T12:27:15.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:14 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.86:6836/2924944116,v1:172.21.15.86:6837/2924944116] up:boot 2024-03-04T12:27:15.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:14 smithi063 ceph-mon[36717]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-04T12:27:15.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:14 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T12:27:15.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:14 smithi063 ceph-mon[36717]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:15.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:14 smithi060 ceph-mon[32745]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-04T12:27:15.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:14 smithi060 ceph-mon[32745]: osdmap e95: 12 total, 12 up, 12 in 2024-03-04T12:27:15.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:14 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.86:6836/2924944116,v1:172.21.15.86:6837/2924944116] up:boot 2024-03-04T12:27:15.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:14 smithi060 ceph-mon[32745]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-04T12:27:15.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:14 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T12:27:15.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:14 smithi060 ceph-mon[32745]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:17.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:16 smithi086 ceph-mon[36851]: pgmap v45: 129 pgs: 129 active+clean; 14 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.3 MiB/s wr, 12 op/s 2024-03-04T12:27:17.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:16 smithi063 ceph-mon[36717]: pgmap v45: 129 pgs: 129 active+clean; 14 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.3 MiB/s wr, 12 op/s 2024-03-04T12:27:17.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:16 smithi060 ceph-mon[32745]: pgmap v45: 129 pgs: 129 active+clean; 14 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.3 MiB/s wr, 12 op/s 2024-03-04T12:27:18.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:17 smithi086 ceph-mon[36851]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:18.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:17 smithi063 ceph-mon[36717]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:18.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:17 smithi060 ceph-mon[32745]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:19.253 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:18 smithi063 ceph-mon[36717]: pgmap v46: 129 pgs: 129 active+clean; 14 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.3 MiB/s wr, 12 op/s 2024-03-04T12:27:19.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:18 smithi086 ceph-mon[36851]: pgmap v46: 129 pgs: 129 active+clean; 14 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.3 MiB/s wr, 12 op/s 2024-03-04T12:27:19.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:18 smithi060 ceph-mon[32745]: pgmap v46: 129 pgs: 129 active+clean; 14 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 1.3 MiB/s wr, 12 op/s 2024-03-04T12:27:20.778 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T12:27:20.778 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T12:27:20.899 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.889+0000 7f1170bf1640 1 -- 172.21.15.60:0/2121121541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f116c0692a0 msgr2=0x7f116c105f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:20.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.889+0000 7f1170bf1640 1 --2- 172.21.15.60:0/2121121541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f116c0692a0 0x7f116c105f00 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f1154009900 tx=0x7f115402bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:20.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.889+0000 7f1170bf1640 1 -- 172.21.15.60:0/2121121541 shutdown_connections 2024-03-04T12:27:20.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.889+0000 7f1170bf1640 1 --2- 172.21.15.60:0/2121121541 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f116c1065e0 0x7f116c10e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.889+0000 7f1170bf1640 1 --2- 172.21.15.60:0/2121121541 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f116c0692a0 0x7f116c105f00 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.889+0000 7f1170bf1640 1 --2- 172.21.15.60:0/2121121541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f116c0688f0 0x7f116c068cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.889+0000 7f1170bf1640 1 -- 172.21.15.60:0/2121121541 >> 172.21.15.60:0/2121121541 conn(0x7f116c077330 msgr2=0x7f116c077740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:20.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.889+0000 7f1170bf1640 1 -- 172.21.15.60:0/2121121541 shutdown_connections 2024-03-04T12:27:20.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.889+0000 7f1170bf1640 1 -- 172.21.15.60:0/2121121541 wait complete. 2024-03-04T12:27:20.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.890+0000 7f1170bf1640 1 Processor -- start 2024-03-04T12:27:20.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.890+0000 7f1170bf1640 1 -- start start 2024-03-04T12:27:20.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.890+0000 7f1170bf1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f116c0688f0 0x7f116c1001f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.890+0000 7f1170bf1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f116c1065e0 0x7f116c100730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.890+0000 7f1170bf1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f116c100c70 0x7f116c1a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1170bf1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c111650 con 0x7f116c1065e0 2024-03-04T12:27:20.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1170bf1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c1114d0 con 0x7f116c0688f0 2024-03-04T12:27:20.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1170bf1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c1117d0 con 0x7f116c100c70 2024-03-04T12:27:20.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1163fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f116c1065e0 0x7f116c100730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1163fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f116c1065e0 0x7f116c100730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34182/0 (socket says 172.21.15.60:34182) 2024-03-04T12:27:20.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1163fff640 1 -- 172.21.15.60:0/1913235577 learned_addr learned my addr 172.21.15.60:0/1913235577 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:20.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f116ad9d640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f116c100c70 0x7f116c1a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1163fff640 1 -- 172.21.15.60:0/1913235577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f116c100c70 msgr2=0x7f116c1a5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:20.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f116a59c640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f116c0688f0 0x7f116c1001f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1163fff640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f116c100c70 0x7f116c1a5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1163fff640 1 -- 172.21.15.60:0/1913235577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f116c0688f0 msgr2=0x7f116c1001f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:20.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1163fff640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f116c0688f0 0x7f116c1001f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1163fff640 1 -- 172.21.15.60:0/1913235577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c1a63c0 con 0x7f116c1065e0 2024-03-04T12:27:20.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1163fff640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f116c1065e0 0x7f116c100730 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1154036040 tx=0x7f115402ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:20.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f11637fe640 1 -- 172.21.15.60:0/1913235577 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1154041070 con 0x7f116c1065e0 2024-03-04T12:27:20.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f11637fe640 1 -- 172.21.15.60:0/1913235577 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11540054e0 con 0x7f116c1065e0 2024-03-04T12:27:20.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1170bf1640 1 -- 172.21.15.60:0/1913235577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116c1a6650 con 0x7f116c1065e0 2024-03-04T12:27:20.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f11637fe640 1 -- 172.21.15.60:0/1913235577 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1154034440 con 0x7f116c1065e0 2024-03-04T12:27:20.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.891+0000 7f1170bf1640 1 -- 172.21.15.60:0/1913235577 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f116c1a6b90 con 0x7f116c1065e0 2024-03-04T12:27:20.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.894+0000 7f11637fe640 1 -- 172.21.15.60:0/1913235577 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f1154044070 con 0x7f116c1065e0 2024-03-04T12:27:20.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.894+0000 7f11737d3740 1 Processor -- start 2024-03-04T12:27:20.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f11737d3740 1 -- start start 2024-03-04T12:27:20.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f11737d3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f11737d3740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271d7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f11737d3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 0x561d271da080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f11737d3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d27131e00 con 0x561d271ce790 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f11737d3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d270dab10 con 0x561d271d7c90 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f11737d3740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d2712dea0 con 0x561d271cf0b0 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f11637fe640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f11580a20c0 0x7f11580a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f116ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f116ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271ceb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34196/0 (socket says 172.21.15.60:34196) 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f116ad9d640 1 -- 172.21.15.60:0/3691480915 learned_addr learned my addr 172.21.15.60:0/3691480915 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f116a59c640 1 --2- 172.21.15.60:0/3691480915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271d7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f116ad9d640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f11580a20c0 0x7f11580a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f11637fe640 1 -- 172.21.15.60:0/1913235577 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f11540f0300 con 0x7f116c1065e0 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f1163fff640 1 --2- 172.21.15.60:0/3691480915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 0x561d271da080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f116ad9d640 1 -- 172.21.15.60:0/3691480915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 msgr2=0x561d271d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f116ad9d640 1 --2- 172.21.15.60:0/3691480915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271d7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f116ad9d640 1 -- 172.21.15.60:0/3691480915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 msgr2=0x561d271da080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f116ad9d640 1 --2- 172.21.15.60:0/3691480915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 0x561d271da080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.895+0000 7f116ad9d640 1 -- 172.21.15.60:0/3691480915 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d26f107d0 con 0x561d271ce790 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.896+0000 7f116ad9d640 1 --2- 172.21.15.60:0/3691480915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271ceb70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f11640139f0 tx=0x7f1164013eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5d831db723622b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.896+0000 7f1160ff9640 1 -- 172.21.15.60:0/3691480915 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11640134c0 con 0x561d271ce790 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.896+0000 7f1160ff9640 1 -- 172.21.15.60:0/3691480915 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1164022070 con 0x561d271ce790 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.896+0000 7f1160ff9640 1 -- 172.21.15.60:0/3691480915 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116401cbe0 con 0x561d271ce790 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 -- 172.21.15.60:0/3691480915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 msgr2=0x561d271ceb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 --2- 172.21.15.60:0/3691480915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271ceb70 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f11640139f0 tx=0x7f1164013eb0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f116ad9d640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f11580a20c0 0x7f11580a4580 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1164021000 tx=0x7f116401f000 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 -- 172.21.15.60:0/3691480915 shutdown_connections 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 --2- 172.21.15.60:0/3691480915 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 0x561d271da080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 --2- 172.21.15.60:0/3691480915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271d7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 --2- 172.21.15.60:0/3691480915 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271ceb70 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 -- 172.21.15.60:0/3691480915 >> 172.21.15.60:0/3691480915 conn(0x561d271c8ec0 msgr2=0x561d271c92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 -- 172.21.15.60:0/3691480915 shutdown_connections 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 -- 172.21.15.60:0/3691480915 wait complete. 2024-03-04T12:27:20.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 -- start start 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 Processor -- start 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271ce790 0x561d271d73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271d4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271d7c90 0x561d271d4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d271daa80 con 0x561d271d7c90 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d270ee0a0 con 0x561d271ce790 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d271dac00 con 0x561d271cf0b0 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.897+0000 7f11737d3740 10 client.0 _ll_register_callbacks cb 0x561d270eda70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f116ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271d7c90 0x561d271d4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f116ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271d7c90 0x561d271d4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34204/0 (socket says 172.21.15.60:34204) 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f116ad9d640 1 -- 172.21.15.60:0/1398204743 learned_addr learned my addr 172.21.15.60:0/1398204743 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f1163fff640 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271d4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f116a59c640 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271ce790 0x561d271d73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f116ad9d640 1 -- 172.21.15.60:0/1398204743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 msgr2=0x561d271d4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f116ad9d640 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271d4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f116ad9d640 1 -- 172.21.15.60:0/1398204743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271ce790 msgr2=0x561d271d73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f116ad9d640 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271ce790 0x561d271d73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f116ad9d640 1 -- 172.21.15.60:0/1398204743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d26e463f0 con 0x561d271d7c90 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f116a59c640 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271ce790 0x561d271d73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f116ad9d640 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271d7c90 0x561d271d4c80 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1164009640 tx=0x7f1164011880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5880480739f20a0c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f1151ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f1151ffb640 1 -- 172.21.15.60:0/1398204743 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116401cb60 con 0x561d271d7c90 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f1151ffb640 1 -- 172.21.15.60:0/1398204743 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f116401c350 con 0x561d271d7c90 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f11737d3740 1 -- 172.21.15.60:0/1398204743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1164022070 con 0x561d271d7c90 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f1151ffb640 1 -- 172.21.15.60:0/1398204743 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f116403c050 con 0x561d271d7c90 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f11737d3740 10 client.16245 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:20.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.898+0000 7f11737d3740 1 -- 172.21.15.60:0/1398204743 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x561d26e060f0 con 0x561d271d7c90 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.899+0000 7f1151ffb640 1 -- 172.21.15.60:0/1398204743 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1164022070 con 0x561d271d7c90 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.899+0000 7f1151ffb640 1 -- 172.21.15.60:0/1398204743 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f1164037030 con 0x561d271d7c90 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.899+0000 7f11737d3740 10 client.16245 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.899+0000 7f11737d3740 10 client.16245 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.899+0000 7f11737d3740 20 client.16245 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.899+0000 7f11737d3740 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x561d271e76a0 0x561d271e9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.899+0000 7f11737d3740 4 client.16245 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.899+0000 7f11737d3740 1 -- 172.21.15.60:0/1398204743 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561d2713f340 con 0x561d271e76a0 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.899+0000 7f116a59c640 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x561d271e76a0 0x561d271e9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.900+0000 7f1151ffb640 10 client.16245 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.900+0000 7f116a59c640 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x561d271e76a0 0x561d271e9a90 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46bb3b33416b84e4 server_cookie=3f6f729cddde3cc0 in_seq=0 out_seq=0 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.900+0000 7f1151ffb640 1 -- 172.21.15.60:0/1398204743 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561d2713f340 con 0x561d271e76a0 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.900+0000 7f1151ffb640 10 client.16245 handle_command_reply: tid=0 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.900+0000 7f11737d3740 1 client.16245 shutdown 2024-03-04T12:27:20.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.901+0000 7f11737d3740 1 -- 172.21.15.60:0/1398204743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271d7c90 msgr2=0x561d271d4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:20.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.901+0000 7f11737d3740 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271d7c90 0x561d271d4c80 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1164009640 tx=0x7f1164011880 comp rx=0 tx=0).stop 2024-03-04T12:27:20.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.901+0000 7f11737d3740 1 -- 172.21.15.60:0/1398204743 shutdown_connections 2024-03-04T12:27:20.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.901+0000 7f11737d3740 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x561d271e76a0 0x561d271e9a90 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.901+0000 7f11737d3740 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271d7c90 0x561d271d4c80 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.901+0000 7f11737d3740 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271d4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.901+0000 7f11737d3740 1 --2- 172.21.15.60:0/1398204743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271ce790 0x561d271d73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.901+0000 7f1151ffb640 0 client.16245 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:20.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.901+0000 7f11737d3740 1 -- 172.21.15.60:0/1398204743 >> 172.21.15.60:0/1398204743 conn(0x561d271c8ec0 msgr2=0x561d271cac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:20.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.901+0000 7f11737d3740 1 -- 172.21.15.60:0/1398204743 shutdown_connections 2024-03-04T12:27:20.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.901+0000 7f11737d3740 1 -- 172.21.15.60:0/1398204743 wait complete. 2024-03-04T12:27:20.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.901+0000 7f11737d3740 20 client.16245 trim_cache size 0 max 16384 2024-03-04T12:27:20.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.917+0000 7f11737d3740 1 Processor -- start 2024-03-04T12:27:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.917+0000 7f11737d3740 1 -- start start 2024-03-04T12:27:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.917+0000 7f11737d3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271d6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.917+0000 7f11737d3740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271d73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.917+0000 7f11737d3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 0x561d271cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.917+0000 7f11737d3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c1b0570 con 0x561d271ce790 2024-03-04T12:27:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.917+0000 7f11737d3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c1b1f20 con 0x561d271d7c90 2024-03-04T12:27:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.917+0000 7f11737d3740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c1b0bd0 con 0x561d271cf0b0 2024-03-04T12:27:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.917+0000 7f116a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271d6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.917+0000 7f116a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271d6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34206/0 (socket says 172.21.15.60:34206) 2024-03-04T12:27:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.917+0000 7f116a59c640 1 -- 172.21.15.60:0/2233171210 learned_addr learned my addr 172.21.15.60:0/2233171210 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.917+0000 7f1163fff640 1 --2- 172.21.15.60:0/2233171210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271d73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.931 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.918+0000 7f116ad9d640 1 --2- 172.21.15.60:0/2233171210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 0x561d271cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.918+0000 7f116a59c640 1 -- 172.21.15.60:0/2233171210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 msgr2=0x561d271d73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.918+0000 7f116a59c640 1 --2- 172.21.15.60:0/2233171210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271d73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.918+0000 7f116a59c640 1 -- 172.21.15.60:0/2233171210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 msgr2=0x561d271cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.918+0000 7f116a59c640 1 --2- 172.21.15.60:0/2233171210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 0x561d271cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.918+0000 7f116a59c640 1 -- 172.21.15.60:0/2233171210 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d271d2c90 con 0x561d271ce790 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.918+0000 7f116a59c640 1 --2- 172.21.15.60:0/2233171210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271d6e70 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x561d271ea0f0 tx=0x7f115c00f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716c22049eabdf63 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.918+0000 7f1153fff640 1 -- 172.21.15.60:0/2233171210 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c01b3b0 con 0x561d271ce790 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.918+0000 7f1153fff640 1 -- 172.21.15.60:0/2233171210 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f115c00e3b0 con 0x561d271ce790 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.918+0000 7f1153fff640 1 -- 172.21.15.60:0/2233171210 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c00d600 con 0x561d271ce790 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 -- 172.21.15.60:0/2233171210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 msgr2=0x561d271d6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 --2- 172.21.15.60:0/2233171210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271d6e70 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x561d271ea0f0 tx=0x7f115c00f7f0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 -- 172.21.15.60:0/2233171210 shutdown_connections 2024-03-04T12:27:20.932 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 --2- 172.21.15.60:0/2233171210 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 0x561d271cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 --2- 172.21.15.60:0/2233171210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271d73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 --2- 172.21.15.60:0/2233171210 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271d6e70 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 -- 172.21.15.60:0/2233171210 >> 172.21.15.60:0/2233171210 conn(0x561d271c8ec0 msgr2=0x561d271d3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:20.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 -- 172.21.15.60:0/2233171210 shutdown_connections 2024-03-04T12:27:20.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 -- 172.21.15.60:0/2233171210 wait complete. 2024-03-04T12:27:20.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 -- start start 2024-03-04T12:27:20.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 Processor -- start 2024-03-04T12:27:20.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271dbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.933 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271dc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 0x561d2723d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c1b18c0 con 0x561d271ce790 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c1b1230 con 0x561d271d7c90 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c1ae810 con 0x561d271cf0b0 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f11737d3740 10 client.0 _ll_register_callbacks cb 0x7f11640092a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.919+0000 7f116a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271dbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f1163fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271dc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f116a59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271dbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34222/0 (socket says 172.21.15.60:34222) 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f116a59c640 1 -- 172.21.15.60:0/490877501 learned_addr learned my addr 172.21.15.60:0/490877501 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f116ad9d640 1 --2- 172.21.15.60:0/490877501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 0x561d2723d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f116a59c640 1 -- 172.21.15.60:0/490877501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 msgr2=0x561d271dc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f116a59c640 1 --2- 172.21.15.60:0/490877501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271dc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f116a59c640 1 -- 172.21.15.60:0/490877501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 msgr2=0x561d2723d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:20.934 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f116a59c640 1 --2- 172.21.15.60:0/490877501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 0x561d2723d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f116a59c640 1 -- 172.21.15.60:0/490877501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d2723de00 con 0x561d271ce790 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f116a59c640 1 --2- 172.21.15.60:0/490877501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271dbf10 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f115c00b970 tx=0x7f115c00b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dbe9d1dc380353f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f1151ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f1151ffb640 1 -- 172.21.15.60:0/490877501 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c031060 con 0x561d271ce790 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.920+0000 7f1151ffb640 1 -- 172.21.15.60:0/490877501 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f115c01b7c0 con 0x561d271ce790 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f11737d3740 1 -- 172.21.15.60:0/490877501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f115c00e3b0 con 0x561d271ce790 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f1151ffb640 1 -- 172.21.15.60:0/490877501 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f115c038050 con 0x561d271ce790 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f11737d3740 10 client.16251 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f11737d3740 1 -- 172.21.15.60:0/490877501 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x561d2723f080 con 0x561d271ce790 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f1151ffb640 1 -- 172.21.15.60:0/490877501 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f115c00e3b0 con 0x561d271ce790 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f1151ffb640 1 -- 172.21.15.60:0/490877501 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f115c02b030 con 0x561d271ce790 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f11737d3740 10 client.16251 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f11737d3740 10 client.16251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:20.935 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f11737d3740 20 client.16251 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f11737d3740 1 --2- 172.21.15.60:0/490877501 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x561d2723f690 0x561d27241ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f11737d3740 4 client.16251 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f11737d3740 1 -- 172.21.15.60:0/490877501 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561d27139690 con 0x561d2723f690 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.921+0000 7f1163fff640 1 --2- 172.21.15.60:0/490877501 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x561d2723f690 0x561d27241ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f1163fff640 1 --2- 172.21.15.60:0/490877501 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x561d2723f690 0x561d27241ab0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85e5eb7fdbfaa63e server_cookie=c55aecdd93812b74 in_seq=0 out_seq=0 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f1151ffb640 10 client.16251 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f1151ffb640 1 -- 172.21.15.60:0/490877501 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561d27139690 con 0x561d2723f690 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f1151ffb640 10 client.16251 handle_command_reply: tid=0 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f11737d3740 1 client.16251 shutdown 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f11737d3740 1 -- 172.21.15.60:0/490877501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 msgr2=0x561d271dbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f11737d3740 1 --2- 172.21.15.60:0/490877501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271dbf10 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f115c00b970 tx=0x7f115c00b9a0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f11737d3740 1 -- 172.21.15.60:0/490877501 shutdown_connections 2024-03-04T12:27:20.936 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f11737d3740 1 --2- 172.21.15.60:0/490877501 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x561d2723f690 0x561d27241ab0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f11737d3740 1 --2- 172.21.15.60:0/490877501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d271d7c90 0x561d2723d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f11737d3740 1 --2- 172.21.15.60:0/490877501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561d271cf0b0 0x561d271dc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f11737d3740 1 --2- 172.21.15.60:0/490877501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x561d271ce790 0x561d271dbf10 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f11737d3740 1 -- 172.21.15.60:0/490877501 >> 172.21.15.60:0/490877501 conn(0x561d271c8ec0 msgr2=0x561d271cf5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f1151ffb640 0 client.16251 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f11737d3740 1 -- 172.21.15.60:0/490877501 shutdown_connections 2024-03-04T12:27:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f11737d3740 1 -- 172.21.15.60:0/490877501 wait complete. 2024-03-04T12:27:20.937 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.922+0000 7f11737d3740 20 client.16251 trim_cache size 0 max 16384 2024-03-04T12:27:20.937 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:27:20.937 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "no active scrubs running", 2024-03-04T12:27:20.938 INFO:teuthology.orchestra.run.smithi060.stdout: "scrubs": {} 2024-03-04T12:27:20.938 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:27:20.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.923+0000 7f1170bf1640 1 -- 172.21.15.60:0/1913235577 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f11580a20c0 msgr2=0x7f11580a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:20.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.923+0000 7f1170bf1640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f11580a20c0 0x7f11580a4580 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1164021000 tx=0x7f116401f000 comp rx=0 tx=0).stop 2024-03-04T12:27:20.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.923+0000 7f1170bf1640 1 -- 172.21.15.60:0/1913235577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f116c1065e0 msgr2=0x7f116c100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:20.938 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.923+0000 7f1170bf1640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f116c1065e0 0x7f116c100730 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1154036040 tx=0x7f115402ce20 comp rx=0 tx=0).stop 2024-03-04T12:27:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.923+0000 7f1170bf1640 1 -- 172.21.15.60:0/1913235577 shutdown_connections 2024-03-04T12:27:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.923+0000 7f1170bf1640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f116c100c70 0x7f116c1a5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.923+0000 7f1170bf1640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f11580a20c0 0x7f11580a4580 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.923+0000 7f1170bf1640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f116c1065e0 0x7f116c100730 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.923+0000 7f1170bf1640 1 --2- 172.21.15.60:0/1913235577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f116c0688f0 0x7f116c1001f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.923+0000 7f1170bf1640 1 -- 172.21.15.60:0/1913235577 >> 172.21.15.60:0/1913235577 conn(0x7f116c077330 msgr2=0x7f116c0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.923+0000 7f1170bf1640 1 -- 172.21.15.60:0/1913235577 shutdown_connections 2024-03-04T12:27:20.939 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:20.923+0000 7f1170bf1640 1 -- 172.21.15.60:0/1913235577 wait complete. 2024-03-04T12:27:20.942 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T12:27:20.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:27:21.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.038+0000 7f874ee30640 1 -- 172.21.15.60:0/1586650002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8748100de0 msgr2=0x7f8748104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.048 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.038+0000 7f874ee30640 1 --2- 172.21.15.60:0/1586650002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8748100de0 0x7f8748104e20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8734009900 tx=0x7f873402bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:21.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.038+0000 7f874ee30640 1 -- 172.21.15.60:0/1586650002 shutdown_connections 2024-03-04T12:27:21.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.038+0000 7f874ee30640 1 --2- 172.21.15.60:0/1586650002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8748105500 0x7f874810d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.038+0000 7f874ee30640 1 --2- 172.21.15.60:0/1586650002 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8748100de0 0x7f8748104e20 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.038+0000 7f874ee30640 1 --2- 172.21.15.60:0/1586650002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8748100430 0x7f8748100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.038+0000 7f874ee30640 1 -- 172.21.15.60:0/1586650002 >> 172.21.15.60:0/1586650002 conn(0x7f87480fbcb0 msgr2=0x7f87480fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.049 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.038+0000 7f874ee30640 1 -- 172.21.15.60:0/1586650002 shutdown_connections 2024-03-04T12:27:21.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.038+0000 7f874ee30640 1 -- 172.21.15.60:0/1586650002 wait complete. 2024-03-04T12:27:21.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.038+0000 7f874ee30640 1 Processor -- start 2024-03-04T12:27:21.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.038+0000 7f874ee30640 1 -- start start 2024-03-04T12:27:21.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874ee30640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8748100430 0x7f874819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874ee30640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8748105500 0x7f874819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874ee30640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8748196910 0x7f8748196dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.050 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874ee30640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8748110570 con 0x7f8748196910 2024-03-04T12:27:21.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874ee30640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87481103f0 con 0x7f8748105500 2024-03-04T12:27:21.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874ee30640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87481106f0 con 0x7f8748100430 2024-03-04T12:27:21.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874d3cd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8748196910 0x7f8748196dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874d3cd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8748196910 0x7f8748196dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34230/0 (socket says 172.21.15.60:34230) 2024-03-04T12:27:21.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874d3cd640 1 -- 172.21.15.60:0/3384825315 learned_addr learned my addr 172.21.15.60:0/3384825315 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.051 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f8747fff640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8748105500 0x7f874819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874cbcc640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8748100430 0x7f874819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874d3cd640 1 -- 172.21.15.60:0/3384825315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8748100430 msgr2=0x7f874819c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874d3cd640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8748100430 0x7f874819c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874d3cd640 1 -- 172.21.15.60:0/3384825315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8748105500 msgr2=0x7f874819cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874d3cd640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8748105500 0x7f874819cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.052 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874d3cd640 1 -- 172.21.15.60:0/3384825315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8748197430 con 0x7f8748196910 2024-03-04T12:27:21.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874d3cd640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8748196910 0x7f8748196dc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f874000eaa0 tx=0x7f874000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f8745ffb640 1 -- 172.21.15.60:0/3384825315 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874000cdb0 con 0x7f8748196910 2024-03-04T12:27:21.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f874ee30640 1 -- 172.21.15.60:0/3384825315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8748197720 con 0x7f8748196910 2024-03-04T12:27:21.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f8745ffb640 1 -- 172.21.15.60:0/3384825315 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8740005020 con 0x7f8748196910 2024-03-04T12:27:21.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.039+0000 7f8745ffb640 1 -- 172.21.15.60:0/3384825315 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87400116b0 con 0x7f8748196910 2024-03-04T12:27:21.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.040+0000 7f874ee30640 1 -- 172.21.15.60:0/3384825315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87481a3970 con 0x7f8748196910 2024-03-04T12:27:21.053 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.040+0000 7f874ee30640 1 -- 172.21.15.60:0/3384825315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8748101900 con 0x7f8748196910 2024-03-04T12:27:21.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.041+0000 7f8745ffb640 1 -- 172.21.15.60:0/3384825315 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f8740005560 con 0x7f8748196910 2024-03-04T12:27:21.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.042+0000 7f8745ffb640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f87380a2010 0x7f87380a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.055 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.042+0000 7f874cbcc640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f87380a2010 0x7f87380a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.042+0000 7f8745ffb640 1 -- 172.21.15.60:0/3384825315 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f87400cead0 con 0x7f8748196910 2024-03-04T12:27:21.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.042+0000 7f874cbcc640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f87380a2010 0x7f87380a44d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f873c005fd0 tx=0x7f873c005850 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.056 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.046+0000 7f8745ffb640 1 -- 172.21.15.60:0/3384825315 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8740016030 con 0x7f8748196910 2024-03-04T12:27:21.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:20 smithi060 ceph-mon[32745]: pgmap v47: 129 pgs: 129 active+clean; 16 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-03-04T12:27:21.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.265+0000 7f874ee30640 1 -- 172.21.15.60:0/3384825315 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8748100de0 con 0x7f8748196910 2024-03-04T12:27:21.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.265+0000 7f8745ffb640 1 -- 172.21.15.60:0/3384825315 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+11377 (secure 0 0 0) 0x7f874009b5d0 con 0x7f8748196910 2024-03-04T12:27:21.273 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:27:21.273 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:27:16.612374+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_25741":{"gid":25741,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/2924944116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2924944116},{"type":"v1","addr":"172.21.15.86:6837","nonce":2924944116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:27:21.274 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 32 2024-03-04T12:27:21.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.267+0000 7f874ee30640 1 -- 172.21.15.60:0/3384825315 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f87380a2010 msgr2=0x7f87380a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.267+0000 7f874ee30640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f87380a2010 0x7f87380a44d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f873c005fd0 tx=0x7f873c005850 comp rx=0 tx=0).stop 2024-03-04T12:27:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.267+0000 7f874ee30640 1 -- 172.21.15.60:0/3384825315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8748196910 msgr2=0x7f8748196dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.267+0000 7f874ee30640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8748196910 0x7f8748196dc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f874000eaa0 tx=0x7f874000ef60 comp rx=0 tx=0).stop 2024-03-04T12:27:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.267+0000 7f874ee30640 1 -- 172.21.15.60:0/3384825315 shutdown_connections 2024-03-04T12:27:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.267+0000 7f874ee30640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f87380a2010 0x7f87380a44d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.267+0000 7f874ee30640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f8748196910 0x7f8748196dc0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.267+0000 7f874ee30640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8748105500 0x7f874819cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.267+0000 7f874ee30640 1 --2- 172.21.15.60:0/3384825315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8748100430 0x7f874819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.267+0000 7f874ee30640 1 -- 172.21.15.60:0/3384825315 >> 172.21.15.60:0/3384825315 conn(0x7f87480fbcb0 msgr2=0x7f87480fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.268+0000 7f874ee30640 1 -- 172.21.15.60:0/3384825315 shutdown_connections 2024-03-04T12:27:21.276 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.268+0000 7f874ee30640 1 -- 172.21.15.60:0/3384825315 wait complete. 2024-03-04T12:27:21.286 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:27:21.286 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-04T12:27:21.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:20 smithi086 ceph-mon[36851]: pgmap v47: 129 pgs: 129 active+clean; 16 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-03-04T12:27:21.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:20 smithi063 ceph-mon[36717]: pgmap v47: 129 pgs: 129 active+clean; 16 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-03-04T12:27:21.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.383+0000 7f9b6464b640 1 -- 172.21.15.60:0/1330229415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b5c101ef0 msgr2=0x7f9b5c105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.383+0000 7f9b6464b640 1 --2- 172.21.15.60:0/1330229415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b5c101ef0 0x7f9b5c105f30 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009900 tx=0x7f9b4c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:21.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 -- 172.21.15.60:0/1330229415 shutdown_connections 2024-03-04T12:27:21.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 --2- 172.21.15.60:0/1330229415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5c106610 0x7f9b5c10e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 --2- 172.21.15.60:0/1330229415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b5c101ef0 0x7f9b5c105f30 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 --2- 172.21.15.60:0/1330229415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b5c101540 0x7f9b5c101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 -- 172.21.15.60:0/1330229415 >> 172.21.15.60:0/1330229415 conn(0x7f9b5c0fcdc0 msgr2=0x7f9b5c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 -- 172.21.15.60:0/1330229415 shutdown_connections 2024-03-04T12:27:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 -- 172.21.15.60:0/1330229415 wait complete. 2024-03-04T12:27:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 Processor -- start 2024-03-04T12:27:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 -- start start 2024-03-04T12:27:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b5c101540 0x7f9b5c19d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5c106610 0x7f9b5c19dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b5c1979e0 0x7f9b5c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c1116a0 con 0x7f9b5c1979e0 2024-03-04T12:27:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c111520 con 0x7f9b5c101540 2024-03-04T12:27:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b6464b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c111820 con 0x7f9b5c106610 2024-03-04T12:27:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.384+0000 7f9b62be8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b5c1979e0 0x7f9b5c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b62be8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b5c1979e0 0x7f9b5c197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34252/0 (socket says 172.21.15.60:34252) 2024-03-04T12:27:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b62be8640 1 -- 172.21.15.60:0/2487499777 learned_addr learned my addr 172.21.15.60:0/2487499777 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.396 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b61be6640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5c106610 0x7f9b5c19dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b623e7640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b5c101540 0x7f9b5c19d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b62be8640 1 -- 172.21.15.60:0/2487499777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5c106610 msgr2=0x7f9b5c19dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b62be8640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5c106610 0x7f9b5c19dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b62be8640 1 -- 172.21.15.60:0/2487499777 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b5c101540 msgr2=0x7f9b5c19d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b62be8640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b5c101540 0x7f9b5c19d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b62be8640 1 -- 172.21.15.60:0/2487499777 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5c198500 con 0x7f9b5c1979e0 2024-03-04T12:27:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b623e7640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b5c101540 0x7f9b5c19d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:27:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b62be8640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b5c1979e0 0x7f9b5c197e90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800cad0 tx=0x7f9b5800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b537fe640 1 -- 172.21.15.60:0/2487499777 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5800ae20 con 0x7f9b5c1979e0 2024-03-04T12:27:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b537fe640 1 -- 172.21.15.60:0/2487499777 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b58005020 con 0x7f9b5c1979e0 2024-03-04T12:27:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b537fe640 1 -- 172.21.15.60:0/2487499777 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5800f6f0 con 0x7f9b5c1979e0 2024-03-04T12:27:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b6464b640 1 -- 172.21.15.60:0/2487499777 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5c1987f0 con 0x7f9b5c1979e0 2024-03-04T12:27:21.398 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.385+0000 7f9b6464b640 1 -- 172.21.15.60:0/2487499777 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5c1a49f0 con 0x7f9b5c1979e0 2024-03-04T12:27:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.387+0000 7f9b537fe640 1 -- 172.21.15.60:0/2487499777 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f9b580051d0 con 0x7f9b5c1979e0 2024-03-04T12:27:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.388+0000 7f9b6722d740 1 Processor -- start 2024-03-04T12:27:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.388+0000 7f9b6722d740 1 -- start start 2024-03-04T12:27:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.388+0000 7f9b537fe640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f9b480a2090 0x7f9b480a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.388+0000 7f9b6722d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226e79d0 0x55f8226e7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.388+0000 7f9b6722d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f8226f0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.388+0000 7f9b6722d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226f0ed0 0x55f8226f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.388+0000 7f9b6722d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8226aedc0 con 0x55f8226e82f0 2024-03-04T12:27:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.388+0000 7f9b6722d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f82264aec0 con 0x55f8226e79d0 2024-03-04T12:27:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.388+0000 7f9b6722d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8225f3d30 con 0x55f8226f0ed0 2024-03-04T12:27:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b537fe640 1 -- 172.21.15.60:0/2487499777 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f9b580ceb30 con 0x7f9b5c1979e0 2024-03-04T12:27:21.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b61be6640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f9b480a2090 0x7f9b480a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b61be6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f8226f0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b61be6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f8226f0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34254/0 (socket says 172.21.15.60:34254) 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b61be6640 1 -- 172.21.15.60:0/3095014267 learned_addr learned my addr 172.21.15.60:0/3095014267 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b62be8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226f0ed0 0x55f8226f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b623e7640 1 --2- 172.21.15.60:0/3095014267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226e79d0 0x55f8226e7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b62be8640 1 -- 172.21.15.60:0/3095014267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226e79d0 msgr2=0x55f8226e7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b62be8640 1 --2- 172.21.15.60:0/3095014267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226e79d0 0x55f8226e7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b62be8640 1 -- 172.21.15.60:0/3095014267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 msgr2=0x55f8226f0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b62be8640 1 --2- 172.21.15.60:0/3095014267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f8226f0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b62be8640 1 -- 172.21.15.60:0/3095014267 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f82235f3f0 con 0x55f8226f0ed0 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b61be6640 1 --2- 172.21.15.60:0/3095014267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f8226f0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b61be6640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f9b480a2090 0x7f9b480a4550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c005ab0 tx=0x7f9b4c02c760 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.389+0000 7f9b62be8640 1 --2- 172.21.15.60:0/3095014267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226f0ed0 0x55f8226f32c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9b580cbd60 tx=0x7f9b5809a9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9fc7df7f7751da7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.390+0000 7f9b50bfe640 1 -- 172.21.15.60:0/3095014267 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5801d040 con 0x55f8226f0ed0 2024-03-04T12:27:21.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.390+0000 7f9b50bfe640 1 -- 172.21.15.60:0/3095014267 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b5809af30 con 0x55f8226f0ed0 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.390+0000 7f9b50bfe640 1 -- 172.21.15.60:0/3095014267 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b58004a40 con 0x55f8226f0ed0 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.390+0000 7f9b6722d740 1 -- 172.21.15.60:0/3095014267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226f0ed0 msgr2=0x55f8226f32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.390+0000 7f9b6722d740 1 --2- 172.21.15.60:0/3095014267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226f0ed0 0x55f8226f32c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9b580cbd60 tx=0x7f9b5809a9c0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.390+0000 7f9b6722d740 1 -- 172.21.15.60:0/3095014267 shutdown_connections 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.390+0000 7f9b6722d740 1 --2- 172.21.15.60:0/3095014267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226f0ed0 0x55f8226f32c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.390+0000 7f9b6722d740 1 --2- 172.21.15.60:0/3095014267 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f8226f0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.390+0000 7f9b6722d740 1 --2- 172.21.15.60:0/3095014267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226e79d0 0x55f8226e7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.390+0000 7f9b6722d740 1 -- 172.21.15.60:0/3095014267 >> 172.21.15.60:0/3095014267 conn(0x55f8226e2100 msgr2=0x55f8226e24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.390+0000 7f9b6722d740 1 -- 172.21.15.60:0/3095014267 shutdown_connections 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b6722d740 1 -- 172.21.15.60:0/3095014267 wait complete. 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b6722d740 1 -- start start 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b6722d740 1 Processor -- start 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b6722d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e79d0 0x55f8226ede60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b6722d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226e82f0 0x55f8226ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b6722d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226f0ed0 0x55f8226ee8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b6722d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8225e94a0 con 0x55f8226e79d0 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b6722d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f822646f60 con 0x55f8226e82f0 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b6722d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8226f3cc0 con 0x55f8226f0ed0 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b6722d740 10 client.0 _ll_register_callbacks cb 0x55f8226451c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b623e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e79d0 0x55f8226ede60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b623e7640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e79d0 0x55f8226ede60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34256/0 (socket says 172.21.15.60:34256) 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b623e7640 1 -- 172.21.15.60:0/269557177 learned_addr learned my addr 172.21.15.60:0/269557177 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b62be8640 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226f0ed0 0x55f8226ee8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b61be6640 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226e82f0 0x55f8226ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b623e7640 1 -- 172.21.15.60:0/269557177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226f0ed0 msgr2=0x55f8226ee8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b623e7640 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226f0ed0 0x55f8226ee8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b623e7640 1 -- 172.21.15.60:0/269557177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226e82f0 msgr2=0x55f8226ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.391+0000 7f9b623e7640 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226e82f0 0x55f8226ee3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b623e7640 1 -- 172.21.15.60:0/269557177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f8224297d0 con 0x55f8226e79d0 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b62be8640 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226f0ed0 0x55f8226ee8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b623e7640 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e79d0 0x55f8226ede60 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9b54010800 tx=0x7f9b54010cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44167e2f3113d306 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b45ffb640 1 -- 172.21.15.60:0/269557177 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5401d060 con 0x55f8226e79d0 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b45ffb640 1 -- 172.21.15.60:0/269557177 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f9b5400eda0 con 0x55f8226e79d0 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b6722d740 1 -- 172.21.15.60:0/269557177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b5809af30 con 0x55f8226e79d0 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b45ffb640 1 -- 172.21.15.60:0/269557177 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b54024050 con 0x55f8226e79d0 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b6722d740 10 client.16269 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b6722d740 1 -- 172.21.15.60:0/269557177 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f8226ffda0 con 0x55f8226e79d0 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b45ffb640 1 -- 172.21.15.60:0/269557177 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b5809af30 con 0x55f8226e79d0 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b45ffb640 1 -- 172.21.15.60:0/269557177 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f9b54016450 con 0x55f8226e79d0 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b6722d740 10 client.16269 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b6722d740 10 client.16269 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-04T12:27:21.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b6722d740 20 client.16269 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.392+0000 7f9b6722d740 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f822701260 0x55f8227036d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.393+0000 7f9b6722d740 4 client.16269 mds_command: new command op to 24394 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.393+0000 7f9b6722d740 1 -- 172.21.15.60:0/269557177 --> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f822658400 con 0x55f822701260 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.393+0000 7f9b61be6640 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f822701260 0x55f8227036d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.393+0000 7f9b61be6640 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f822701260 0x55f8227036d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=af410f5c73ad8732 server_cookie=6d8b9fae1d3cb2 in_seq=0 out_seq=0 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.393+0000 7f9b45ffb640 10 client.16269 ms_handle_connect on v2:172.21.15.86:6832/777854669 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b45ffb640 1 -- 172.21.15.60:0/269557177 <== mds.4 v2:172.21.15.86:6832/777854669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f822658400 con 0x55f822701260 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b45ffb640 10 client.16269 handle_command_reply: tid=0 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b6722d740 1 client.16269 shutdown 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b6722d740 1 -- 172.21.15.60:0/269557177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e79d0 msgr2=0x55f8226ede60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b6722d740 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e79d0 0x55f8226ede60 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9b54010800 tx=0x7f9b54010cc0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b6722d740 1 -- 172.21.15.60:0/269557177 shutdown_connections 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b6722d740 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226f0ed0 0x55f8226ee8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b6722d740 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f822701260 0x55f8227036d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b6722d740 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226e82f0 0x55f8226ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b6722d740 1 --2- 172.21.15.60:0/269557177 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e79d0 0x55f8226ede60 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b45ffb640 0 client.16269 ms_handle_reset on v2:172.21.15.86:6832/777854669 2024-03-04T12:27:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b6722d740 1 -- 172.21.15.60:0/269557177 >> 172.21.15.60:0/269557177 conn(0x55f8226e2100 msgr2=0x55f8226e3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b6722d740 1 -- 172.21.15.60:0/269557177 shutdown_connections 2024-03-04T12:27:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b6722d740 1 -- 172.21.15.60:0/269557177 wait complete. 2024-03-04T12:27:21.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.394+0000 7f9b6722d740 20 client.16269 trim_cache size 0 max 16384 2024-03-04T12:27:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.406+0000 7f9b6722d740 1 Processor -- start 2024-03-04T12:27:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.406+0000 7f9b6722d740 1 -- start start 2024-03-04T12:27:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.406+0000 7f9b6722d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226e79d0 0x55f8226f0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.406+0000 7f9b6722d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f8226f82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.406+0000 7f9b6722d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226f0ed0 0x55f8226ec730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.406+0000 7f9b6722d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c1ae550 con 0x55f8226e82f0 2024-03-04T12:27:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.406+0000 7f9b6722d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c1aff00 con 0x55f8226f0ed0 2024-03-04T12:27:21.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.406+0000 7f9b6722d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c1aebb0 con 0x55f8226e79d0 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.406+0000 7f9b61be6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f8226f82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.406+0000 7f9b61be6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f8226f82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34258/0 (socket says 172.21.15.60:34258) 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.406+0000 7f9b61be6640 1 -- 172.21.15.60:0/93279533 learned_addr learned my addr 172.21.15.60:0/93279533 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.406+0000 7f9b623e7640 1 --2- 172.21.15.60:0/93279533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226e79d0 0x55f8226f0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b61be6640 1 -- 172.21.15.60:0/93279533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226e79d0 msgr2=0x55f8226f0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b61be6640 1 --2- 172.21.15.60:0/93279533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226e79d0 0x55f8226f0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b61be6640 1 -- 172.21.15.60:0/93279533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226f0ed0 msgr2=0x55f8226ec730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b61be6640 1 --2- 172.21.15.60:0/93279533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226f0ed0 0x55f8226ec730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b61be6640 1 -- 172.21.15.60:0/93279533 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8226f32c0 con 0x55f8226e82f0 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b623e7640 1 --2- 172.21.15.60:0/93279533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226e79d0 0x55f8226f0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b61be6640 1 --2- 172.21.15.60:0/93279533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f8226f82e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009230 tx=0x7f9b4c04a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6374a5d7720204 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b47fff640 1 -- 172.21.15.60:0/93279533 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c0046c0 con 0x55f8226e82f0 2024-03-04T12:27:21.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b47fff640 1 -- 172.21.15.60:0/93279533 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b4c051170 con 0x55f8226e82f0 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b47fff640 1 -- 172.21.15.60:0/93279533 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c005170 con 0x55f8226e82f0 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b6722d740 1 -- 172.21.15.60:0/93279533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 msgr2=0x55f8226f82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b6722d740 1 --2- 172.21.15.60:0/93279533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f8226f82e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009230 tx=0x7f9b4c04a000 comp rx=0 tx=0).stop 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b6722d740 1 -- 172.21.15.60:0/93279533 shutdown_connections 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b6722d740 1 --2- 172.21.15.60:0/93279533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226f0ed0 0x55f8226ec730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b6722d740 1 --2- 172.21.15.60:0/93279533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f8226f82e0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b6722d740 1 --2- 172.21.15.60:0/93279533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226e79d0 0x55f8226f0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b6722d740 1 -- 172.21.15.60:0/93279533 >> 172.21.15.60:0/93279533 conn(0x55f8226e2100 msgr2=0x55f8226e9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.407+0000 7f9b6722d740 1 -- 172.21.15.60:0/93279533 shutdown_connections 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b6722d740 1 -- 172.21.15.60:0/93279533 wait complete. 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b6722d740 1 -- start start 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b6722d740 1 Processor -- start 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b6722d740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226e79d0 0x55f822754250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b6722d740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f822754790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b6722d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226f0ed0 0x55f82274f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b6722d740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c1af8a0 con 0x55f8226e82f0 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b6722d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c1af210 con 0x55f8226f0ed0 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b6722d740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5c1ac7f0 con 0x55f8226e79d0 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b6722d740 10 client.0 _ll_register_callbacks cb 0x7f9b5400b680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b61be6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f822754790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b61be6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f822754790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34266/0 (socket says 172.21.15.60:34266) 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b61be6640 1 -- 172.21.15.60:0/1784691873 learned_addr learned my addr 172.21.15.60:0/1784691873 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b623e7640 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226e79d0 0x55f822754250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b62be8640 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226f0ed0 0x55f82274f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b61be6640 1 -- 172.21.15.60:0/1784691873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226e79d0 msgr2=0x55f822754250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b61be6640 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226e79d0 0x55f822754250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b61be6640 1 -- 172.21.15.60:0/1784691873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226f0ed0 msgr2=0x55f82274f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b61be6640 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226f0ed0 0x55f82274f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.408+0000 7f9b61be6640 1 -- 172.21.15.60:0/1784691873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f82274ff60 con 0x55f8226e82f0 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b62be8640 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226f0ed0 0x55f82274f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b61be6640 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f822754790 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009900 tx=0x7f9b4c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245c351eb1d69ee9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b45ffb640 1 -- 172.21.15.60:0/1784691873 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c060020 con 0x55f8226e82f0 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b45ffb640 1 -- 172.21.15.60:0/1784691873 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f9b4c051420 con 0x55f8226e82f0 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b6722d740 1 -- 172.21.15.60:0/1784691873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b4c051170 con 0x55f8226e82f0 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b45ffb640 1 -- 172.21.15.60:0/1784691873 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b4c051620 con 0x55f8226e82f0 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b6722d740 10 client.16275 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b6722d740 1 -- 172.21.15.60:0/1784691873 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f82275caa0 con 0x55f8226e82f0 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b45ffb640 1 -- 172.21.15.60:0/1784691873 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b4c051170 con 0x55f8226e82f0 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b45ffb640 1 -- 172.21.15.60:0/1784691873 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f9b4c058030 con 0x55f8226e82f0 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b6722d740 10 client.16275 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b6722d740 10 client.16275 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-04T12:27:21.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b6722d740 20 client.16275 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b6722d740 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f82275d130 0x55f82275f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b6722d740 4 client.16275 mds_command: new command op to 24394 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.409+0000 7f9b6722d740 1 -- 172.21.15.60:0/1784691873 --> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f822652750 con 0x55f82275d130 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b623e7640 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f82275d130 0x55f82275f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b45ffb640 10 client.16275 ms_handle_connect on v2:172.21.15.86:6832/777854669 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b623e7640 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f82275d130 0x55f82275f550 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5ecedff2c4677d17 server_cookie=c65da0834036cad4 in_seq=0 out_seq=0 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b45ffb640 1 -- 172.21.15.60:0/1784691873 <== mds.4 v2:172.21.15.86:6832/777854669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f822652750 con 0x55f82275d130 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b45ffb640 10 client.16275 handle_command_reply: tid=0 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b6722d740 1 client.16275 shutdown 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b6722d740 1 -- 172.21.15.60:0/1784691873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 msgr2=0x55f822754790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b6722d740 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f822754790 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009900 tx=0x7f9b4c009260 comp rx=0 tx=0).stop 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b6722d740 1 -- 172.21.15.60:0/1784691873 shutdown_connections 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b6722d740 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x55f82275d130 0x55f82275f550 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b6722d740 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f8226f0ed0 0x55f82274f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b6722d740 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55f8226e82f0 0x55f822754790 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b6722d740 1 --2- 172.21.15.60:0/1784691873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f8226e79d0 0x55f822754250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b45ffb640 0 client.16275 ms_handle_reset on v2:172.21.15.86:6832/777854669 2024-03-04T12:27:21.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b6722d740 1 -- 172.21.15.60:0/1784691873 >> 172.21.15.60:0/1784691873 conn(0x55f8226e2100 msgr2=0x55f8226ec6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b6722d740 1 -- 172.21.15.60:0/1784691873 shutdown_connections 2024-03-04T12:27:21.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.410+0000 7f9b6722d740 1 -- 172.21.15.60:0/1784691873 wait complete. 2024-03-04T12:27:21.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6722d740 20 client.16275 trim_cache size 0 max 16384 2024-03-04T12:27:21.425 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:27:21.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6464b640 1 -- 172.21.15.60:0/2487499777 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f9b480a2090 msgr2=0x7f9b480a4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6464b640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f9b480a2090 0x7f9b480a4550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c005ab0 tx=0x7f9b4c02c760 comp rx=0 tx=0).stop 2024-03-04T12:27:21.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6464b640 1 -- 172.21.15.60:0/2487499777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b5c1979e0 msgr2=0x7f9b5c197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6464b640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b5c1979e0 0x7f9b5c197e90 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9b5800cad0 tx=0x7f9b5800cf90 comp rx=0 tx=0).stop 2024-03-04T12:27:21.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6464b640 1 -- 172.21.15.60:0/2487499777 shutdown_connections 2024-03-04T12:27:21.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6464b640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f9b480a2090 0x7f9b480a4550 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6464b640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9b5c1979e0 0x7f9b5c197e90 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6464b640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5c106610 0x7f9b5c19dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6464b640 1 --2- 172.21.15.60:0/2487499777 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b5c101540 0x7f9b5c19d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6464b640 1 -- 172.21.15.60:0/2487499777 >> 172.21.15.60:0/2487499777 conn(0x7f9b5c0fcdc0 msgr2=0x7f9b5c0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6464b640 1 -- 172.21.15.60:0/2487499777 shutdown_connections 2024-03-04T12:27:21.427 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.411+0000 7f9b6464b640 1 -- 172.21.15.60:0/2487499777 wait complete. 2024-03-04T12:27:21.429 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:27:21.429 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-04T12:27:21.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.526+0000 7f4c6dd05640 1 -- 172.21.15.60:0/2268561366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c681025a0 msgr2=0x7f4c68102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.536 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.526+0000 7f4c6dd05640 1 --2- 172.21.15.60:0/2268561366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c681025a0 0x7f4c68102980 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009900 tx=0x7f4c5802bcd0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.526+0000 7f4c6dd05640 1 -- 172.21.15.60:0/2268561366 shutdown_connections 2024-03-04T12:27:21.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.526+0000 7f4c6dd05640 1 --2- 172.21.15.60:0/2268561366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c6810ba10 0x7f4c6810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.526+0000 7f4c6dd05640 1 --2- 172.21.15.60:0/2268561366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c68102ec0 0x7f4c6810b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.526+0000 7f4c6dd05640 1 --2- 172.21.15.60:0/2268561366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c681025a0 0x7f4c68102980 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.526+0000 7f4c6dd05640 1 -- 172.21.15.60:0/2268561366 >> 172.21.15.60:0/2268561366 conn(0x7f4c680773a0 msgr2=0x7f4c680777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.526+0000 7f4c6dd05640 1 -- 172.21.15.60:0/2268561366 shutdown_connections 2024-03-04T12:27:21.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.526+0000 7f4c6dd05640 1 -- 172.21.15.60:0/2268561366 wait complete. 2024-03-04T12:27:21.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c6dd05640 1 Processor -- start 2024-03-04T12:27:21.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c6dd05640 1 -- start start 2024-03-04T12:27:21.537 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c6dd05640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c681025a0 0x7f4c681001f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c6dd05640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c68102ec0 0x7f4c68100730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c6dd05640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c6810ba10 0x7f4c68100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c6dd05640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c681117c0 con 0x7f4c681025a0 2024-03-04T12:27:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c6dd05640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c68111640 con 0x7f4c68102ec0 2024-03-04T12:27:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c6dd05640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c68111940 con 0x7f4c6810ba10 2024-03-04T12:27:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c677fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c681025a0 0x7f4c681001f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c677fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c681025a0 0x7f4c681001f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34292/0 (socket says 172.21.15.60:34292) 2024-03-04T12:27:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c677fe640 1 -- 172.21.15.60:0/1307428280 learned_addr learned my addr 172.21.15.60:0/1307428280 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c66ffd640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c68102ec0 0x7f4c68100730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.538 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c67fff640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c6810ba10 0x7f4c68100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c677fe640 1 -- 172.21.15.60:0/1307428280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c6810ba10 msgr2=0x7f4c68100c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c677fe640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c6810ba10 0x7f4c68100c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c677fe640 1 -- 172.21.15.60:0/1307428280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c68102ec0 msgr2=0x7f4c68100730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c677fe640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c68102ec0 0x7f4c68100730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.527+0000 7f4c677fe640 1 -- 172.21.15.60:0/1307428280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c681a61c0 con 0x7f4c681025a0 2024-03-04T12:27:21.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.528+0000 7f4c677fe640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c681025a0 0x7f4c681001f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f4c58005940 tx=0x7f4c5802cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.528+0000 7f4c64ff9640 1 -- 172.21.15.60:0/1307428280 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58041070 con 0x7f4c681025a0 2024-03-04T12:27:21.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.528+0000 7f4c6dd05640 1 -- 172.21.15.60:0/1307428280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c681a6450 con 0x7f4c681025a0 2024-03-04T12:27:21.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.528+0000 7f4c64ff9640 1 -- 172.21.15.60:0/1307428280 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c58004450 con 0x7f4c681025a0 2024-03-04T12:27:21.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.528+0000 7f4c64ff9640 1 -- 172.21.15.60:0/1307428280 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c58034440 con 0x7f4c681025a0 2024-03-04T12:27:21.539 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.528+0000 7f4c6dd05640 1 -- 172.21.15.60:0/1307428280 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c681a6940 con 0x7f4c681025a0 2024-03-04T12:27:21.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.530+0000 7f4c64ff9640 1 -- 172.21.15.60:0/1307428280 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f4c58044070 con 0x7f4c681025a0 2024-03-04T12:27:21.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.530+0000 7f4c709bd740 1 Processor -- start 2024-03-04T12:27:21.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c709bd740 1 -- start start 2024-03-04T12:27:21.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c64ff9640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4c500a2010 0x7f4c500a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c709bd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb169d0 0x55bdeeb16db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c709bd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c709bd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb222c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c67fff640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4c500a2010 0x7f4c500a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c64ff9640 1 -- 172.21.15.60:0/1307428280 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f4c580f0300 con 0x7f4c681025a0 2024-03-04T12:27:21.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34302/0 (socket says 172.21.15.60:34302) 2024-03-04T12:27:21.548 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c66ffd640 1 -- 172.21.15.60:0/823562362 learned_addr learned my addr 172.21.15.60:0/823562362 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c709bd740 1 -- 172.21.15.60:0/823562362 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeeadddc0 con 0x55bdeeb172f0 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c709bd740 1 -- 172.21.15.60:0/823562362 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeea79ec0 con 0x55bdeeb169d0 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c709bd740 1 -- 172.21.15.60:0/823562362 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeea22d30 con 0x55bdeeb1fed0 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c677fe640 1 --2- 172.21.15.60:0/823562362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb169d0 0x55bdeeb16db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c67fff640 1 --2- 172.21.15.60:0/823562362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb222c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c66ffd640 1 -- 172.21.15.60:0/823562362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb1fed0 msgr2=0x55bdeeb222c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c66ffd640 1 --2- 172.21.15.60:0/823562362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb222c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c66ffd640 1 -- 172.21.15.60:0/823562362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb169d0 msgr2=0x55bdeeb16db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c66ffd640 1 --2- 172.21.15.60:0/823562362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb169d0 0x55bdeeb16db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c66ffd640 1 -- 172.21.15.60:0/823562362 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdee78e3f0 con 0x55bdeeb172f0 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c67fff640 1 --2- 172.21.15.60:0/823562362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb222c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c67fff640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4c500a2010 0x7f4c500a44d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4c60005fd0 tx=0x7f4c60009a50 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.531+0000 7f4c66ffd640 1 --2- 172.21.15.60:0/823562362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1f990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00bcb0 tx=0x7f4c5c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df02e3a0f17b9f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.532+0000 7f4c4dffb640 1 -- 172.21.15.60:0/823562362 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c004030 con 0x55bdeeb172f0 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.532+0000 7f4c4dffb640 1 -- 172.21.15.60:0/823562362 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c5c0041d0 con 0x55bdeeb172f0 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.532+0000 7f4c4dffb640 1 -- 172.21.15.60:0/823562362 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c0116a0 con 0x55bdeeb172f0 2024-03-04T12:27:21.549 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.532+0000 7f4c709bd740 1 -- 172.21.15.60:0/823562362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 msgr2=0x55bdeeb1f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.532+0000 7f4c709bd740 1 --2- 172.21.15.60:0/823562362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1f990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00bcb0 tx=0x7f4c5c0107a0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.532+0000 7f4c709bd740 1 -- 172.21.15.60:0/823562362 shutdown_connections 2024-03-04T12:27:21.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.532+0000 7f4c709bd740 1 --2- 172.21.15.60:0/823562362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb222c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.532+0000 7f4c709bd740 1 --2- 172.21.15.60:0/823562362 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1f990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.532+0000 7f4c709bd740 1 --2- 172.21.15.60:0/823562362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb169d0 0x55bdeeb16db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.532+0000 7f4c709bd740 1 -- 172.21.15.60:0/823562362 >> 172.21.15.60:0/823562362 conn(0x55bdeeb11100 msgr2=0x55bdeeb114e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.532+0000 7f4c709bd740 1 -- 172.21.15.60:0/823562362 shutdown_connections 2024-03-04T12:27:21.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.532+0000 7f4c709bd740 1 -- 172.21.15.60:0/823562362 wait complete. 2024-03-04T12:27:21.550 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c709bd740 1 -- start start 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c709bd740 1 Processor -- start 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c709bd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb169d0 0x55bdeeb1ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c709bd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c709bd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb1d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c709bd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeea184a0 con 0x55bdeeb1fed0 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c709bd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeea75f60 con 0x55bdeeb172f0 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c709bd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeeb22cc0 con 0x55bdeeb169d0 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c709bd740 10 client.0 _ll_register_callbacks cb 0x55bdeea741c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c67fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb1d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c67fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb1d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34316/0 (socket says 172.21.15.60:34316) 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c67fff640 1 -- 172.21.15.60:0/2953867940 learned_addr learned my addr 172.21.15.60:0/2953867940 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c66ffd640 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c677fe640 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb169d0 0x55bdeeb1ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.551 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c67fff640 1 -- 172.21.15.60:0/2953867940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb169d0 msgr2=0x55bdeeb1ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c67fff640 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb169d0 0x55bdeeb1ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c67fff640 1 -- 172.21.15.60:0/2953867940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb172f0 msgr2=0x55bdeeb1d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.533+0000 7f4c67fff640 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c67fff640 1 -- 172.21.15.60:0/2953867940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bdee8587d0 con 0x55bdeeb1fed0 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c67fff640 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb1d930 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4c60013ce0 tx=0x7f4c60004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448c6255a6e77a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c46ffd640 1 -- 172.21.15.60:0/2953867940 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c60020620 con 0x55bdeeb1fed0 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c46ffd640 1 -- 172.21.15.60:0/2953867940 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f4c600207c0 con 0x55bdeeb1fed0 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c709bd740 1 -- 172.21.15.60:0/2953867940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c5c0041d0 con 0x55bdeeb1fed0 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c46ffd640 1 -- 172.21.15.60:0/2953867940 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c6000fe30 con 0x55bdeeb1fed0 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c709bd740 10 client.16287 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c709bd740 1 -- 172.21.15.60:0/2953867940 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55bdeeb2ed90 con 0x55bdeeb1fed0 2024-03-04T12:27:21.552 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c46ffd640 1 -- 172.21.15.60:0/2953867940 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c5c0041d0 con 0x55bdeeb1fed0 2024-03-04T12:27:21.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c46ffd640 1 -- 172.21.15.60:0/2953867940 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4c6000f040 con 0x55bdeeb1fed0 2024-03-04T12:27:21.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c709bd740 10 client.16287 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:21.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c709bd740 10 client.16287 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-04T12:27:21.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c709bd740 20 client.16287 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:21.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c709bd740 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bdeeb30250 0x55bdeeb32690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c709bd740 4 client.16287 mds_command: new command op to 24407 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:21.553 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.534+0000 7f4c709bd740 1 -- 172.21.15.60:0/2953867940 --> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bdeea87400 con 0x55bdeeb30250 2024-03-04T12:27:21.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.535+0000 7f4c677fe640 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bdeeb30250 0x55bdeeb32690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.535+0000 7f4c677fe640 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bdeeb30250 0x55bdeeb32690 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5d94f58e46b84d48 server_cookie=5efdc197cf6c25f9 in_seq=0 out_seq=0 2024-03-04T12:27:21.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.535+0000 7f4c46ffd640 10 client.16287 ms_handle_connect on v2:172.21.15.63:6832/2270964219 2024-03-04T12:27:21.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c46ffd640 1 -- 172.21.15.60:0/2953867940 <== mds.3 v2:172.21.15.63:6832/2270964219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55bdeea87400 con 0x55bdeeb30250 2024-03-04T12:27:21.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c46ffd640 10 client.16287 handle_command_reply: tid=0 2024-03-04T12:27:21.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c709bd740 1 client.16287 shutdown 2024-03-04T12:27:21.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c709bd740 1 -- 172.21.15.60:0/2953867940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb1fed0 msgr2=0x55bdeeb1d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c709bd740 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb1d930 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f4c60013ce0 tx=0x7f4c60004ab0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c709bd740 1 -- 172.21.15.60:0/2953867940 shutdown_connections 2024-03-04T12:27:21.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c709bd740 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bdeeb30250 0x55bdeeb32690 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.554 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c709bd740 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb1d930 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c709bd740 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c46ffd640 0 client.16287 ms_handle_reset on v2:172.21.15.63:6832/2270964219 2024-03-04T12:27:21.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c709bd740 1 --2- 172.21.15.60:0/2953867940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb169d0 0x55bdeeb1ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c709bd740 1 -- 172.21.15.60:0/2953867940 >> 172.21.15.60:0/2953867940 conn(0x55bdeeb11100 msgr2=0x55bdeeb12e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.555 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c709bd740 1 -- 172.21.15.60:0/2953867940 shutdown_connections 2024-03-04T12:27:21.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c709bd740 1 -- 172.21.15.60:0/2953867940 wait complete. 2024-03-04T12:27:21.556 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.536+0000 7f4c709bd740 20 client.16287 trim_cache size 0 max 16384 2024-03-04T12:27:21.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c709bd740 1 Processor -- start 2024-03-04T12:27:21.565 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c709bd740 1 -- start start 2024-03-04T12:27:21.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c709bd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb169d0 0x55bdeeb1f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c709bd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c709bd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb1bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c709bd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c681b0630 con 0x55bdeeb172f0 2024-03-04T12:27:21.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c709bd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c681b1fe0 con 0x55bdeeb1fed0 2024-03-04T12:27:21.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c709bd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c681b0c90 con 0x55bdeeb169d0 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34332/0 (socket says 172.21.15.60:34332) 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c66ffd640 1 -- 172.21.15.60:0/2654379609 learned_addr learned my addr 172.21.15.60:0/2654379609 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c67fff640 1 --2- 172.21.15.60:0/2654379609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb1bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c677fe640 1 --2- 172.21.15.60:0/2654379609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb169d0 0x55bdeeb1f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c66ffd640 1 -- 172.21.15.60:0/2654379609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb169d0 msgr2=0x55bdeeb1f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c66ffd640 1 --2- 172.21.15.60:0/2654379609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb169d0 0x55bdeeb1f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c66ffd640 1 -- 172.21.15.60:0/2654379609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb1fed0 msgr2=0x55bdeeb1bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c66ffd640 1 --2- 172.21.15.60:0/2654379609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb1bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c66ffd640 1 -- 172.21.15.60:0/2654379609 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdeeb17d40 con 0x55bdeeb172f0 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.548+0000 7f4c67fff640 1 --2- 172.21.15.60:0/2654379609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb1bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c66ffd640 1 --2- 172.21.15.60:0/2654379609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1b780 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00bcb0 tx=0x7f4c5c003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f618138775aac9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c4d7fa640 1 -- 172.21.15.60:0/2654379609 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c02b070 con 0x55bdeeb172f0 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c4d7fa640 1 -- 172.21.15.60:0/2654379609 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c5c00be90 con 0x55bdeeb172f0 2024-03-04T12:27:21.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c4d7fa640 1 -- 172.21.15.60:0/2654379609 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c011730 con 0x55bdeeb172f0 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c709bd740 1 -- 172.21.15.60:0/2654379609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 msgr2=0x55bdeeb1b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c709bd740 1 --2- 172.21.15.60:0/2654379609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1b780 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00bcb0 tx=0x7f4c5c003ea0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c709bd740 1 -- 172.21.15.60:0/2654379609 shutdown_connections 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c709bd740 1 --2- 172.21.15.60:0/2654379609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb1bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c709bd740 1 --2- 172.21.15.60:0/2654379609 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb172f0 0x55bdeeb1b780 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c709bd740 1 --2- 172.21.15.60:0/2654379609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb169d0 0x55bdeeb1f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c709bd740 1 -- 172.21.15.60:0/2654379609 >> 172.21.15.60:0/2654379609 conn(0x55bdeeb11100 msgr2=0x55bdeeb36ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c709bd740 1 -- 172.21.15.60:0/2654379609 shutdown_connections 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c709bd740 1 -- 172.21.15.60:0/2654379609 wait complete. 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.549+0000 7f4c709bd740 1 -- start start 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c709bd740 1 Processor -- start 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c709bd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb169d0 0x55bdeeb270a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c709bd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb172f0 0x55bdeeb25530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c709bd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb25a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c709bd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c681b1980 con 0x55bdeeb169d0 2024-03-04T12:27:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c709bd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c681b12f0 con 0x55bdeeb172f0 2024-03-04T12:27:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c709bd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c681ae8d0 con 0x55bdeeb1fed0 2024-03-04T12:27:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c709bd740 10 client.0 _ll_register_callbacks cb 0x7f4c5c00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c677fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb169d0 0x55bdeeb270a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c677fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb169d0 0x55bdeeb270a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34338/0 (socket says 172.21.15.60:34338) 2024-03-04T12:27:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c677fe640 1 -- 172.21.15.60:0/279855992 learned_addr learned my addr 172.21.15.60:0/279855992 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c67fff640 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb25a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c66ffd640 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb172f0 0x55bdeeb25530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c677fe640 1 -- 172.21.15.60:0/279855992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb1fed0 msgr2=0x55bdeeb25a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c677fe640 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb25a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c677fe640 1 -- 172.21.15.60:0/279855992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb172f0 msgr2=0x55bdeeb25530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c677fe640 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb172f0 0x55bdeeb25530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c677fe640 1 -- 172.21.15.60:0/279855992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bdeeb260b0 con 0x55bdeeb169d0 2024-03-04T12:27:21.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.550+0000 7f4c677fe640 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb169d0 0x55bdeeb270a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f4c58000c00 tx=0x7f4c580b7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37c2a9c6bb4ea7 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c46ffd640 1 -- 172.21.15.60:0/279855992 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5804b040 con 0x55bdeeb169d0 2024-03-04T12:27:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c46ffd640 1 -- 172.21.15.60:0/279855992 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f4c580b70b0 con 0x55bdeeb169d0 2024-03-04T12:27:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c709bd740 1 -- 172.21.15.60:0/279855992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4c5c00be90 con 0x55bdeeb169d0 2024-03-04T12:27:21.571 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c46ffd640 1 -- 172.21.15.60:0/279855992 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c580ac050 con 0x55bdeeb169d0 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c709bd740 10 client.16293 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c709bd740 1 -- 172.21.15.60:0/279855992 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55bdeeb875f0 con 0x55bdeeb169d0 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c46ffd640 1 -- 172.21.15.60:0/279855992 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c5c00be90 con 0x55bdeeb169d0 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c46ffd640 1 -- 172.21.15.60:0/279855992 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4c580d5120 con 0x55bdeeb169d0 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c709bd740 10 client.16293 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c709bd740 10 client.16293 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c709bd740 20 client.16293 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c709bd740 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bdeeb87ca0 0x55bdeeb8a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c709bd740 4 client.16293 mds_command: new command op to 24407 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.551+0000 7f4c709bd740 1 -- 172.21.15.60:0/279855992 --> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bdeea81750 con 0x55bdeeb87ca0 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c66ffd640 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bdeeb87ca0 0x55bdeeb8a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c66ffd640 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bdeeb87ca0 0x55bdeeb8a0c0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fcdb143e32b3ab12 server_cookie=f00933c7982bd685 in_seq=0 out_seq=0 2024-03-04T12:27:21.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c46ffd640 10 client.16293 ms_handle_connect on v2:172.21.15.63:6832/2270964219 2024-03-04T12:27:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c46ffd640 1 -- 172.21.15.60:0/279855992 <== mds.3 v2:172.21.15.63:6832/2270964219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bdeea81750 con 0x55bdeeb87ca0 2024-03-04T12:27:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c46ffd640 10 client.16293 handle_command_reply: tid=0 2024-03-04T12:27:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c709bd740 1 client.16293 shutdown 2024-03-04T12:27:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c709bd740 1 -- 172.21.15.60:0/279855992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb169d0 msgr2=0x55bdeeb270a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c709bd740 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb169d0 0x55bdeeb270a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f4c58000c00 tx=0x7f4c580b7040 comp rx=0 tx=0).stop 2024-03-04T12:27:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c709bd740 1 -- 172.21.15.60:0/279855992 shutdown_connections 2024-03-04T12:27:21.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c709bd740 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55bdeeb87ca0 0x55bdeeb8a0c0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c709bd740 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55bdeeb1fed0 0x55bdeeb25a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c709bd740 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bdeeb172f0 0x55bdeeb25530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c46ffd640 0 client.16293 ms_handle_reset on v2:172.21.15.63:6832/2270964219 2024-03-04T12:27:21.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c709bd740 1 --2- 172.21.15.60:0/279855992 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55bdeeb169d0 0x55bdeeb270a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c709bd740 1 -- 172.21.15.60:0/279855992 >> 172.21.15.60:0/279855992 conn(0x55bdeeb11100 msgr2=0x55bdeeb1b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c709bd740 1 -- 172.21.15.60:0/279855992 shutdown_connections 2024-03-04T12:27:21.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c709bd740 1 -- 172.21.15.60:0/279855992 wait complete. 2024-03-04T12:27:21.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.552+0000 7f4c709bd740 20 client.16293 trim_cache size 0 max 16384 2024-03-04T12:27:21.575 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:27:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.553+0000 7f4c6dd05640 1 -- 172.21.15.60:0/1307428280 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4c500a2010 msgr2=0x7f4c500a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.553+0000 7f4c6dd05640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4c500a2010 0x7f4c500a44d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4c60005fd0 tx=0x7f4c60009a50 comp rx=0 tx=0).stop 2024-03-04T12:27:21.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.553+0000 7f4c6dd05640 1 -- 172.21.15.60:0/1307428280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c681025a0 msgr2=0x7f4c681001f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.553+0000 7f4c6dd05640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c681025a0 0x7f4c681001f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f4c58005940 tx=0x7f4c5802cd10 comp rx=0 tx=0).stop 2024-03-04T12:27:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.553+0000 7f4c6dd05640 1 -- 172.21.15.60:0/1307428280 shutdown_connections 2024-03-04T12:27:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.553+0000 7f4c6dd05640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c6810ba10 0x7f4c68100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.553+0000 7f4c6dd05640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c68102ec0 0x7f4c68100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.553+0000 7f4c6dd05640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4c500a2010 0x7f4c500a44d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.553+0000 7f4c6dd05640 1 --2- 172.21.15.60:0/1307428280 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4c681025a0 0x7f4c681001f0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.553+0000 7f4c6dd05640 1 -- 172.21.15.60:0/1307428280 >> 172.21.15.60:0/1307428280 conn(0x7f4c680773a0 msgr2=0x7f4c680fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.553+0000 7f4c6dd05640 1 -- 172.21.15.60:0/1307428280 shutdown_connections 2024-03-04T12:27:21.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.553+0000 7f4c6dd05640 1 -- 172.21.15.60:0/1307428280 wait complete. 2024-03-04T12:27:21.577 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:27:21.577 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-04T12:27:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.677+0000 7f1703db9640 1 -- 172.21.15.60:0/2888676059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16fc101530 msgr2=0x7f16fc101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.677+0000 7f1703db9640 1 --2- 172.21.15.60:0/2888676059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16fc101530 0x7f16fc101910 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009900 tx=0x7f16f802bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.677+0000 7f1703db9640 1 -- 172.21.15.60:0/2888676059 shutdown_connections 2024-03-04T12:27:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.677+0000 7f1703db9640 1 --2- 172.21.15.60:0/2888676059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16fc106600 0x7f16fc10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.677+0000 7f1703db9640 1 --2- 172.21.15.60:0/2888676059 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16fc101ee0 0x7f16fc105f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.677+0000 7f1703db9640 1 --2- 172.21.15.60:0/2888676059 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16fc101530 0x7f16fc101910 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.677+0000 7f1703db9640 1 -- 172.21.15.60:0/2888676059 >> 172.21.15.60:0/2888676059 conn(0x7f16fc0fcdd0 msgr2=0x7f16fc0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.677+0000 7f1703db9640 1 -- 172.21.15.60:0/2888676059 shutdown_connections 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.677+0000 7f1703db9640 1 -- 172.21.15.60:0/2888676059 wait complete. 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1703db9640 1 Processor -- start 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1703db9640 1 -- start start 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1703db9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16fc101530 0x7f16fc19d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1703db9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16fc106600 0x7f16fc19dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1703db9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16fc1979e0 0x7f16fc197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1703db9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc1116c0 con 0x7f16fc106600 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1703db9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc111540 con 0x7f16fc1979e0 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1703db9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc111840 con 0x7f16fc101530 2024-03-04T12:27:21.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1701354640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16fc106600 0x7f16fc19dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1701354640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16fc106600 0x7f16fc19dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34356/0 (socket says 172.21.15.60:34356) 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1701354640 1 -- 172.21.15.60:0/4270349741 learned_addr learned my addr 172.21.15.60:0/4270349741 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1702356640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16fc1979e0 0x7f16fc197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1701b55640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16fc101530 0x7f16fc19d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1701354640 1 -- 172.21.15.60:0/4270349741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16fc101530 msgr2=0x7f16fc19d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1701354640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16fc101530 0x7f16fc19d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1701354640 1 -- 172.21.15.60:0/4270349741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16fc1979e0 msgr2=0x7f16fc197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1701354640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16fc1979e0 0x7f16fc197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.678+0000 7f1701354640 1 -- 172.21.15.60:0/4270349741 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16fc198500 con 0x7f16fc106600 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.679+0000 7f1701354640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16fc106600 0x7f16fc19dd70 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00e9c0 tx=0x7f16ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.679+0000 7f16f2ffd640 1 -- 172.21.15.60:0/4270349741 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec00cd80 con 0x7f16fc106600 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.679+0000 7f16f2ffd640 1 -- 172.21.15.60:0/4270349741 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16ec005500 con 0x7f16fc106600 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.679+0000 7f1703db9640 1 -- 172.21.15.60:0/4270349741 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16fc1987f0 con 0x7f16fc106600 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.679+0000 7f16f2ffd640 1 -- 172.21.15.60:0/4270349741 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec010690 con 0x7f16fc106600 2024-03-04T12:27:21.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.679+0000 7f1703db9640 1 -- 172.21.15.60:0/4270349741 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16fc1a4a40 con 0x7f16fc106600 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.681+0000 7f16f2ffd640 1 -- 172.21.15.60:0/4270349741 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f16ec0046c0 con 0x7f16fc106600 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1706998740 1 Processor -- start 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1706998740 1 -- start start 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1706998740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c969d0 0x55e503c96db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1706998740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503c9f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1706998740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c9fed0 0x55e503ca22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1706998740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e503c5ddc0 con 0x55e503c9fed0 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1706998740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e503bf9ec0 con 0x55e503c972f0 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1706998740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e503ba2d30 con 0x55e503c969d0 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f16f2ffd640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f16e80a2010 0x7f16e80a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1701b55640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503c9f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1701354640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c9fed0 0x55e503ca22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1701b55640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503c9f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:36314/0 (socket says 172.21.15.60:36314) 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1702356640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c969d0 0x55e503c96db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1701b55640 1 -- 172.21.15.60:0/3710277807 learned_addr learned my addr 172.21.15.60:0/3710277807 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.682+0000 7f1702356640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f16e80a2010 0x7f16e80a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.683+0000 7f16f2ffd640 1 -- 172.21.15.60:0/4270349741 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f16ec0cebe0 con 0x7f16fc106600 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.683+0000 7f1701b55640 1 -- 172.21.15.60:0/3710277807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c969d0 msgr2=0x55e503c96db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.683+0000 7f1701b55640 1 --2- 172.21.15.60:0/3710277807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c969d0 0x55e503c96db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.683+0000 7f1701b55640 1 -- 172.21.15.60:0/3710277807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c9fed0 msgr2=0x55e503ca22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.683+0000 7f1701b55640 1 --2- 172.21.15.60:0/3710277807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c9fed0 0x55e503ca22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.683+0000 7f1701b55640 1 -- 172.21.15.60:0/3710277807 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e50390e3f0 con 0x55e503c972f0 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.683+0000 7f1702356640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f16e80a2010 0x7f16e80a44d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005fd0 tx=0x7f16f4009500 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.683+0000 7f1701b55640 1 --2- 172.21.15.60:0/3710277807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503c9f990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f16f802c170 tx=0x7f16f8005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc805c70ec6bfe55 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.683+0000 7f16e7fff640 1 -- 172.21.15.60:0/3710277807 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8042070 con 0x55e503c972f0 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.683+0000 7f16e7fff640 1 -- 172.21.15.60:0/3710277807 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16f8004030 con 0x55e503c972f0 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.683+0000 7f16e7fff640 1 -- 172.21.15.60:0/3710277807 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8004ec0 con 0x55e503c972f0 2024-03-04T12:27:21.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.684+0000 7f1706998740 1 -- 172.21.15.60:0/3710277807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 msgr2=0x55e503c9f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.684+0000 7f1706998740 1 --2- 172.21.15.60:0/3710277807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503c9f990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f16f802c170 tx=0x7f16f8005be0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.684+0000 7f1706998740 1 -- 172.21.15.60:0/3710277807 shutdown_connections 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.684+0000 7f1706998740 1 --2- 172.21.15.60:0/3710277807 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c9fed0 0x55e503ca22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.684+0000 7f1706998740 1 --2- 172.21.15.60:0/3710277807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503c9f990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.684+0000 7f1706998740 1 --2- 172.21.15.60:0/3710277807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c969d0 0x55e503c96db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.684+0000 7f1706998740 1 -- 172.21.15.60:0/3710277807 >> 172.21.15.60:0/3710277807 conn(0x55e503c91100 msgr2=0x55e503c914e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.684+0000 7f1706998740 1 -- 172.21.15.60:0/3710277807 shutdown_connections 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.684+0000 7f1706998740 1 -- 172.21.15.60:0/3710277807 wait complete. 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.684+0000 7f1706998740 1 -- start start 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.684+0000 7f1706998740 1 Processor -- start 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1706998740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c969d0 0x55e503c9ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1706998740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503c9d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1706998740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c9fed0 0x55e503c9d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1706998740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e503b984a0 con 0x55e503c969d0 2024-03-04T12:27:21.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1706998740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e503bf5f60 con 0x55e503c972f0 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1706998740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e503ca2c60 con 0x55e503c9fed0 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1706998740 10 client.0 _ll_register_callbacks cb 0x55e503bf41c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1701b55640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c969d0 0x55e503c9ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1701b55640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c969d0 0x55e503c9ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34364/0 (socket says 172.21.15.60:34364) 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1701b55640 1 -- 172.21.15.60:0/726507425 learned_addr learned my addr 172.21.15.60:0/726507425 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1701354640 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503c9d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1702356640 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c9fed0 0x55e503c9d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1701b55640 1 -- 172.21.15.60:0/726507425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c9fed0 msgr2=0x55e503c9d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1701b55640 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c9fed0 0x55e503c9d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1701b55640 1 -- 172.21.15.60:0/726507425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 msgr2=0x55e503c9d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1701b55640 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503c9d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1701b55640 1 -- 172.21.15.60:0/726507425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5039d87d0 con 0x55e503c969d0 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1702356640 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c9fed0 0x55e503c9d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1701b55640 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c969d0 0x55e503c9ce90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f16f80044a0 tx=0x7f16f80044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9dab00aa14f5003 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f16e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f16e57fa640 1 -- 172.21.15.60:0/726507425 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f804e060 con 0x55e503c969d0 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f16e57fa640 1 -- 172.21.15.60:0/726507425 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f16f802ca90 con 0x55e503c969d0 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.685+0000 7f1706998740 1 -- 172.21.15.60:0/726507425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16f8004030 con 0x55e503c969d0 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.686+0000 7f16e57fa640 1 -- 172.21.15.60:0/726507425 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16f8055050 con 0x55e503c969d0 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.686+0000 7f1706998740 10 client.16305 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.686+0000 7f1706998740 1 -- 172.21.15.60:0/726507425 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55e503caed20 con 0x55e503c969d0 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.686+0000 7f16e57fa640 1 -- 172.21.15.60:0/726507425 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16f8004030 con 0x55e503c969d0 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.686+0000 7f16e57fa640 1 -- 172.21.15.60:0/726507425 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f16f8003710 con 0x55e503c969d0 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.686+0000 7f1706998740 10 client.16305 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.686+0000 7f1706998740 10 client.16305 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.686+0000 7f1706998740 20 client.16305 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.686+0000 7f1706998740 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55e503cb0070 0x55e503cb2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.686+0000 7f1706998740 4 client.16305 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.686+0000 7f1706998740 1 -- 172.21.15.60:0/726507425 --> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e503c07400 con 0x55e503cb0070 2024-03-04T12:27:21.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.686+0000 7f1701354640 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55e503cb0070 0x55e503cb2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.687+0000 7f1701354640 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55e503cb0070 0x55e503cb2460 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4a45747d02b0f5ea server_cookie=bf920f261dfbe653 in_seq=0 out_seq=0 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.687+0000 7f16e57fa640 10 client.16305 ms_handle_connect on v2:172.21.15.63:6836/3974574804 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.687+0000 7f16e57fa640 1 -- 172.21.15.60:0/726507425 <== mds.2 v2:172.21.15.63:6836/3974574804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e503c07400 con 0x55e503cb0070 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.687+0000 7f16e57fa640 10 client.16305 handle_command_reply: tid=0 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.687+0000 7f1706998740 1 client.16305 shutdown 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.687+0000 7f1706998740 1 -- 172.21.15.60:0/726507425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c969d0 msgr2=0x55e503c9ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.687+0000 7f1706998740 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c969d0 0x55e503c9ce90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f16f80044a0 tx=0x7f16f80044d0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.688+0000 7f1706998740 1 -- 172.21.15.60:0/726507425 shutdown_connections 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.688+0000 7f1706998740 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55e503cb0070 0x55e503cb2460 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.688+0000 7f1706998740 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c9fed0 0x55e503c9d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.688+0000 7f1706998740 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503c9d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.688+0000 7f1706998740 1 --2- 172.21.15.60:0/726507425 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c969d0 0x55e503c9ce90 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.688+0000 7f1706998740 1 -- 172.21.15.60:0/726507425 >> 172.21.15.60:0/726507425 conn(0x55e503c91100 msgr2=0x55e503c92e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.688+0000 7f16e57fa640 0 client.16305 ms_handle_reset on v2:172.21.15.63:6836/3974574804 2024-03-04T12:27:21.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.688+0000 7f1706998740 1 -- 172.21.15.60:0/726507425 shutdown_connections 2024-03-04T12:27:21.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.688+0000 7f1706998740 1 -- 172.21.15.60:0/726507425 wait complete. 2024-03-04T12:27:21.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.688+0000 7f1706998740 20 client.16305 trim_cache size 0 max 16384 2024-03-04T12:27:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1706998740 1 Processor -- start 2024-03-04T12:27:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1706998740 1 -- start start 2024-03-04T12:27:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1706998740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c969d0 0x55e503c9f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1706998740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c972f0 0x55e503c9a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1706998740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c9fed0 0x55e503c9a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1706998740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc1ae3d0 con 0x55e503c972f0 2024-03-04T12:27:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1706998740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc1afd80 con 0x55e503c969d0 2024-03-04T12:27:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1706998740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc1aea30 con 0x55e503c9fed0 2024-03-04T12:27:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1701354640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c972f0 0x55e503c9a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1701354640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c972f0 0x55e503c9a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34366/0 (socket says 172.21.15.60:34366) 2024-03-04T12:27:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1701354640 1 -- 172.21.15.60:0/2932892355 learned_addr learned my addr 172.21.15.60:0/2932892355 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.709 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1702356640 1 --2- 172.21.15.60:0/2932892355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c9fed0 0x55e503c9a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.700+0000 7f1701b55640 1 --2- 172.21.15.60:0/2932892355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c969d0 0x55e503c9f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.701+0000 7f1701354640 1 -- 172.21.15.60:0/2932892355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c9fed0 msgr2=0x55e503c9a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.701+0000 7f1701354640 1 --2- 172.21.15.60:0/2932892355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c9fed0 0x55e503c9a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.701+0000 7f1701354640 1 -- 172.21.15.60:0/2932892355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c969d0 msgr2=0x55e503c9f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.701+0000 7f1701354640 1 --2- 172.21.15.60:0/2932892355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c969d0 0x55e503c9f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.701+0000 7f1701354640 1 -- 172.21.15.60:0/2932892355 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e503c90f30 con 0x55e503c972f0 2024-03-04T12:27:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.701+0000 7f1702356640 1 --2- 172.21.15.60:0/2932892355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c9fed0 0x55e503c9a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.701+0000 7f1701b55640 1 --2- 172.21.15.60:0/2932892355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c969d0 0x55e503c9f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.701+0000 7f1701354640 1 --2- 172.21.15.60:0/2932892355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c972f0 0x55e503c9a0f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f16ec097950 tx=0x7f16ec097220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab454e4ff931cd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.701+0000 7f16e77fe640 1 -- 172.21.15.60:0/2932892355 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec09f0d0 con 0x55e503c972f0 2024-03-04T12:27:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.701+0000 7f16e77fe640 1 -- 172.21.15.60:0/2932892355 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16ec0a3720 con 0x55e503c972f0 2024-03-04T12:27:21.710 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.701+0000 7f16e77fe640 1 -- 172.21.15.60:0/2932892355 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0cc090 con 0x55e503c972f0 2024-03-04T12:27:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 -- 172.21.15.60:0/2932892355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c972f0 msgr2=0x55e503c9a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 --2- 172.21.15.60:0/2932892355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c972f0 0x55e503c9a0f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f16ec097950 tx=0x7f16ec097220 comp rx=0 tx=0).stop 2024-03-04T12:27:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 -- 172.21.15.60:0/2932892355 shutdown_connections 2024-03-04T12:27:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 --2- 172.21.15.60:0/2932892355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c9fed0 0x55e503c9a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 --2- 172.21.15.60:0/2932892355 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c972f0 0x55e503c9a0f0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 --2- 172.21.15.60:0/2932892355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c969d0 0x55e503c9f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 -- 172.21.15.60:0/2932892355 >> 172.21.15.60:0/2932892355 conn(0x55e503c91100 msgr2=0x55e503c98170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 -- 172.21.15.60:0/2932892355 shutdown_connections 2024-03-04T12:27:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 -- 172.21.15.60:0/2932892355 wait complete. 2024-03-04T12:27:21.716 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 -- start start 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 Processor -- start 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c969d0 0x55e503ca71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.702+0000 7f1706998740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503ca5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1706998740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c9fed0 0x55e503ca5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1706998740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc1af720 con 0x55e503c9fed0 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1706998740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc1af090 con 0x55e503c972f0 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1706998740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc1ac670 con 0x55e503c969d0 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1706998740 10 client.0 _ll_register_callbacks cb 0x7f16f40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1702356640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c9fed0 0x55e503ca5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1702356640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c9fed0 0x55e503ca5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34378/0 (socket says 172.21.15.60:34378) 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1702356640 1 -- 172.21.15.60:0/1232414057 learned_addr learned my addr 172.21.15.60:0/1232414057 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1701b55640 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c969d0 0x55e503ca71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1701354640 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503ca5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1702356640 1 -- 172.21.15.60:0/1232414057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c969d0 msgr2=0x55e503ca71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.717 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1702356640 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c969d0 0x55e503ca71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1702356640 1 -- 172.21.15.60:0/1232414057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 msgr2=0x55e503ca5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1702356640 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503ca5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1702356640 1 -- 172.21.15.60:0/1232414057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e503ca6280 con 0x55e503c9fed0 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1702356640 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c9fed0 0x55e503ca5ad0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009e40 tx=0x7f16f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102effc2ba76c7f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f16e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f16e57fa640 1 -- 172.21.15.60:0/1232414057 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4030060 con 0x55e503c9fed0 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f1706998740 1 -- 172.21.15.60:0/1232414057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16ec0a3720 con 0x55e503c9fed0 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.703+0000 7f16e57fa640 1 -- 172.21.15.60:0/1232414057 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f16f400fdb0 con 0x55e503c9fed0 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.704+0000 7f16e57fa640 1 -- 172.21.15.60:0/1232414057 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16f4037050 con 0x55e503c9fed0 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.704+0000 7f1706998740 10 client.16311 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.704+0000 7f1706998740 1 -- 172.21.15.60:0/1232414057 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55e503d07210 con 0x55e503c9fed0 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.704+0000 7f16e57fa640 1 -- 172.21.15.60:0/1232414057 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16ec0a3720 con 0x55e503c9fed0 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.704+0000 7f16e57fa640 1 -- 172.21.15.60:0/1232414057 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f16f401d070 con 0x55e503c9fed0 2024-03-04T12:27:21.718 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.704+0000 7f1706998740 10 client.16311 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:21.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.704+0000 7f1706998740 10 client.16311 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-04T12:27:21.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.704+0000 7f1706998740 20 client.16311 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:21.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.704+0000 7f1706998740 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55e503d07920 0x55e503d09d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.704+0000 7f1706998740 4 client.16311 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:27:21.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.704+0000 7f1706998740 1 -- 172.21.15.60:0/1232414057 --> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e503c01750 con 0x55e503d07920 2024-03-04T12:27:21.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.704+0000 7f1701b55640 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55e503d07920 0x55e503d09d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.705+0000 7f16e57fa640 10 client.16311 ms_handle_connect on v2:172.21.15.63:6836/3974574804 2024-03-04T12:27:21.719 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.705+0000 7f1701b55640 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55e503d07920 0x55e503d09d40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c1daa69c6c79546d server_cookie=9d2159e57ca6d98f in_seq=0 out_seq=0 2024-03-04T12:27:21.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f16e57fa640 1 -- 172.21.15.60:0/1232414057 <== mds.2 v2:172.21.15.63:6836/3974574804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e503c01750 con 0x55e503d07920 2024-03-04T12:27:21.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f16e57fa640 10 client.16311 handle_command_reply: tid=0 2024-03-04T12:27:21.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f1706998740 1 client.16311 shutdown 2024-03-04T12:27:21.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f1706998740 1 -- 172.21.15.60:0/1232414057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c9fed0 msgr2=0x55e503ca5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.772 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f1706998740 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c9fed0 0x55e503ca5ad0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009e40 tx=0x7f16f4004a40 comp rx=0 tx=0).stop 2024-03-04T12:27:21.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f1706998740 1 -- 172.21.15.60:0/1232414057 shutdown_connections 2024-03-04T12:27:21.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f1706998740 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x55e503d07920 0x55e503d09d40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f1706998740 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55e503c9fed0 0x55e503ca5ad0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f1706998740 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e503c972f0 0x55e503ca5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f16e57fa640 0 client.16311 ms_handle_reset on v2:172.21.15.63:6836/3974574804 2024-03-04T12:27:21.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f1706998740 1 --2- 172.21.15.60:0/1232414057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e503c969d0 0x55e503ca71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f1706998740 1 -- 172.21.15.60:0/1232414057 >> 172.21.15.60:0/1232414057 conn(0x55e503c91100 msgr2=0x55e503c9b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f1706998740 1 -- 172.21.15.60:0/1232414057 shutdown_connections 2024-03-04T12:27:21.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f1706998740 1 -- 172.21.15.60:0/1232414057 wait complete. 2024-03-04T12:27:21.773 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.764+0000 7f1706998740 20 client.16311 trim_cache size 0 max 16384 2024-03-04T12:27:21.773 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:27:21.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.765+0000 7f1703db9640 1 -- 172.21.15.60:0/4270349741 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f16e80a2010 msgr2=0x7f16e80a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.765+0000 7f1703db9640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f16e80a2010 0x7f16e80a44d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f16f4005fd0 tx=0x7f16f4009500 comp rx=0 tx=0).stop 2024-03-04T12:27:21.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.765+0000 7f1703db9640 1 -- 172.21.15.60:0/4270349741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16fc106600 msgr2=0x7f16fc19dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.765+0000 7f1703db9640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16fc106600 0x7f16fc19dd70 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00e9c0 tx=0x7f16ec00ee80 comp rx=0 tx=0).stop 2024-03-04T12:27:21.774 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.765+0000 7f1703db9640 1 -- 172.21.15.60:0/4270349741 shutdown_connections 2024-03-04T12:27:21.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.765+0000 7f1703db9640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16fc1979e0 0x7f16fc197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.765+0000 7f1703db9640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f16e80a2010 0x7f16e80a44d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.765+0000 7f1703db9640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f16fc106600 0x7f16fc19dd70 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.765+0000 7f1703db9640 1 --2- 172.21.15.60:0/4270349741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f16fc101530 0x7f16fc19d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.765+0000 7f1703db9640 1 -- 172.21.15.60:0/4270349741 >> 172.21.15.60:0/4270349741 conn(0x7f16fc0fcdd0 msgr2=0x7f16fc0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.765+0000 7f1703db9640 1 -- 172.21.15.60:0/4270349741 shutdown_connections 2024-03-04T12:27:21.775 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.765+0000 7f1703db9640 1 -- 172.21.15.60:0/4270349741 wait complete. 2024-03-04T12:27:21.783 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:27:21.783 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-04T12:27:21.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.881+0000 7fd40e174640 1 -- 172.21.15.60:0/243355432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd408105510 msgr2=0x7fd40810d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.881+0000 7fd40e174640 1 --2- 172.21.15.60:0/243355432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd408105510 0x7fd40810d0e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd400009be0 tx=0x7fd40002bef0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.881+0000 7fd40e174640 1 -- 172.21.15.60:0/243355432 shutdown_connections 2024-03-04T12:27:21.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.881+0000 7fd40e174640 1 --2- 172.21.15.60:0/243355432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd408105510 0x7fd40810d0e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.881+0000 7fd40e174640 1 --2- 172.21.15.60:0/243355432 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd408100df0 0x7fd408104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.881+0000 7fd40e174640 1 --2- 172.21.15.60:0/243355432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd408100440 0x7fd408100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.881+0000 7fd40e174640 1 -- 172.21.15.60:0/243355432 >> 172.21.15.60:0/243355432 conn(0x7fd4080773e0 msgr2=0x7fd4080777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.881+0000 7fd40e174640 1 -- 172.21.15.60:0/243355432 shutdown_connections 2024-03-04T12:27:21.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.881+0000 7fd40e174640 1 -- 172.21.15.60:0/243355432 wait complete. 2024-03-04T12:27:21.891 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.881+0000 7fd40e174640 1 Processor -- start 2024-03-04T12:27:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.881+0000 7fd40e174640 1 -- start start 2024-03-04T12:27:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd40e174640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd408100440 0x7fd40819d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd40e174640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd408105510 0x7fd40819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd40e174640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4081979a0 0x7fd408197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd40e174640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081105d0 con 0x7fd4081979a0 2024-03-04T12:27:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd40e174640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd408110450 con 0x7fd408100440 2024-03-04T12:27:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd40e174640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd408110750 con 0x7fd408105510 2024-03-04T12:27:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd407fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4081979a0 0x7fd408197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd4077fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd408100440 0x7fd40819d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.892 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd407fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4081979a0 0x7fd408197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34402/0 (socket says 172.21.15.60:34402) 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd407fff640 1 -- 172.21.15.60:0/103547972 learned_addr learned my addr 172.21.15.60:0/103547972 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd406ffd640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd408105510 0x7fd40819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd407fff640 1 -- 172.21.15.60:0/103547972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd408105510 msgr2=0x7fd40819dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd407fff640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd408105510 0x7fd40819dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd407fff640 1 -- 172.21.15.60:0/103547972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd408100440 msgr2=0x7fd40819d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd407fff640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd408100440 0x7fd40819d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd407fff640 1 -- 172.21.15.60:0/103547972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4081984c0 con 0x7fd4081979a0 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd4077fe640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd408100440 0x7fd40819d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd407fff640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4081979a0 0x7fd408197e50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd4000096f0 tx=0x7fd400033480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd404ff9640 1 -- 172.21.15.60:0/103547972 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400043020 con 0x7fd4081979a0 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd40e174640 1 -- 172.21.15.60:0/103547972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd408198750 con 0x7fd4081979a0 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.882+0000 7fd404ff9640 1 -- 172.21.15.60:0/103547972 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd400003e50 con 0x7fd4081979a0 2024-03-04T12:27:21.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.883+0000 7fd40e174640 1 -- 172.21.15.60:0/103547972 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4081a49f0 con 0x7fd4081979a0 2024-03-04T12:27:21.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.883+0000 7fd404ff9640 1 -- 172.21.15.60:0/103547972 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40003e450 con 0x7fd4081979a0 2024-03-04T12:27:21.900 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd404ff9640 1 -- 172.21.15.60:0/103547972 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7fd400046020 con 0x7fd4081979a0 2024-03-04T12:27:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd410e2b740 1 Processor -- start 2024-03-04T12:27:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd410e2b740 1 -- start start 2024-03-04T12:27:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd404ff9640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd3f00a2010 0x7fd3f00a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd410e2b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667c9d0 0x55ad2667cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd410e2b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667d2f0 0x55ad26685990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd410e2b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad26685ed0 0x55ad266882c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd410e2b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad26643dc0 con 0x55ad2667c9d0 2024-03-04T12:27:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd410e2b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad265dfec0 con 0x55ad2667d2f0 2024-03-04T12:27:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd410e2b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad26588d30 con 0x55ad26685ed0 2024-03-04T12:27:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd406ffd640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd3f00a2010 0x7fd3f00a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.901 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd404ff9640 1 -- 172.21.15.60:0/103547972 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fd4000f0b30 con 0x7fd4081979a0 2024-03-04T12:27:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd406ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667d2f0 0x55ad26685990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd4077fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667c9d0 0x55ad2667cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd4077fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667c9d0 0x55ad2667cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34414/0 (socket says 172.21.15.60:34414) 2024-03-04T12:27:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd4077fe640 1 -- 172.21.15.60:0/1122693181 learned_addr learned my addr 172.21.15.60:0/1122693181 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.885+0000 7fd407fff640 1 --2- 172.21.15.60:0/1122693181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad26685ed0 0x55ad266882c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.886+0000 7fd406ffd640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd3f00a2010 0x7fd3f00a44d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0096c0 tx=0x7fd3fc009610 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.886+0000 7fd406ffd640 1 -- 172.21.15.60:0/1122693181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad26685ed0 msgr2=0x55ad266882c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.886+0000 7fd406ffd640 1 --2- 172.21.15.60:0/1122693181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad26685ed0 0x55ad266882c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.886+0000 7fd406ffd640 1 -- 172.21.15.60:0/1122693181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667c9d0 msgr2=0x55ad2667cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.886+0000 7fd406ffd640 1 --2- 172.21.15.60:0/1122693181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667c9d0 0x55ad2667cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.886+0000 7fd406ffd640 1 -- 172.21.15.60:0/1122693181 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad262f43f0 con 0x55ad2667d2f0 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.886+0000 7fd4077fe640 1 --2- 172.21.15.60:0/1122693181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667c9d0 0x55ad2667cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.886+0000 7fd406ffd640 1 --2- 172.21.15.60:0/1122693181 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667d2f0 0x55ad26685990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd408199190 tx=0x7fd3fc020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a370d15379bcf17a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.886+0000 7fd3edffb640 1 -- 172.21.15.60:0/1122693181 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0130d0 con 0x55ad2667d2f0 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.886+0000 7fd3edffb640 1 -- 172.21.15.60:0/1122693181 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3fc024070 con 0x55ad2667d2f0 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.886+0000 7fd3edffb640 1 -- 172.21.15.60:0/1122693181 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc004e20 con 0x55ad2667d2f0 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.887+0000 7fd410e2b740 1 -- 172.21.15.60:0/1122693181 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667d2f0 msgr2=0x55ad26685990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.887+0000 7fd410e2b740 1 --2- 172.21.15.60:0/1122693181 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667d2f0 0x55ad26685990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd408199190 tx=0x7fd3fc020000 comp rx=0 tx=0).stop 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.887+0000 7fd410e2b740 1 -- 172.21.15.60:0/1122693181 shutdown_connections 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.887+0000 7fd410e2b740 1 --2- 172.21.15.60:0/1122693181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad26685ed0 0x55ad266882c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.887+0000 7fd410e2b740 1 --2- 172.21.15.60:0/1122693181 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667d2f0 0x55ad26685990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.887+0000 7fd410e2b740 1 --2- 172.21.15.60:0/1122693181 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667c9d0 0x55ad2667cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.887+0000 7fd410e2b740 1 -- 172.21.15.60:0/1122693181 >> 172.21.15.60:0/1122693181 conn(0x55ad26677100 msgr2=0x55ad266774e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.887+0000 7fd410e2b740 1 -- 172.21.15.60:0/1122693181 shutdown_connections 2024-03-04T12:27:21.903 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.887+0000 7fd410e2b740 1 -- 172.21.15.60:0/1122693181 wait complete. 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.887+0000 7fd410e2b740 1 -- start start 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.887+0000 7fd410e2b740 1 Processor -- start 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd410e2b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad2667c9d0 0x55ad26682f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd410e2b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 0x55ad266834a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd410e2b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad26685ed0 0x55ad266839e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd410e2b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad2657e4a0 con 0x55ad2667d2f0 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd410e2b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad265dbf60 con 0x55ad26685ed0 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd410e2b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad26688c60 con 0x55ad2667c9d0 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd410e2b740 10 client.0 _ll_register_callbacks cb 0x55ad265da1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd406ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 0x55ad266834a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd406ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 0x55ad266834a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34416/0 (socket says 172.21.15.60:34416) 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd406ffd640 1 -- 172.21.15.60:0/2528011841 learned_addr learned my addr 172.21.15.60:0/2528011841 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd407fff640 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad26685ed0 0x55ad266839e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.904 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd4077fe640 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad2667c9d0 0x55ad26682f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd406ffd640 1 -- 172.21.15.60:0/2528011841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad2667c9d0 msgr2=0x55ad26682f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd406ffd640 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad2667c9d0 0x55ad26682f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd406ffd640 1 -- 172.21.15.60:0/2528011841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad26685ed0 msgr2=0x55ad266839e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd406ffd640 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad26685ed0 0x55ad266839e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd406ffd640 1 -- 172.21.15.60:0/2528011841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad263be7d0 con 0x55ad2667d2f0 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd407fff640 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad26685ed0 0x55ad266839e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd406ffd640 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 0x55ad266834a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0048f0 tx=0x7fd3fc004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86a5fb4fdbfcba server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd3e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd3e6ffd640 1 -- 172.21.15.60:0/2528011841 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc020980 con 0x55ad2667d2f0 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd410e2b740 1 -- 172.21.15.60:0/2528011841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd3fc024070 con 0x55ad2667d2f0 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.888+0000 7fd3e6ffd640 1 -- 172.21.15.60:0/2528011841 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fd3fc020b20 con 0x55ad2667d2f0 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.889+0000 7fd3e6ffd640 1 -- 172.21.15.60:0/2528011841 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3fc020d20 con 0x55ad2667d2f0 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.889+0000 7fd410e2b740 10 client.16323 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:21.905 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.889+0000 7fd410e2b740 1 -- 172.21.15.60:0/2528011841 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55ad26694de0 con 0x55ad2667d2f0 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.889+0000 7fd3e6ffd640 1 -- 172.21.15.60:0/2528011841 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3fc024070 con 0x55ad2667d2f0 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.889+0000 7fd3e6ffd640 1 -- 172.21.15.60:0/2528011841 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fd3fc01f070 con 0x55ad2667d2f0 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.889+0000 7fd410e2b740 10 client.16323 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.889+0000 7fd410e2b740 10 client.16323 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.889+0000 7fd410e2b740 20 client.16323 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.889+0000 7fd410e2b740 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55ad26696240 0x55ad26698630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.889+0000 7fd410e2b740 4 client.16323 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.889+0000 7fd410e2b740 1 -- 172.21.15.60:0/2528011841 --> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad265ed400 con 0x55ad26696240 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.889+0000 7fd4077fe640 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55ad26696240 0x55ad26698630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.890+0000 7fd4077fe640 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55ad26696240 0x55ad26698630 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=55e19f3947b3d1d0 server_cookie=ca19a77d8c25d4c in_seq=0 out_seq=0 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.890+0000 7fd3e6ffd640 10 client.16323 ms_handle_connect on v2:172.21.15.86:6838/3347800322 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.890+0000 7fd3e6ffd640 1 -- 172.21.15.60:0/2528011841 <== mds.1 v2:172.21.15.86:6838/3347800322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ad265ed400 con 0x55ad26696240 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.890+0000 7fd3e6ffd640 10 client.16323 handle_command_reply: tid=0 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.890+0000 7fd410e2b740 1 client.16323 shutdown 2024-03-04T12:27:21.906 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.891+0000 7fd410e2b740 1 -- 172.21.15.60:0/2528011841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 msgr2=0x55ad266834a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.891+0000 7fd410e2b740 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 0x55ad266834a0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0048f0 tx=0x7fd3fc004920 comp rx=0 tx=0).stop 2024-03-04T12:27:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.891+0000 7fd410e2b740 1 -- 172.21.15.60:0/2528011841 shutdown_connections 2024-03-04T12:27:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.891+0000 7fd410e2b740 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad26685ed0 0x55ad266839e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.891+0000 7fd410e2b740 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55ad26696240 0x55ad26698630 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.891+0000 7fd410e2b740 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 0x55ad266834a0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.891+0000 7fd410e2b740 1 --2- 172.21.15.60:0/2528011841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad2667c9d0 0x55ad26682f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.891+0000 7fd3e6ffd640 0 client.16323 ms_handle_reset on v2:172.21.15.86:6838/3347800322 2024-03-04T12:27:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.891+0000 7fd410e2b740 1 -- 172.21.15.60:0/2528011841 >> 172.21.15.60:0/2528011841 conn(0x55ad26677100 msgr2=0x55ad26678e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.891+0000 7fd410e2b740 1 -- 172.21.15.60:0/2528011841 shutdown_connections 2024-03-04T12:27:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.891+0000 7fd410e2b740 1 -- 172.21.15.60:0/2528011841 wait complete. 2024-03-04T12:27:21.907 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.891+0000 7fd410e2b740 20 client.16323 trim_cache size 0 max 16384 2024-03-04T12:27:21.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.902+0000 7fd410e2b740 1 Processor -- start 2024-03-04T12:27:21.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.902+0000 7fd410e2b740 1 -- start start 2024-03-04T12:27:21.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.902+0000 7fd410e2b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad2667c9d0 0x55ad26685690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd410e2b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667d2f0 0x55ad266801c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd410e2b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad26685ed0 0x55ad26680700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd410e2b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081ae460 con 0x55ad26685ed0 2024-03-04T12:27:21.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd410e2b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081afe10 con 0x55ad2667d2f0 2024-03-04T12:27:21.916 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd410e2b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081aeac0 con 0x55ad2667c9d0 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd407fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad26685ed0 0x55ad26680700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd407fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad26685ed0 0x55ad26680700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34428/0 (socket says 172.21.15.60:34428) 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd407fff640 1 -- 172.21.15.60:0/2882839724 learned_addr learned my addr 172.21.15.60:0/2882839724 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd406ffd640 1 --2- 172.21.15.60:0/2882839724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667d2f0 0x55ad266801c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd4077fe640 1 --2- 172.21.15.60:0/2882839724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad2667c9d0 0x55ad26685690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd407fff640 1 -- 172.21.15.60:0/2882839724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad2667c9d0 msgr2=0x55ad26685690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd407fff640 1 --2- 172.21.15.60:0/2882839724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad2667c9d0 0x55ad26685690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd407fff640 1 -- 172.21.15.60:0/2882839724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667d2f0 msgr2=0x55ad266801c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd407fff640 1 --2- 172.21.15.60:0/2882839724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667d2f0 0x55ad266801c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd407fff640 1 -- 172.21.15.60:0/2882839724 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad2667d0f0 con 0x55ad26685ed0 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd407fff640 1 --2- 172.21.15.60:0/2882839724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad26685ed0 0x55ad26680700 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd4000b94a0 tx=0x7fd400009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad618c9d095f95a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd3ed7fa640 1 -- 172.21.15.60:0/2882839724 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40000ab20 con 0x55ad26685ed0 2024-03-04T12:27:21.917 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd3ed7fa640 1 -- 172.21.15.60:0/2882839724 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4000bd740 con 0x55ad26685ed0 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.903+0000 7fd3ed7fa640 1 -- 172.21.15.60:0/2882839724 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd400004160 con 0x55ad26685ed0 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.904+0000 7fd410e2b740 1 -- 172.21.15.60:0/2882839724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad26685ed0 msgr2=0x55ad26680700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.904+0000 7fd410e2b740 1 --2- 172.21.15.60:0/2882839724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad26685ed0 0x55ad26680700 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd4000b94a0 tx=0x7fd400009500 comp rx=0 tx=0).stop 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.904+0000 7fd410e2b740 1 -- 172.21.15.60:0/2882839724 shutdown_connections 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.904+0000 7fd410e2b740 1 --2- 172.21.15.60:0/2882839724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad26685ed0 0x55ad26680700 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.904+0000 7fd410e2b740 1 --2- 172.21.15.60:0/2882839724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667d2f0 0x55ad266801c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.904+0000 7fd410e2b740 1 --2- 172.21.15.60:0/2882839724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad2667c9d0 0x55ad26685690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.904+0000 7fd410e2b740 1 -- 172.21.15.60:0/2882839724 >> 172.21.15.60:0/2882839724 conn(0x55ad26677100 msgr2=0x55ad26681830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.904+0000 7fd410e2b740 1 -- 172.21.15.60:0/2882839724 shutdown_connections 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.904+0000 7fd410e2b740 1 -- 172.21.15.60:0/2882839724 wait complete. 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.904+0000 7fd410e2b740 1 -- start start 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.904+0000 7fd410e2b740 1 Processor -- start 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd410e2b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667c9d0 0x55ad2668ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.918 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd410e2b740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 0x55ad2668bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd410e2b740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad26685ed0 0x55ad2668c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd410e2b740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081af7b0 con 0x55ad2667d2f0 2024-03-04T12:27:21.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd410e2b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081af120 con 0x55ad2667c9d0 2024-03-04T12:27:21.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd410e2b740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081ac700 con 0x55ad26685ed0 2024-03-04T12:27:21.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd410e2b740 10 client.0 _ll_register_callbacks cb 0x7fd3f8009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:21.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd406ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 0x55ad2668bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd406ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 0x55ad2668bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34442/0 (socket says 172.21.15.60:34442) 2024-03-04T12:27:21.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd4077fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667c9d0 0x55ad2668ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd406ffd640 1 -- 172.21.15.60:0/430405991 learned_addr learned my addr 172.21.15.60:0/430405991 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:21.919 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd407fff640 1 --2- 172.21.15.60:0/430405991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad26685ed0 0x55ad2668c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd406ffd640 1 -- 172.21.15.60:0/430405991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad26685ed0 msgr2=0x55ad2668c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd406ffd640 1 --2- 172.21.15.60:0/430405991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad26685ed0 0x55ad2668c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd406ffd640 1 -- 172.21.15.60:0/430405991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667c9d0 msgr2=0x55ad2668ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd406ffd640 1 --2- 172.21.15.60:0/430405991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667c9d0 0x55ad2668ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd406ffd640 1 -- 172.21.15.60:0/430405991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad266ec580 con 0x55ad2667d2f0 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd406ffd640 1 --2- 172.21.15.60:0/430405991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 0x55ad2668bfb0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x55ad266827c0 tx=0x7fd3fc005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff581c6766050479 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd3e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd3e6ffd640 1 -- 172.21.15.60:0/430405991 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc02aa90 con 0x55ad2667d2f0 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd3e6ffd640 1 -- 172.21.15.60:0/430405991 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fd3fc02ac30 con 0x55ad2667d2f0 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd410e2b740 1 -- 172.21.15.60:0/430405991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4000bd740 con 0x55ad2667d2f0 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd3e6ffd640 1 -- 172.21.15.60:0/430405991 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3fc045050 con 0x55ad2667d2f0 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd410e2b740 10 client.16329 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.905+0000 7fd410e2b740 1 -- 172.21.15.60:0/430405991 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55ad266ed860 con 0x55ad2667d2f0 2024-03-04T12:27:21.920 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.906+0000 7fd3e6ffd640 1 -- 172.21.15.60:0/430405991 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4000bd740 con 0x55ad2667d2f0 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.906+0000 7fd3e6ffd640 1 -- 172.21.15.60:0/430405991 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fd3fc01f040 con 0x55ad2667d2f0 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.906+0000 7fd410e2b740 10 client.16329 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.906+0000 7fd410e2b740 10 client.16329 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.906+0000 7fd410e2b740 20 client.16329 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.906+0000 7fd410e2b740 1 --2- 172.21.15.60:0/430405991 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55ad266ede70 0x55ad266f0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.906+0000 7fd410e2b740 4 client.16329 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.906+0000 7fd410e2b740 1 -- 172.21.15.60:0/430405991 --> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ad265e7750 con 0x55ad266ede70 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.906+0000 7fd4077fe640 1 --2- 172.21.15.60:0/430405991 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55ad266ede70 0x55ad266f0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.906+0000 7fd3e6ffd640 10 client.16329 ms_handle_connect on v2:172.21.15.86:6838/3347800322 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.906+0000 7fd4077fe640 1 --2- 172.21.15.60:0/430405991 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55ad266ede70 0x55ad266f0290 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8a1005aa9b8a3bf2 server_cookie=ee73ba1125ae42a4 in_seq=0 out_seq=0 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd3e6ffd640 1 -- 172.21.15.60:0/430405991 <== mds.1 v2:172.21.15.86:6838/3347800322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ad265e7750 con 0x55ad266ede70 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd3e6ffd640 10 client.16329 handle_command_reply: tid=0 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd410e2b740 1 client.16329 shutdown 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd410e2b740 1 -- 172.21.15.60:0/430405991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 msgr2=0x55ad2668bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.921 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd410e2b740 1 --2- 172.21.15.60:0/430405991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 0x55ad2668bfb0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x55ad266827c0 tx=0x7fd3fc005df0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd410e2b740 1 -- 172.21.15.60:0/430405991 shutdown_connections 2024-03-04T12:27:21.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd410e2b740 1 --2- 172.21.15.60:0/430405991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad26685ed0 0x55ad2668c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd410e2b740 1 --2- 172.21.15.60:0/430405991 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55ad266ede70 0x55ad266f0290 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd410e2b740 1 --2- 172.21.15.60:0/430405991 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ad2667d2f0 0x55ad2668bfb0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd410e2b740 1 --2- 172.21.15.60:0/430405991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ad2667c9d0 0x55ad2668ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd3e6ffd640 0 client.16329 ms_handle_reset on v2:172.21.15.86:6838/3347800322 2024-03-04T12:27:21.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd410e2b740 1 -- 172.21.15.60:0/430405991 >> 172.21.15.60:0/430405991 conn(0x55ad26677100 msgr2=0x55ad26681d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd410e2b740 1 -- 172.21.15.60:0/430405991 shutdown_connections 2024-03-04T12:27:21.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd410e2b740 1 -- 172.21.15.60:0/430405991 wait complete. 2024-03-04T12:27:21.922 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd410e2b740 20 client.16329 trim_cache size 0 max 16384 2024-03-04T12:27:21.923 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:27:21.923 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd40e174640 1 -- 172.21.15.60:0/103547972 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd3f00a2010 msgr2=0x7fd3f00a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd40e174640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd3f00a2010 0x7fd3f00a44d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0096c0 tx=0x7fd3fc009610 comp rx=0 tx=0).stop 2024-03-04T12:27:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd40e174640 1 -- 172.21.15.60:0/103547972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4081979a0 msgr2=0x7fd408197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd40e174640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4081979a0 0x7fd408197e50 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fd4000096f0 tx=0x7fd400033480 comp rx=0 tx=0).stop 2024-03-04T12:27:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd40e174640 1 -- 172.21.15.60:0/103547972 shutdown_connections 2024-03-04T12:27:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd40e174640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd3f00a2010 0x7fd3f00a44d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd40e174640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd4081979a0 0x7fd408197e50 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd40e174640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd408105510 0x7fd40819dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd40e174640 1 --2- 172.21.15.60:0/103547972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd408100440 0x7fd40819d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.907+0000 7fd40e174640 1 -- 172.21.15.60:0/103547972 >> 172.21.15.60:0/103547972 conn(0x7fd4080773e0 msgr2=0x7fd4081040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.908+0000 7fd40e174640 1 -- 172.21.15.60:0/103547972 shutdown_connections 2024-03-04T12:27:21.924 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:21.908+0000 7fd40e174640 1 -- 172.21.15.60:0/103547972 wait complete. 2024-03-04T12:27:21.926 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:27:21.926 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T12:27:22.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.070+0000 7fc894e59640 1 -- 172.21.15.60:0/3732615950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc890100350 msgr2=0x7fc890105f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:22.080 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.070+0000 7fc894e59640 1 --2- 172.21.15.60:0/3732615950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc890100350 0x7fc890105f70 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fc878009900 tx=0x7fc87802bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.070+0000 7fc894e59640 1 -- 172.21.15.60:0/3732615950 shutdown_connections 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.070+0000 7fc894e59640 1 --2- 172.21.15.60:0/3732615950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc890106650 0x7fc89010e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.070+0000 7fc894e59640 1 --2- 172.21.15.60:0/3732615950 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc890100350 0x7fc890105f70 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.070+0000 7fc894e59640 1 --2- 172.21.15.60:0/3732615950 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8900ff9a0 0x7fc8900ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.070+0000 7fc894e59640 1 -- 172.21.15.60:0/3732615950 >> 172.21.15.60:0/3732615950 conn(0x7fc8900777a0 msgr2=0x7fc8900fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.070+0000 7fc894e59640 1 -- 172.21.15.60:0/3732615950 shutdown_connections 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.070+0000 7fc894e59640 1 -- 172.21.15.60:0/3732615950 wait complete. 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.071+0000 7fc894e59640 1 Processor -- start 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.071+0000 7fc894e59640 1 -- start start 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.071+0000 7fc894e59640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8900ff9a0 0x7fc890100910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.071+0000 7fc894e59640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc890106650 0x7fc890100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.071+0000 7fc894e59640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc890101390 0x7fc8901a5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.081 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.071+0000 7fc894e59640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc890111720 con 0x7fc8900ff9a0 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.071+0000 7fc894e59640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8901115a0 con 0x7fc890101390 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.071+0000 7fc894e59640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8901118a0 con 0x7fc890106650 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.071+0000 7fc88e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8900ff9a0 0x7fc890100910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.071+0000 7fc88e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8900ff9a0 0x7fc890100910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34450/0 (socket says 172.21.15.60:34450) 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.071+0000 7fc88e59c640 1 -- 172.21.15.60:0/304212421 learned_addr learned my addr 172.21.15.60:0/304212421 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.071+0000 7fc887fff640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc890106650 0x7fc890100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.072+0000 7fc88ed9d640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc890101390 0x7fc8901a5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.072+0000 7fc88e59c640 1 -- 172.21.15.60:0/304212421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc890106650 msgr2=0x7fc890100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.072+0000 7fc88e59c640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc890106650 0x7fc890100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.072+0000 7fc88e59c640 1 -- 172.21.15.60:0/304212421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc890101390 msgr2=0x7fc8901a5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.072+0000 7fc88e59c640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc890101390 0x7fc8901a5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.072+0000 7fc88e59c640 1 -- 172.21.15.60:0/304212421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8901a6530 con 0x7fc8900ff9a0 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.072+0000 7fc88e59c640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8900ff9a0 0x7fc890100910 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc88000d950 tx=0x7fc88000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:22.082 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.072+0000 7fc8877fe640 1 -- 172.21.15.60:0/304212421 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc880017070 con 0x7fc8900ff9a0 2024-03-04T12:27:22.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.072+0000 7fc8877fe640 1 -- 172.21.15.60:0/304212421 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc880009c30 con 0x7fc8900ff9a0 2024-03-04T12:27:22.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.072+0000 7fc8877fe640 1 -- 172.21.15.60:0/304212421 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88000f6b0 con 0x7fc8900ff9a0 2024-03-04T12:27:22.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.072+0000 7fc894e59640 1 -- 172.21.15.60:0/304212421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8901a6820 con 0x7fc8900ff9a0 2024-03-04T12:27:22.083 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.072+0000 7fc894e59640 1 -- 172.21.15.60:0/304212421 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8901a6d90 con 0x7fc8900ff9a0 2024-03-04T12:27:22.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.074+0000 7fc8877fe640 1 -- 172.21.15.60:0/304212421 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7fc8800109d0 con 0x7fc8900ff9a0 2024-03-04T12:27:22.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc897a37740 1 Processor -- start 2024-03-04T12:27:22.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc897a37740 1 -- start start 2024-03-04T12:27:22.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc897a37740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e189adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc897a37740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e189b2f0 0x55c4e18a3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc897a37740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 0x55c4e18a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc897a37740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4e1861dc0 con 0x55c4e18a3ed0 2024-03-04T12:27:22.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc897a37740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4e17fdec0 con 0x55c4e189a9d0 2024-03-04T12:27:22.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc897a37740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4e17a6d30 con 0x55c4e189b2f0 2024-03-04T12:27:22.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc8877fe640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc87c0a20c0 0x7fc87c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc887fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 0x55c4e18a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc887fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 0x55c4e18a62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34466/0 (socket says 172.21.15.60:34466) 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc887fff640 1 -- 172.21.15.60:0/1187699837 learned_addr learned my addr 172.21.15.60:0/1187699837 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc88ed9d640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc87c0a20c0 0x7fc87c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc8877fe640 1 -- 172.21.15.60:0/304212421 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fc8800ce480 con 0x7fc8900ff9a0 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc88e59c640 1 --2- 172.21.15.60:0/1187699837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e189b2f0 0x55c4e18a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.075+0000 7fc88ed9d640 1 --2- 172.21.15.60:0/1187699837 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e189adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.076+0000 7fc887fff640 1 -- 172.21.15.60:0/1187699837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e189b2f0 msgr2=0x55c4e18a3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.076+0000 7fc887fff640 1 --2- 172.21.15.60:0/1187699837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e189b2f0 0x55c4e18a3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.076+0000 7fc887fff640 1 -- 172.21.15.60:0/1187699837 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 msgr2=0x55c4e189adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.076+0000 7fc887fff640 1 --2- 172.21.15.60:0/1187699837 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e189adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.076+0000 7fc887fff640 1 -- 172.21.15.60:0/1187699837 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4e15123f0 con 0x55c4e18a3ed0 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.076+0000 7fc88ed9d640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc87c0a20c0 0x7fc87c0a4580 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc888005c40 tx=0x7fc888009c90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.076+0000 7fc887fff640 1 --2- 172.21.15.60:0/1187699837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 0x55c4e18a62c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc8900738d0 tx=0x7fc878005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f29ebe575fbfd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.076+0000 7fc884ff9640 1 -- 172.21.15.60:0/1187699837 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87802ca80 con 0x55c4e18a3ed0 2024-03-04T12:27:22.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.076+0000 7fc884ff9640 1 -- 172.21.15.60:0/1187699837 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc87802cc20 con 0x55c4e18a3ed0 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.076+0000 7fc884ff9640 1 -- 172.21.15.60:0/1187699837 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8780046c0 con 0x55c4e18a3ed0 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 -- 172.21.15.60:0/1187699837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 msgr2=0x55c4e18a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 --2- 172.21.15.60:0/1187699837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 0x55c4e18a62c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc8900738d0 tx=0x7fc878005e00 comp rx=0 tx=0).stop 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 -- 172.21.15.60:0/1187699837 shutdown_connections 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 --2- 172.21.15.60:0/1187699837 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 0x55c4e18a62c0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 --2- 172.21.15.60:0/1187699837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e189b2f0 0x55c4e18a3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 --2- 172.21.15.60:0/1187699837 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e189adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 -- 172.21.15.60:0/1187699837 >> 172.21.15.60:0/1187699837 conn(0x55c4e1895100 msgr2=0x55c4e18954e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 -- 172.21.15.60:0/1187699837 shutdown_connections 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 -- 172.21.15.60:0/1187699837 wait complete. 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 -- start start 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 Processor -- start 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e18a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e189b2f0 0x55c4e18a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.077+0000 7fc897a37740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 0x55c4e18a1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc897a37740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4e179c4a0 con 0x55c4e18a3ed0 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc897a37740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4e17f9f60 con 0x55c4e189a9d0 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc897a37740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4e18a6c60 con 0x55c4e189b2f0 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc897a37740 10 client.0 _ll_register_callbacks cb 0x55c4e17f81c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc88ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 0x55c4e18a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc88ed9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 0x55c4e18a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34470/0 (socket says 172.21.15.60:34470) 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc88ed9d640 1 -- 172.21.15.60:0/3641520033 learned_addr learned my addr 172.21.15.60:0/3641520033 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc887fff640 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e189b2f0 0x55c4e18a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc88ed9d640 1 -- 172.21.15.60:0/3641520033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e189b2f0 msgr2=0x55c4e18a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc88e59c640 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e18a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc88ed9d640 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e189b2f0 0x55c4e18a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc88ed9d640 1 -- 172.21.15.60:0/3641520033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 msgr2=0x55c4e18a0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:22.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc88ed9d640 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e18a0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc88ed9d640 1 -- 172.21.15.60:0/3641520033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4e15dc7d0 con 0x55c4e18a3ed0 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc88ed9d640 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 0x55c4e18a1910 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc888010c40 tx=0x7fc88801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639284f1c263b0b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc875ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc875ffb640 1 -- 172.21.15.60:0/3641520033 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc888027070 con 0x55c4e18a3ed0 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc875ffb640 1 -- 172.21.15.60:0/3641520033 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fc88801ee00 con 0x55c4e18a3ed0 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc897a37740 1 -- 172.21.15.60:0/3641520033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc87802cc20 con 0x55c4e18a3ed0 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc875ffb640 1 -- 172.21.15.60:0/3641520033 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc888033050 con 0x55c4e18a3ed0 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc897a37740 10 client.16341 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.078+0000 7fc897a37740 1 -- 172.21.15.60:0/3641520033 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55c4e18b2d70 con 0x55c4e18a3ed0 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.079+0000 7fc875ffb640 1 -- 172.21.15.60:0/3641520033 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc87802cc20 con 0x55c4e18a3ed0 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.079+0000 7fc875ffb640 1 -- 172.21.15.60:0/3641520033 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc88801f070 con 0x55c4e18a3ed0 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.079+0000 7fc897a37740 10 client.16341 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.079+0000 7fc897a37740 10 client.16341 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.079+0000 7fc897a37740 20 client.16341 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.079+0000 7fc897a37740 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55c4e18b4240 0x55c4e18b6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.079+0000 7fc897a37740 4 client.16341 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:22.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.079+0000 7fc897a37740 1 -- 172.21.15.60:0/3641520033 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4e180b400 con 0x55c4e18b4240 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.079+0000 7fc88e59c640 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55c4e18b4240 0x55c4e18b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.079+0000 7fc88e59c640 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55c4e18b4240 0x55c4e18b6690 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60e61cecc3530d80 server_cookie=fc43de21264d2f15 in_seq=0 out_seq=0 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.079+0000 7fc875ffb640 10 client.16341 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc875ffb640 1 -- 172.21.15.60:0/3641520033 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c4e180b400 con 0x55c4e18b4240 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc875ffb640 10 client.16341 handle_command_reply: tid=0 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc897a37740 1 client.16341 shutdown 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc897a37740 1 -- 172.21.15.60:0/3641520033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 msgr2=0x55c4e18a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc897a37740 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 0x55c4e18a1910 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fc888010c40 tx=0x7fc88801c000 comp rx=0 tx=0).stop 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc897a37740 1 -- 172.21.15.60:0/3641520033 shutdown_connections 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc897a37740 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55c4e18b4240 0x55c4e18b6690 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc897a37740 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e18a3ed0 0x55c4e18a1910 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc897a37740 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e189b2f0 0x55c4e18a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc875ffb640 0 client.16341 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc897a37740 1 --2- 172.21.15.60:0/3641520033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e18a0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.094 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc897a37740 1 -- 172.21.15.60:0/3641520033 >> 172.21.15.60:0/3641520033 conn(0x55c4e1895100 msgr2=0x55c4e1896e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:22.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc897a37740 1 -- 172.21.15.60:0/3641520033 shutdown_connections 2024-03-04T12:27:22.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc897a37740 1 -- 172.21.15.60:0/3641520033 wait complete. 2024-03-04T12:27:22.095 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.080+0000 7fc897a37740 20 client.16341 trim_cache size 0 max 16384 2024-03-04T12:27:22.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc897a37740 1 Processor -- start 2024-03-04T12:27:22.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc897a37740 1 -- start start 2024-03-04T12:27:22.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc897a37740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189a9d0 0x55c4e18a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc897a37740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189b2f0 0x55c4e189fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc897a37740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e18a3ed0 0x55c4e189e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc897a37740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8901b0680 con 0x55c4e189a9d0 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc897a37740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8901b2030 con 0x55c4e189b2f0 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc897a37740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8901b0ce0 con 0x55c4e18a3ed0 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc88e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189a9d0 0x55c4e18a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc88e59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189a9d0 0x55c4e18a35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34472/0 (socket says 172.21.15.60:34472) 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc88e59c640 1 -- 172.21.15.60:0/3227613140 learned_addr learned my addr 172.21.15.60:0/3227613140 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc88ed9d640 1 --2- 172.21.15.60:0/3227613140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e18a3ed0 0x55c4e189e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc887fff640 1 --2- 172.21.15.60:0/3227613140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189b2f0 0x55c4e189fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc88e59c640 1 -- 172.21.15.60:0/3227613140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e18a3ed0 msgr2=0x55c4e189e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc88e59c640 1 --2- 172.21.15.60:0/3227613140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e18a3ed0 0x55c4e189e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc88e59c640 1 -- 172.21.15.60:0/3227613140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189b2f0 msgr2=0x55c4e189fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc88e59c640 1 --2- 172.21.15.60:0/3227613140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189b2f0 0x55c4e189fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.092+0000 7fc88e59c640 1 -- 172.21.15.60:0/3227613140 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4e189b120 con 0x55c4e189a9d0 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc887fff640 1 --2- 172.21.15.60:0/3227613140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189b2f0 0x55c4e189fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc88e59c640 1 --2- 172.21.15.60:0/3227613140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189a9d0 0x55c4e18a35c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc88009a7f0 tx=0x7fc880096420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3ba76e6def5680 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc877fff640 1 -- 172.21.15.60:0/3227613140 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc880020040 con 0x55c4e189a9d0 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc877fff640 1 -- 172.21.15.60:0/3227613140 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8800a3000 con 0x55c4e189a9d0 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc877fff640 1 -- 172.21.15.60:0/3227613140 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8800850b0 con 0x55c4e189a9d0 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc897a37740 1 -- 172.21.15.60:0/3227613140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189a9d0 msgr2=0x55c4e18a35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc897a37740 1 --2- 172.21.15.60:0/3227613140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189a9d0 0x55c4e18a35c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fc88009a7f0 tx=0x7fc880096420 comp rx=0 tx=0).stop 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc897a37740 1 -- 172.21.15.60:0/3227613140 shutdown_connections 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc897a37740 1 --2- 172.21.15.60:0/3227613140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e18a3ed0 0x55c4e189e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc897a37740 1 --2- 172.21.15.60:0/3227613140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189b2f0 0x55c4e189fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc897a37740 1 --2- 172.21.15.60:0/3227613140 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189a9d0 0x55c4e18a35c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc897a37740 1 -- 172.21.15.60:0/3227613140 >> 172.21.15.60:0/3227613140 conn(0x55c4e1895100 msgr2=0x55c4e189f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc897a37740 1 -- 172.21.15.60:0/3227613140 shutdown_connections 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.093+0000 7fc897a37740 1 -- 172.21.15.60:0/3227613140 wait complete. 2024-03-04T12:27:22.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc897a37740 1 -- start start 2024-03-04T12:27:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc897a37740 1 Processor -- start 2024-03-04T12:27:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc897a37740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e18a9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc897a37740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189b2f0 0x55c4e18aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc897a37740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e18a3ed0 0x55c4e190a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc897a37740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8901b19d0 con 0x55c4e189b2f0 2024-03-04T12:27:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc897a37740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8901b1340 con 0x55c4e189a9d0 2024-03-04T12:27:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc897a37740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8901ae920 con 0x55c4e18a3ed0 2024-03-04T12:27:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc897a37740 10 client.0 _ll_register_callbacks cb 0x7fc878009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc887fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189b2f0 0x55c4e18aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc887fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189b2f0 0x55c4e18aa1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34488/0 (socket says 172.21.15.60:34488) 2024-03-04T12:27:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc88e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e18a9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc887fff640 1 -- 172.21.15.60:0/31288126 learned_addr learned my addr 172.21.15.60:0/31288126 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc88ed9d640 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e18a3ed0 0x55c4e190a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc887fff640 1 -- 172.21.15.60:0/31288126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e18a3ed0 msgr2=0x55c4e190a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc887fff640 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e18a3ed0 0x55c4e190a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc887fff640 1 -- 172.21.15.60:0/31288126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 msgr2=0x55c4e18a9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc887fff640 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e18a9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc887fff640 1 -- 172.21.15.60:0/31288126 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4e190acc0 con 0x55c4e189b2f0 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc88e59c640 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e18a9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.094+0000 7fc887fff640 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189b2f0 0x55c4e18aa1e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc8900738d0 tx=0x7fc878005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313e1f1e05e20e7b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc875ffb640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc875ffb640 1 -- 172.21.15.60:0/31288126 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc878005020 con 0x55c4e189b2f0 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc875ffb640 1 -- 172.21.15.60:0/31288126 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fc8780051c0 con 0x55c4e189b2f0 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc897a37740 1 -- 172.21.15.60:0/31288126 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc8800a3000 con 0x55c4e189b2f0 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc875ffb640 1 -- 172.21.15.60:0/31288126 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8780053c0 con 0x55c4e189b2f0 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc897a37740 10 client.16347 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:22.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc897a37740 1 -- 172.21.15.60:0/31288126 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55c4e190bf00 con 0x55c4e189b2f0 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc875ffb640 1 -- 172.21.15.60:0/31288126 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8800a3000 con 0x55c4e189b2f0 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc875ffb640 1 -- 172.21.15.60:0/31288126 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc8780036f0 con 0x55c4e189b2f0 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc897a37740 10 client.16347 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc897a37740 10 client.16347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc897a37740 20 client.16347 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc897a37740 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55c4e190c2e0 0x55c4e190e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc897a37740 4 client.16347 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc897a37740 1 -- 172.21.15.60:0/31288126 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c4e1805750 con 0x55c4e190c2e0 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.095+0000 7fc88e59c640 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55c4e190c2e0 0x55c4e190e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc88e59c640 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55c4e190c2e0 0x55c4e190e6d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f4852ddad500053 server_cookie=9b4767576c88109d in_seq=0 out_seq=0 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc875ffb640 10 client.16347 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc875ffb640 1 -- 172.21.15.60:0/31288126 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c4e1805750 con 0x55c4e190c2e0 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc875ffb640 10 client.16347 handle_command_reply: tid=0 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc897a37740 1 client.16347 shutdown 2024-03-04T12:27:22.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc897a37740 1 -- 172.21.15.60:0/31288126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189b2f0 msgr2=0x55c4e18aa1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc897a37740 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189b2f0 0x55c4e18aa1e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc8900738d0 tx=0x7fc878005710 comp rx=0 tx=0).stop 2024-03-04T12:27:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc897a37740 1 -- 172.21.15.60:0/31288126 shutdown_connections 2024-03-04T12:27:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc897a37740 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55c4e190c2e0 0x55c4e190e6d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc897a37740 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c4e18a3ed0 0x55c4e190a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc875ffb640 0 client.16347 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc897a37740 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55c4e189b2f0 0x55c4e18aa1e0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc897a37740 1 --2- 172.21.15.60:0/31288126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c4e189a9d0 0x55c4e18a9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc897a37740 1 -- 172.21.15.60:0/31288126 >> 172.21.15.60:0/31288126 conn(0x55c4e1895100 msgr2=0x55c4e189fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc897a37740 1 -- 172.21.15.60:0/31288126 shutdown_connections 2024-03-04T12:27:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc897a37740 1 -- 172.21.15.60:0/31288126 wait complete. 2024-03-04T12:27:22.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.096+0000 7fc897a37740 20 client.16347 trim_cache size 0 max 16384 2024-03-04T12:27:22.112 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:27:22.112 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.097+0000 7fc894e59640 1 -- 172.21.15.60:0/304212421 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc87c0a20c0 msgr2=0x7fc87c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.097+0000 7fc894e59640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc87c0a20c0 0x7fc87c0a4580 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc888005c40 tx=0x7fc888009c90 comp rx=0 tx=0).stop 2024-03-04T12:27:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.097+0000 7fc894e59640 1 -- 172.21.15.60:0/304212421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8900ff9a0 msgr2=0x7fc890100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.097+0000 7fc894e59640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8900ff9a0 0x7fc890100910 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc88000d950 tx=0x7fc88000de10 comp rx=0 tx=0).stop 2024-03-04T12:27:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.097+0000 7fc894e59640 1 -- 172.21.15.60:0/304212421 shutdown_connections 2024-03-04T12:27:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.097+0000 7fc894e59640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc890101390 0x7fc8901a5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.097+0000 7fc894e59640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc890106650 0x7fc890100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.097+0000 7fc894e59640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc87c0a20c0 0x7fc87c0a4580 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.097+0000 7fc894e59640 1 --2- 172.21.15.60:0/304212421 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc8900ff9a0 0x7fc890100910 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.097+0000 7fc894e59640 1 -- 172.21.15.60:0/304212421 >> 172.21.15.60:0/304212421 conn(0x7fc8900777a0 msgr2=0x7fc8901049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.097+0000 7fc894e59640 1 -- 172.21.15.60:0/304212421 shutdown_connections 2024-03-04T12:27:22.113 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:22.097+0000 7fc894e59640 1 -- 172.21.15.60:0/304212421 wait complete. 2024-03-04T12:27:22.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:21 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3384825315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:27:22.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:21 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3384825315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:27:22.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:21 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3384825315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:27:23.116 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-04T12:27:23.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:27:23.116 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-04T12:27:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.228+0000 7f3076c52640 1 -- 172.21.15.60:0/77087533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30701000d0 msgr2=0x7f30701004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.228+0000 7f3076c52640 1 --2- 172.21.15.60:0/77087533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30701000d0 0x7f30701004b0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f3060009900 tx=0x7f306002bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.228+0000 7f3076c52640 1 -- 172.21.15.60:0/77087533 shutdown_connections 2024-03-04T12:27:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.228+0000 7f3076c52640 1 --2- 172.21.15.60:0/77087533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f307010dc80 0x7f3070110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.228+0000 7f3076c52640 1 --2- 172.21.15.60:0/77087533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30701009f0 0x7f307010d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.228+0000 7f3076c52640 1 --2- 172.21.15.60:0/77087533 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f30701000d0 0x7f30701004b0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.228+0000 7f3076c52640 1 -- 172.21.15.60:0/77087533 >> 172.21.15.60:0/77087533 conn(0x7f3070077350 msgr2=0x7f3070077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.228+0000 7f3076c52640 1 -- 172.21.15.60:0/77087533 shutdown_connections 2024-03-04T12:27:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.228+0000 7f3076c52640 1 -- 172.21.15.60:0/77087533 wait complete. 2024-03-04T12:27:23.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.228+0000 7f3076c52640 1 Processor -- start 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.228+0000 7f3076c52640 1 -- start start 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f3076c52640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30701000d0 0x7f3070199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f3076c52640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30701009f0 0x7f307019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f3076c52640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f307010dc80 0x7f307019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f3076c52640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30701138f0 con 0x7f307010dc80 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f3076c52640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3070113770 con 0x7f30701000d0 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f3076c52640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3070113a70 con 0x7f30701009f0 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f30751ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f307010dc80 0x7f307019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f30751ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f307010dc80 0x7f307019e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34512/0 (socket says 172.21.15.60:34512) 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f30751ef640 1 -- 172.21.15.60:0/830681186 learned_addr learned my addr 172.21.15.60:0/830681186 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f30749ee640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30701000d0 0x7f3070199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f306ffff640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30701009f0 0x7f307019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f30751ef640 1 -- 172.21.15.60:0/830681186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30701009f0 msgr2=0x7f307019a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f30751ef640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30701009f0 0x7f307019a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f30751ef640 1 -- 172.21.15.60:0/830681186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30701000d0 msgr2=0x7f3070199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f30751ef640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30701000d0 0x7f3070199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f30751ef640 1 -- 172.21.15.60:0/830681186 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307019ede0 con 0x7f307010dc80 2024-03-04T12:27:23.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f306ffff640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30701009f0 0x7f307019a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:27:23.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f30751ef640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f307010dc80 0x7f307019e6b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3068009fd0 tx=0x7f306800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:23.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f306dffb640 1 -- 172.21.15.60:0/830681186 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3068004d20 con 0x7f307010dc80 2024-03-04T12:27:23.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f306dffb640 1 -- 172.21.15.60:0/830681186 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3068005020 con 0x7f307010dc80 2024-03-04T12:27:23.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f3076c52640 1 -- 172.21.15.60:0/830681186 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307019f0d0 con 0x7f307010dc80 2024-03-04T12:27:23.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.229+0000 7f306dffb640 1 -- 172.21.15.60:0/830681186 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30680116d0 con 0x7f307010dc80 2024-03-04T12:27:23.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.230+0000 7f3076c52640 1 -- 172.21.15.60:0/830681186 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30701a6960 con 0x7f307010dc80 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.232+0000 7f306dffb640 1 -- 172.21.15.60:0/830681186 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7f30680051d0 con 0x7f307010dc80 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.232+0000 7f3079834740 1 Processor -- start 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.232+0000 7f3079834740 1 -- start start 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.232+0000 7f306dffb640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f30580a2010 0x7f30580a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.232+0000 7f3079834740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee3f20 0x55dcc7ee4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.232+0000 7f3079834740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 0x55dcc7eecee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.232+0000 7f3079834740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7eed420 0x55dcc7eef810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f3079834740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcc7e47b40 con 0x55dcc7ee4840 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f3079834740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcc7df0d80 con 0x55dcc7eed420 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f3079834740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcc7e43be0 con 0x55dcc7ee3f20 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f306ffff640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f30580a2010 0x7f30580a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f306dffb640 1 -- 172.21.15.60:0/830681186 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f30680cede0 con 0x7f307010dc80 2024-03-04T12:27:23.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f306ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 0x55dcc7eecee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f306ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 0x55dcc7eecee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34522/0 (socket says 172.21.15.60:34522) 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f306ffff640 1 -- 172.21.15.60:0/4172362106 learned_addr learned my addr 172.21.15.60:0/4172362106 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f30749ee640 1 --2- 172.21.15.60:0/4172362106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee3f20 0x55dcc7ee4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f30751ef640 1 --2- 172.21.15.60:0/4172362106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7eed420 0x55dcc7eef810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f306ffff640 1 -- 172.21.15.60:0/4172362106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee3f20 msgr2=0x55dcc7ee4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f306ffff640 1 --2- 172.21.15.60:0/4172362106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee3f20 0x55dcc7ee4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f306ffff640 1 -- 172.21.15.60:0/4172362106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7eed420 msgr2=0x55dcc7eef810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f306ffff640 1 --2- 172.21.15.60:0/4172362106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7eed420 0x55dcc7eef810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f306ffff640 1 -- 172.21.15.60:0/4172362106 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dcc7c247e0 con 0x55dcc7ee4840 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f306ffff640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f30580a2010 0x7f30580a44d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f306400bfd0 tx=0x7f306400bb20 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f306ffff640 1 --2- 172.21.15.60:0/4172362106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 0x55dcc7eecee0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f3064005d00 tx=0x7f306401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99608aa43cdca9ec server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f3056ffd640 1 -- 172.21.15.60:0/4172362106 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064023070 con 0x55dcc7ee4840 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.233+0000 7f3056ffd640 1 -- 172.21.15.60:0/4172362106 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3064010e40 con 0x55dcc7ee4840 2024-03-04T12:27:23.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.234+0000 7f3056ffd640 1 -- 172.21.15.60:0/4172362106 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064003830 con 0x55dcc7ee4840 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.234+0000 7f3079834740 1 -- 172.21.15.60:0/4172362106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 msgr2=0x55dcc7eecee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.234+0000 7f3079834740 1 --2- 172.21.15.60:0/4172362106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 0x55dcc7eecee0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f3064005d00 tx=0x7f306401a000 comp rx=0 tx=0).stop 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.234+0000 7f3079834740 1 -- 172.21.15.60:0/4172362106 shutdown_connections 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.234+0000 7f3079834740 1 --2- 172.21.15.60:0/4172362106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7eed420 0x55dcc7eef810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.234+0000 7f3079834740 1 --2- 172.21.15.60:0/4172362106 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 0x55dcc7eecee0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.234+0000 7f3079834740 1 --2- 172.21.15.60:0/4172362106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee3f20 0x55dcc7ee4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.234+0000 7f3079834740 1 -- 172.21.15.60:0/4172362106 >> 172.21.15.60:0/4172362106 conn(0x55dcc7ede650 msgr2=0x55dcc7edea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.234+0000 7f3079834740 1 -- 172.21.15.60:0/4172362106 shutdown_connections 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.234+0000 7f3079834740 1 -- 172.21.15.60:0/4172362106 wait complete. 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f3079834740 1 -- start start 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f3079834740 1 Processor -- start 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f3079834740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7eecdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f3079834740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee4840 0x55dcc7eea160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f3079834740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 0x55dcc7eea700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f3079834740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcc7ef01b0 con 0x55dcc7eed420 2024-03-04T12:27:23.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f3079834740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcc7e041c0 con 0x55dcc7ee3f20 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f3079834740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dcc7ef0330 con 0x55dcc7ee4840 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f3079834740 10 client.0 _ll_register_callbacks cb 0x55dcc7de9f80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f30751ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 0x55dcc7eea700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f30751ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 0x55dcc7eea700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34534/0 (socket says 172.21.15.60:34534) 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f30751ef640 1 -- 172.21.15.60:0/229391709 learned_addr learned my addr 172.21.15.60:0/229391709 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f306ffff640 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee4840 0x55dcc7eea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.235+0000 7f30749ee640 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7eecdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f30751ef640 1 -- 172.21.15.60:0/229391709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee4840 msgr2=0x55dcc7eea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f30751ef640 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee4840 0x55dcc7eea160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f30751ef640 1 -- 172.21.15.60:0/229391709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 msgr2=0x55dcc7eecdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f30751ef640 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7eecdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f30751ef640 1 -- 172.21.15.60:0/229391709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dcc7c84f70 con 0x55dcc7eed420 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f30751ef640 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 0x55dcc7eea700 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3068097b50 tx=0x7f306809b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9f6cf924db99d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f304ffff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f304ffff640 1 -- 172.21.15.60:0/229391709 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306801f070 con 0x55dcc7eed420 2024-03-04T12:27:23.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f304ffff640 1 -- 172.21.15.60:0/229391709 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f30680977f0 con 0x55dcc7eed420 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f3079834740 1 -- 172.21.15.60:0/229391709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3064010e40 con 0x55dcc7eed420 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f304ffff640 1 -- 172.21.15.60:0/229391709 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f30680a3920 con 0x55dcc7eed420 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f3079834740 10 client.16359 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f3079834740 1 -- 172.21.15.60:0/229391709 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55dcc7b1c170 con 0x55dcc7eed420 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f304ffff640 1 -- 172.21.15.60:0/229391709 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3064010e40 con 0x55dcc7eed420 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f304ffff640 1 -- 172.21.15.60:0/229391709 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f306800f040 con 0x55dcc7eed420 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f3079834740 10 client.16359 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f3079834740 10 client.16359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.236+0000 7f3079834740 20 client.16359 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.237+0000 7f3079834740 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55dcc7efd3c0 0x55dcc7eff840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.237+0000 7f3079834740 4 client.16359 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.237+0000 7f3079834740 1 -- 172.21.15.60:0/229391709 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dcc7e1d960 con 0x55dcc7efd3c0 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.237+0000 7f30749ee640 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55dcc7efd3c0 0x55dcc7eff840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.237+0000 7f30749ee640 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55dcc7efd3c0 0x55dcc7eff840 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6702312965cc2467 server_cookie=10591f6c54d7ed49 in_seq=0 out_seq=0 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.237+0000 7f304ffff640 10 client.16359 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:23.251 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f304ffff640 1 -- 172.21.15.60:0/229391709 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dcc7e1d960 con 0x55dcc7efd3c0 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f304ffff640 10 client.16359 handle_command_reply: tid=0 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f3079834740 1 client.16359 shutdown 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f3079834740 1 -- 172.21.15.60:0/229391709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 msgr2=0x55dcc7eea700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f3079834740 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 0x55dcc7eea700 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f3068097b50 tx=0x7f306809b940 comp rx=0 tx=0).stop 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f3079834740 1 -- 172.21.15.60:0/229391709 shutdown_connections 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f3079834740 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55dcc7efd3c0 0x55dcc7eff840 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f3079834740 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 0x55dcc7eea700 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f3079834740 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee4840 0x55dcc7eea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f3079834740 1 --2- 172.21.15.60:0/229391709 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7eecdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f304ffff640 0 client.16359 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f3079834740 1 -- 172.21.15.60:0/229391709 >> 172.21.15.60:0/229391709 conn(0x55dcc7ede650 msgr2=0x55dcc7ee03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f3079834740 1 -- 172.21.15.60:0/229391709 shutdown_connections 2024-03-04T12:27:23.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.238+0000 7f3079834740 1 -- 172.21.15.60:0/229391709 wait complete. 2024-03-04T12:27:23.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.239+0000 7f3079834740 20 client.16359 trim_cache size 0 max 16384 2024-03-04T12:27:23.262 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f3079834740 1 Processor -- start 2024-03-04T12:27:23.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f3079834740 1 -- start start 2024-03-04T12:27:23.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f3079834740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7eec890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f3079834740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 0x55dcc7eecdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f3079834740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7eed420 0x55dcc7ef4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f3079834740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30701b06d0 con 0x55dcc7ee4840 2024-03-04T12:27:23.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f3079834740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30701b2080 con 0x55dcc7ee3f20 2024-03-04T12:27:23.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f3079834740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30701b0d30 con 0x55dcc7eed420 2024-03-04T12:27:23.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f306ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 0x55dcc7eecdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f306ffff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 0x55dcc7eecdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34540/0 (socket says 172.21.15.60:34540) 2024-03-04T12:27:23.263 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f306ffff640 1 -- 172.21.15.60:0/2779447499 learned_addr learned my addr 172.21.15.60:0/2779447499 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f30749ee640 1 --2- 172.21.15.60:0/2779447499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7eec890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.254+0000 7f30751ef640 1 --2- 172.21.15.60:0/2779447499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7eed420 0x55dcc7ef4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.255+0000 7f306ffff640 1 -- 172.21.15.60:0/2779447499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7eed420 msgr2=0x55dcc7ef4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.255+0000 7f306ffff640 1 --2- 172.21.15.60:0/2779447499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7eed420 0x55dcc7ef4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.255+0000 7f306ffff640 1 -- 172.21.15.60:0/2779447499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 msgr2=0x55dcc7eec890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.255+0000 7f306ffff640 1 --2- 172.21.15.60:0/2779447499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7eec890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.255+0000 7f306ffff640 1 -- 172.21.15.60:0/2779447499 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dcc7ee4670 con 0x55dcc7ee4840 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.255+0000 7f30749ee640 1 --2- 172.21.15.60:0/2779447499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7eec890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.255+0000 7f30751ef640 1 --2- 172.21.15.60:0/2779447499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7eed420 0x55dcc7ef4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.255+0000 7f306ffff640 1 --2- 172.21.15.60:0/2779447499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 0x55dcc7eecdd0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f306400ee80 tx=0x7f306400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e81c3fe98fad27 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.255+0000 7f30567fc640 1 -- 172.21.15.60:0/2779447499 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064023040 con 0x55dcc7ee4840 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.255+0000 7f30567fc640 1 -- 172.21.15.60:0/2779447499 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f306400ebc0 con 0x55dcc7ee4840 2024-03-04T12:27:23.264 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.255+0000 7f30567fc640 1 -- 172.21.15.60:0/2779447499 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064005290 con 0x55dcc7ee4840 2024-03-04T12:27:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.256+0000 7f3079834740 1 -- 172.21.15.60:0/2779447499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 msgr2=0x55dcc7eecdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.256+0000 7f3079834740 1 --2- 172.21.15.60:0/2779447499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 0x55dcc7eecdd0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f306400ee80 tx=0x7f306400ef20 comp rx=0 tx=0).stop 2024-03-04T12:27:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.256+0000 7f3079834740 1 -- 172.21.15.60:0/2779447499 shutdown_connections 2024-03-04T12:27:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.256+0000 7f3079834740 1 --2- 172.21.15.60:0/2779447499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7eed420 0x55dcc7ef4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.256+0000 7f3079834740 1 --2- 172.21.15.60:0/2779447499 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7ee4840 0x55dcc7eecdd0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.256+0000 7f3079834740 1 --2- 172.21.15.60:0/2779447499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7eec890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.256+0000 7f3079834740 1 -- 172.21.15.60:0/2779447499 >> 172.21.15.60:0/2779447499 conn(0x55dcc7ede650 msgr2=0x55dcc7eea120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.256+0000 7f3079834740 1 -- 172.21.15.60:0/2779447499 shutdown_connections 2024-03-04T12:27:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.256+0000 7f3079834740 1 -- 172.21.15.60:0/2779447499 wait complete. 2024-03-04T12:27:23.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.256+0000 7f3079834740 1 -- start start 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.256+0000 7f3079834740 1 Processor -- start 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f3079834740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7ef1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f3079834740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee4840 0x55dcc7ef18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f3079834740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 0x55dcc7f53920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f3079834740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30701b1a20 con 0x55dcc7eed420 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f3079834740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30701b1390 con 0x55dcc7ee3f20 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f3079834740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30701ae970 con 0x55dcc7ee4840 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f3079834740 10 client.0 _ll_register_callbacks cb 0x7f3064013420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f30751ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 0x55dcc7f53920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f30751ef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 0x55dcc7f53920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34552/0 (socket says 172.21.15.60:34552) 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f30751ef640 1 -- 172.21.15.60:0/3162045894 learned_addr learned my addr 172.21.15.60:0/3162045894 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f30749ee640 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7ef1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f306ffff640 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee4840 0x55dcc7ef18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f30751ef640 1 -- 172.21.15.60:0/3162045894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee4840 msgr2=0x55dcc7ef18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.270 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f30751ef640 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee4840 0x55dcc7ef18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f30751ef640 1 -- 172.21.15.60:0/3162045894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 msgr2=0x55dcc7ef1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f30751ef640 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7ef1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f30751ef640 1 -- 172.21.15.60:0/3162045894 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dcc7ef2170 con 0x55dcc7eed420 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f30751ef640 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 0x55dcc7f53920 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f3068051710 tx=0x7f30680092b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83cde85df76f97ea server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f304ffff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f304ffff640 1 -- 172.21.15.60:0/3162045894 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306801f040 con 0x55dcc7eed420 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.257+0000 7f304ffff640 1 -- 172.21.15.60:0/3162045894 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f306809a5d0 con 0x55dcc7eed420 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f3079834740 1 -- 172.21.15.60:0/3162045894 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f306400ebc0 con 0x55dcc7eed420 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f304ffff640 1 -- 172.21.15.60:0/3162045894 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f306809aae0 con 0x55dcc7eed420 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f3079834740 10 client.16365 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f3079834740 1 -- 172.21.15.60:0/3162045894 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55dcc7f54fa0 con 0x55dcc7eed420 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f304ffff640 1 -- 172.21.15.60:0/3162045894 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f306400ebc0 con 0x55dcc7eed420 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f304ffff640 1 -- 172.21.15.60:0/3162045894 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f30680cc320 con 0x55dcc7eed420 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f3079834740 10 client.16365 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:23.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f3079834740 10 client.16365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f3079834740 20 client.16365 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f3079834740 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55dcc7f55610 0x55dcc7f57a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f3079834740 4 client.16365 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f3079834740 1 -- 172.21.15.60:0/3162045894 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55dcc7e09080 con 0x55dcc7f55610 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.258+0000 7f30749ee640 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55dcc7f55610 0x55dcc7f57a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.259+0000 7f304ffff640 10 client.16365 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.259+0000 7f30749ee640 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55dcc7f55610 0x55dcc7f57a30 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57571fbd546e8577 server_cookie=dfee2c6e1b98d9a1 in_seq=0 out_seq=0 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.259+0000 7f304ffff640 1 -- 172.21.15.60:0/3162045894 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55dcc7e09080 con 0x55dcc7f55610 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.259+0000 7f304ffff640 10 client.16365 handle_command_reply: tid=0 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.259+0000 7f3079834740 1 client.16365 shutdown 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3079834740 1 -- 172.21.15.60:0/3162045894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 msgr2=0x55dcc7f53920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3079834740 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 0x55dcc7f53920 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f3068051710 tx=0x7f30680092b0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3079834740 1 -- 172.21.15.60:0/3162045894 shutdown_connections 2024-03-04T12:27:23.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3079834740 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55dcc7f55610 0x55dcc7f57a30 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3079834740 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55dcc7eed420 0x55dcc7f53920 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f304ffff640 0 client.16365 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:23.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3079834740 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55dcc7ee4840 0x55dcc7ef18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3079834740 1 --2- 172.21.15.60:0/3162045894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dcc7ee3f20 0x55dcc7ef1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3079834740 1 -- 172.21.15.60:0/3162045894 >> 172.21.15.60:0/3162045894 conn(0x55dcc7ede650 msgr2=0x55dcc7ee8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:23.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3079834740 1 -- 172.21.15.60:0/3162045894 shutdown_connections 2024-03-04T12:27:23.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3079834740 1 -- 172.21.15.60:0/3162045894 wait complete. 2024-03-04T12:27:23.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3079834740 20 client.16365 trim_cache size 0 max 16384 2024-03-04T12:27:23.273 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:27:23.273 INFO:teuthology.orchestra.run.smithi060.stdout: "return_code": 0, 2024-03-04T12:27:23.273 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_tag": "84da1cba-13e6-4cf2-8d43-dc95e1b3b36f", 2024-03-04T12:27:23.274 INFO:teuthology.orchestra.run.smithi060.stdout: "mode": "asynchronous" 2024-03-04T12:27:23.274 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:27:23.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3076c52640 1 -- 172.21.15.60:0/830681186 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f30580a2010 msgr2=0x7f30580a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3076c52640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f30580a2010 0x7f30580a44d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f306400bfd0 tx=0x7f306400bb20 comp rx=0 tx=0).stop 2024-03-04T12:27:23.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3076c52640 1 -- 172.21.15.60:0/830681186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f307010dc80 msgr2=0x7f307019e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.274 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.260+0000 7f3076c52640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f307010dc80 0x7f307019e6b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3068009fd0 tx=0x7f306800ee60 comp rx=0 tx=0).stop 2024-03-04T12:27:23.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.261+0000 7f3076c52640 1 -- 172.21.15.60:0/830681186 shutdown_connections 2024-03-04T12:27:23.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.261+0000 7f3076c52640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f30580a2010 0x7f30580a44d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.261+0000 7f3076c52640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f307010dc80 0x7f307019e6b0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.261+0000 7f3076c52640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30701009f0 0x7f307019a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.261+0000 7f3076c52640 1 --2- 172.21.15.60:0/830681186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30701000d0 0x7f3070199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.261+0000 7f3076c52640 1 -- 172.21.15.60:0/830681186 >> 172.21.15.60:0/830681186 conn(0x7f3070077350 msgr2=0x7f307010f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:23.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.261+0000 7f3076c52640 1 -- 172.21.15.60:0/830681186 shutdown_connections 2024-03-04T12:27:23.275 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.261+0000 7f3076c52640 1 -- 172.21.15.60:0/830681186 wait complete. 2024-03-04T12:27:23.278 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:22 smithi063 ceph-mon[36717]: pgmap v48: 129 pgs: 129 active+clean; 16 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-03-04T12:27:23.278 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:22 smithi060 ceph-mon[32745]: pgmap v48: 129 pgs: 129 active+clean; 16 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-03-04T12:27:23.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T12:27:23.279 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T12:27:23.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:22 smithi086 ceph-mon[36851]: pgmap v48: 129 pgs: 129 active+clean; 16 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-03-04T12:27:23.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.379+0000 7fd1ee589640 1 -- 172.21.15.60:0/1563077648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1e8100410 msgr2=0x7fd1e81007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.379+0000 7fd1ee589640 1 --2- 172.21.15.60:0/1563077648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1e8100410 0x7fd1e81007f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009900 tx=0x7fd1d802bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:23.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.379+0000 7fd1ee589640 1 -- 172.21.15.60:0/1563077648 shutdown_connections 2024-03-04T12:27:23.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.379+0000 7fd1ee589640 1 --2- 172.21.15.60:0/1563077648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1e81054e0 0x7fd1e810d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.379+0000 7fd1ee589640 1 --2- 172.21.15.60:0/1563077648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e8100dc0 0x7fd1e8104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.391 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.379+0000 7fd1ee589640 1 --2- 172.21.15.60:0/1563077648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1e8100410 0x7fd1e81007f0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.379+0000 7fd1ee589640 1 -- 172.21.15.60:0/1563077648 >> 172.21.15.60:0/1563077648 conn(0x7fd1e8077330 msgr2=0x7fd1e8077740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:23.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.379+0000 7fd1ee589640 1 -- 172.21.15.60:0/1563077648 shutdown_connections 2024-03-04T12:27:23.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.379+0000 7fd1ee589640 1 -- 172.21.15.60:0/1563077648 wait complete. 2024-03-04T12:27:23.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.380+0000 7fd1ee589640 1 Processor -- start 2024-03-04T12:27:23.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.380+0000 7fd1ee589640 1 -- start start 2024-03-04T12:27:23.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.380+0000 7fd1ee589640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1e8100410 0x7fd1e8199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.380+0000 7fd1ee589640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e81054e0 0x7fd1e819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.392 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.380+0000 7fd1ee589640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1e819e600 0x7fd1e819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.380+0000 7fd1ee589640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e810f4d0 con 0x7fd1e819e600 2024-03-04T12:27:23.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.380+0000 7fd1ee589640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e810f350 con 0x7fd1e81054e0 2024-03-04T12:27:23.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.380+0000 7fd1ee589640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e810f650 con 0x7fd1e8100410 2024-03-04T12:27:23.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.380+0000 7fd1ecb26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1e819e600 0x7fd1e819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.380+0000 7fd1e77fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e81054e0 0x7fd1e819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1ecb26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1e819e600 0x7fd1e819eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34562/0 (socket says 172.21.15.60:34562) 2024-03-04T12:27:23.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1ecb26640 1 -- 172.21.15.60:0/1550128621 learned_addr learned my addr 172.21.15.60:0/1550128621 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:23.393 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1e7fff640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1e8100410 0x7fd1e8199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1e77fe640 1 -- 172.21.15.60:0/1550128621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1e8100410 msgr2=0x7fd1e8199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1e77fe640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1e8100410 0x7fd1e8199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1e77fe640 1 -- 172.21.15.60:0/1550128621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1e819e600 msgr2=0x7fd1e819eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1e77fe640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1e819e600 0x7fd1e819eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1e77fe640 1 -- 172.21.15.60:0/1550128621 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e819f0f0 con 0x7fd1e81054e0 2024-03-04T12:27:23.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1ecb26640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1e819e600 0x7fd1e819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:23.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1e77fe640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e81054e0 0x7fd1e819a280 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00b3c0 tx=0x7fd1dc009c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:23.394 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1e57fa640 1 -- 172.21.15.60:0/1550128621 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc00e8a0 con 0x7fd1e81054e0 2024-03-04T12:27:23.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1ee589640 1 -- 172.21.15.60:0/1550128621 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e819f3e0 con 0x7fd1e81054e0 2024-03-04T12:27:23.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1e57fa640 1 -- 172.21.15.60:0/1550128621 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1dc019070 con 0x7fd1e81054e0 2024-03-04T12:27:23.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1e57fa640 1 -- 172.21.15.60:0/1550128621 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc013380 con 0x7fd1e81054e0 2024-03-04T12:27:23.395 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.381+0000 7fd1ee589640 1 -- 172.21.15.60:0/1550128621 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e81a6bd0 con 0x7fd1e81054e0 2024-03-04T12:27:23.402 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1e57fa640 1 -- 172.21.15.60:0/1550128621 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 26) v1 ==== 136986+0+0 (secure 0 0 0) 0x7fd1dc00ea40 con 0x7fd1e81054e0 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1f116c740 1 Processor -- start 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1f116c740 1 -- start start 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1f116c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393e6790 0x5619393e6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1f116c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393ef750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1f116c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 0x5619393f2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1f116c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x561939349e00 con 0x5619393efc90 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1f116c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619392f2b10 con 0x5619393e6790 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1f116c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x561939345ea0 con 0x5619393e70b0 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1e77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 0x5619393f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1e77fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 0x5619393f2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34568/0 (socket says 172.21.15.60:34568) 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1e77fe640 1 -- 172.21.15.60:0/1758742835 learned_addr learned my addr 172.21.15.60:0/1758742835 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1ecb26640 1 --2- 172.21.15.60:0/1758742835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393e6790 0x5619393e6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.403 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.384+0000 7fd1e7fff640 1 --2- 172.21.15.60:0/1758742835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393ef750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1e57fa640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd1d00a2010 0x7fd1d00a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1e77fe640 1 -- 172.21.15.60:0/1758742835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 msgr2=0x5619393ef750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1e77fe640 1 --2- 172.21.15.60:0/1758742835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393ef750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1e77fe640 1 -- 172.21.15.60:0/1758742835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393e6790 msgr2=0x5619393e6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1e77fe640 1 --2- 172.21.15.60:0/1758742835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393e6790 0x5619393e6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1e77fe640 1 -- 172.21.15.60:0/1758742835 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619391287d0 con 0x5619393efc90 2024-03-04T12:27:23.404 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1ecb26640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd1d00a2010 0x7fd1d00a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1e57fa640 1 -- 172.21.15.60:0/1550128621 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fd1dc0ce520 con 0x7fd1e81054e0 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1e77fe640 1 --2- 172.21.15.60:0/1758742835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 0x5619393f2080 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0cb780 tx=0x7fd1dc0659c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5cfeff3d5f89b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1ecb26640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd1d00a2010 0x7fd1d00a44d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00049d0 tx=0x7fd1e0004a40 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1ce7fc640 1 -- 172.21.15.60:0/1758742835 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc09a400 con 0x5619393efc90 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1ce7fc640 1 -- 172.21.15.60:0/1758742835 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1dc09afc0 con 0x5619393efc90 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.385+0000 7fd1ce7fc640 1 -- 172.21.15.60:0/1758742835 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc0c4de0 con 0x5619393efc90 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.386+0000 7fd1f116c740 1 -- 172.21.15.60:0/1758742835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 msgr2=0x5619393f2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.386+0000 7fd1f116c740 1 --2- 172.21.15.60:0/1758742835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 0x5619393f2080 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0cb780 tx=0x7fd1dc0659c0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.386+0000 7fd1f116c740 1 -- 172.21.15.60:0/1758742835 shutdown_connections 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.386+0000 7fd1f116c740 1 --2- 172.21.15.60:0/1758742835 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 0x5619393f2080 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.386+0000 7fd1f116c740 1 --2- 172.21.15.60:0/1758742835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393ef750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.386+0000 7fd1f116c740 1 --2- 172.21.15.60:0/1758742835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393e6790 0x5619393e6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.386+0000 7fd1f116c740 1 -- 172.21.15.60:0/1758742835 >> 172.21.15.60:0/1758742835 conn(0x5619393e0ec0 msgr2=0x5619393e12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:23.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.386+0000 7fd1f116c740 1 -- 172.21.15.60:0/1758742835 shutdown_connections 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.386+0000 7fd1f116c740 1 -- 172.21.15.60:0/1758742835 wait complete. 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.386+0000 7fd1f116c740 1 -- start start 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.386+0000 7fd1f116c740 1 Processor -- start 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1f116c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393e6790 0x5619393ef340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1f116c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393ec6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1f116c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 0x5619393ecc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1f116c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619393f29c0 con 0x5619393efc90 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1f116c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619393060a0 con 0x5619393e6790 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1f116c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619393f2b40 con 0x5619393e70b0 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1f116c740 10 client.0 _ll_register_callbacks cb 0x561939305a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1ecb26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 0x5619393ecc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1e7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393e6790 0x5619393ef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1ecb26640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 0x5619393ecc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34584/0 (socket says 172.21.15.60:34584) 2024-03-04T12:27:23.406 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1e7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393e6790 0x5619393ef340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:36554/0 (socket says 172.21.15.60:36554) 2024-03-04T12:27:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1ecb26640 1 -- 172.21.15.60:0/4267416191 learned_addr learned my addr 172.21.15.60:0/4267416191 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1e77fe640 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393ec6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1ecb26640 1 -- 172.21.15.60:0/4267416191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 msgr2=0x5619393ec6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1ecb26640 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393ec6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1ecb26640 1 -- 172.21.15.60:0/4267416191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393e6790 msgr2=0x5619393ef340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1ecb26640 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393e6790 0x5619393ef340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1ecb26640 1 -- 172.21.15.60:0/4267416191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56193905e3f0 con 0x5619393efc90 2024-03-04T12:27:23.407 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1e7fff640 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393e6790 0x5619393ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1ecb26640 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 0x5619393ecc10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd1e001c6a0 tx=0x7fd1e0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef22164d1147a0af server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.387+0000 7fd1c77fe640 1 -- 172.21.15.60:0/4267416191 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e002c060 con 0x5619393efc90 2024-03-04T12:27:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1c77fe640 1 -- 172.21.15.60:0/4267416191 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fd1e000e940 con 0x5619393efc90 2024-03-04T12:27:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1f116c740 1 -- 172.21.15.60:0/4267416191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1dc09afc0 con 0x5619393efc90 2024-03-04T12:27:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1c77fe640 1 -- 172.21.15.60:0/4267416191 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1e0033050 con 0x5619393efc90 2024-03-04T12:27:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1f116c740 10 client.16377 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1f116c740 1 -- 172.21.15.60:0/4267416191 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x56193901e0f0 con 0x5619393efc90 2024-03-04T12:27:23.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1c77fe640 1 -- 172.21.15.60:0/4267416191 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1dc09afc0 con 0x5619393efc90 2024-03-04T12:27:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1c77fe640 1 -- 172.21.15.60:0/4267416191 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fd1e0027ca0 con 0x5619393efc90 2024-03-04T12:27:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1f116c740 10 client.16377 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1f116c740 10 client.16377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1f116c740 20 client.16377 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1f116c740 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5619393ff670 0x561939401aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1f116c740 4 client.16377 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1f116c740 1 -- 172.21.15.60:0/4267416191 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561939357340 con 0x5619393ff670 2024-03-04T12:27:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.388+0000 7fd1e7fff640 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5619393ff670 0x561939401aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.389+0000 7fd1e7fff640 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5619393ff670 0x561939401aa0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a50c2798ede18eb server_cookie=1685be25f2f6c1bd in_seq=0 out_seq=0 2024-03-04T12:27:23.409 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.389+0000 7fd1c77fe640 10 client.16377 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.389+0000 7fd1c77fe640 1 -- 172.21.15.60:0/4267416191 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561939357340 con 0x5619393ff670 2024-03-04T12:27:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.389+0000 7fd1c77fe640 10 client.16377 handle_command_reply: tid=0 2024-03-04T12:27:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.389+0000 7fd1f116c740 1 client.16377 shutdown 2024-03-04T12:27:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.390+0000 7fd1f116c740 1 -- 172.21.15.60:0/4267416191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 msgr2=0x5619393ecc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.390+0000 7fd1f116c740 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 0x5619393ecc10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fd1e001c6a0 tx=0x7fd1e0004ab0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.390+0000 7fd1f116c740 1 -- 172.21.15.60:0/4267416191 shutdown_connections 2024-03-04T12:27:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.390+0000 7fd1f116c740 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x5619393ff670 0x561939401aa0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.390+0000 7fd1f116c740 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393efc90 0x5619393ecc10 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.410 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.390+0000 7fd1f116c740 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393ec6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.390+0000 7fd1f116c740 1 --2- 172.21.15.60:0/4267416191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393e6790 0x5619393ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.390+0000 7fd1c77fe640 0 client.16377 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:23.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.390+0000 7fd1f116c740 1 -- 172.21.15.60:0/4267416191 >> 172.21.15.60:0/4267416191 conn(0x5619393e0ec0 msgr2=0x5619393e2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:23.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.390+0000 7fd1f116c740 1 -- 172.21.15.60:0/4267416191 shutdown_connections 2024-03-04T12:27:23.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.390+0000 7fd1f116c740 1 -- 172.21.15.60:0/4267416191 wait complete. 2024-03-04T12:27:23.411 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.390+0000 7fd1f116c740 20 client.16377 trim_cache size 0 max 16384 2024-03-04T12:27:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1f116c740 1 Processor -- start 2024-03-04T12:27:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1f116c740 1 -- start start 2024-03-04T12:27:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1f116c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 0x5619393eee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1f116c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393ef340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1f116c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393efc90 0x5619393f1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1f116c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e81b0570 con 0x5619393e6790 2024-03-04T12:27:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1f116c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e81b1f20 con 0x5619393efc90 2024-03-04T12:27:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1f116c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e81b0bd0 con 0x5619393e70b0 2024-03-04T12:27:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1e7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 0x5619393eee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1e7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 0x5619393eee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34596/0 (socket says 172.21.15.60:34596) 2024-03-04T12:27:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1e7fff640 1 -- 172.21.15.60:0/791841999 learned_addr learned my addr 172.21.15.60:0/791841999 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:23.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1ecb26640 1 --2- 172.21.15.60:0/791841999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393efc90 0x5619393f1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1e77fe640 1 --2- 172.21.15.60:0/791841999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393ef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1e7fff640 1 -- 172.21.15.60:0/791841999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 msgr2=0x5619393ef340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1e7fff640 1 --2- 172.21.15.60:0/791841999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393ef340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1e7fff640 1 -- 172.21.15.60:0/791841999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393efc90 msgr2=0x5619393f1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1e7fff640 1 --2- 172.21.15.60:0/791841999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393efc90 0x5619393f1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1e7fff640 1 -- 172.21.15.60:0/791841999 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561939406c90 con 0x5619393e6790 2024-03-04T12:27:23.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.405+0000 7fd1ecb26640 1 --2- 172.21.15.60:0/791841999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393efc90 0x5619393f1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:23.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1e7fff640 1 --2- 172.21.15.60:0/791841999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 0x5619393eee00 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009230 tx=0x7fd1d802df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae4e4e7c74efb6e server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:23.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1cdffb640 1 -- 172.21.15.60:0/791841999 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d80043d0 con 0x5619393e6790 2024-03-04T12:27:23.420 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1cdffb640 1 -- 172.21.15.60:0/791841999 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1d802c880 con 0x5619393e6790 2024-03-04T12:27:23.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1cdffb640 1 -- 172.21.15.60:0/791841999 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8004ae0 con 0x5619393e6790 2024-03-04T12:27:23.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1f116c740 1 -- 172.21.15.60:0/791841999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 msgr2=0x5619393eee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1f116c740 1 --2- 172.21.15.60:0/791841999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 0x5619393eee00 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009230 tx=0x7fd1d802df90 comp rx=0 tx=0).stop 2024-03-04T12:27:23.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1f116c740 1 -- 172.21.15.60:0/791841999 shutdown_connections 2024-03-04T12:27:23.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1f116c740 1 --2- 172.21.15.60:0/791841999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393efc90 0x5619393f1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1f116c740 1 --2- 172.21.15.60:0/791841999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1f116c740 1 --2- 172.21.15.60:0/791841999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 0x5619393eee00 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1f116c740 1 -- 172.21.15.60:0/791841999 >> 172.21.15.60:0/791841999 conn(0x5619393e0ec0 msgr2=0x5619393ec690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:23.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1f116c740 1 -- 172.21.15.60:0/791841999 shutdown_connections 2024-03-04T12:27:23.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.406+0000 7fd1f116c740 1 -- 172.21.15.60:0/791841999 wait complete. 2024-03-04T12:27:23.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1f116c740 1 -- start start 2024-03-04T12:27:23.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1f116c740 1 Processor -- start 2024-03-04T12:27:23.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1f116c740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 0x5619393f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1f116c740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393f71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1f116c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393efc90 0x5619393f3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1f116c740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e81b18c0 con 0x5619393e6790 2024-03-04T12:27:23.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1f116c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e81b1230 con 0x5619393efc90 2024-03-04T12:27:23.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1f116c740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e81ae810 con 0x5619393e70b0 2024-03-04T12:27:23.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1f116c740 10 client.0 _ll_register_callbacks cb 0x7fd1e000fde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1e7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 0x5619393f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1e7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 0x5619393f6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:34612/0 (socket says 172.21.15.60:34612) 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1e7fff640 1 -- 172.21.15.60:0/485118615 learned_addr learned my addr 172.21.15.60:0/485118615 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1ecb26640 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393efc90 0x5619393f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.407+0000 7fd1e77fe640 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393f71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1e7fff640 1 -- 172.21.15.60:0/485118615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 msgr2=0x5619393f71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1e7fff640 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393f71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1e7fff640 1 -- 172.21.15.60:0/485118615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393efc90 msgr2=0x5619393f3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1e7fff640 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393efc90 0x5619393f3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1e7fff640 1 -- 172.21.15.60:0/485118615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619393f3a60 con 0x5619393e6790 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1ecb26640 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393efc90 0x5619393f3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1e7fff640 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 0x5619393f6cb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8005940 tx=0x7fd1d80045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a8b64ab0109b0d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:23.423 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1c77fe640 1 -- 172.21.15.60:0/485118615 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8004a60 con 0x5619393e6790 2024-03-04T12:27:23.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1c77fe640 1 -- 172.21.15.60:0/485118615 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fd1d8004c00 con 0x5619393e6790 2024-03-04T12:27:23.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1f116c740 1 -- 172.21.15.60:0/485118615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1d802c880 con 0x5619393e6790 2024-03-04T12:27:23.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1c77fe640 1 -- 172.21.15.60:0/485118615 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1d804c430 con 0x5619393e6790 2024-03-04T12:27:23.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1f116c740 10 client.16383 fetch_fsmap learned FSMap version 32 2024-03-04T12:27:23.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1f116c740 1 -- 172.21.15.60:0/485118615 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x561939456790 con 0x5619393e6790 2024-03-04T12:27:23.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1c77fe640 1 -- 172.21.15.60:0/485118615 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1d802c880 con 0x5619393e6790 2024-03-04T12:27:23.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1c77fe640 1 -- 172.21.15.60:0/485118615 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 32) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fd1d80035f0 con 0x5619393e6790 2024-03-04T12:27:23.424 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1f116c740 10 client.16383 fetch_fsmap finished waiting for FSMap version 32 2024-03-04T12:27:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1f116c740 10 client.16383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1f116c740 20 client.16383 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.408+0000 7fd1f116c740 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x561939456e90 0x5619394592b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1f116c740 4 client.16383 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T12:27:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1f116c740 1 -- 172.21.15.60:0/485118615 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561939351690 con 0x561939456e90 2024-03-04T12:27:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1e77fe640 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x561939456e90 0x5619394592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1c77fe640 10 client.16383 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1e77fe640 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x561939456e90 0x5619394592b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee56490c0c560814 server_cookie=a1aa2cda50273808 in_seq=0 out_seq=0 2024-03-04T12:27:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1c77fe640 1 -- 172.21.15.60:0/485118615 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x561939351690 con 0x561939456e90 2024-03-04T12:27:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1c77fe640 10 client.16383 handle_command_reply: tid=0 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1f116c740 1 client.16383 shutdown 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1f116c740 1 -- 172.21.15.60:0/485118615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 msgr2=0x5619393f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1f116c740 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 0x5619393f6cb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8005940 tx=0x7fd1d80045b0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1f116c740 1 -- 172.21.15.60:0/485118615 shutdown_connections 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1f116c740 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x561939456e90 0x5619394592b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1f116c740 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5619393efc90 0x5619393f3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1f116c740 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5619393e70b0 0x5619393f71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1f116c740 1 --2- 172.21.15.60:0/485118615 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x5619393e6790 0x5619393f6cb0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1c77fe640 0 client.16383 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.409+0000 7fd1f116c740 1 -- 172.21.15.60:0/485118615 >> 172.21.15.60:0/485118615 conn(0x5619393e0ec0 msgr2=0x5619393eafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1f116c740 1 -- 172.21.15.60:0/485118615 shutdown_connections 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1f116c740 1 -- 172.21.15.60:0/485118615 wait complete. 2024-03-04T12:27:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1f116c740 20 client.16383 trim_cache size 0 max 16384 2024-03-04T12:27:23.427 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:27:23.427 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-04T12:27:23.427 INFO:teuthology.orchestra.run.smithi060.stdout: "scrubs": { 2024-03-04T12:27:23.427 INFO:teuthology.orchestra.run.smithi060.stdout: "84da1cba-13e6-4cf2-8d43-dc95e1b3b36f": { 2024-03-04T12:27:23.427 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/", 2024-03-04T12:27:23.427 INFO:teuthology.orchestra.run.smithi060.stdout: "tag": "84da1cba-13e6-4cf2-8d43-dc95e1b3b36f", 2024-03-04T12:27:23.427 INFO:teuthology.orchestra.run.smithi060.stdout: "options": "recursive,force" 2024-03-04T12:27:23.427 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:27:23.427 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:27:23.427 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:27:23.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1ee589640 1 -- 172.21.15.60:0/1550128621 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd1d00a2010 msgr2=0x7fd1d00a44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1ee589640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd1d00a2010 0x7fd1d00a44d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00049d0 tx=0x7fd1e0004a40 comp rx=0 tx=0).stop 2024-03-04T12:27:23.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1ee589640 1 -- 172.21.15.60:0/1550128621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e81054e0 msgr2=0x7fd1e819a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:23.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1ee589640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e81054e0 0x7fd1e819a280 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00b3c0 tx=0x7fd1dc009c60 comp rx=0 tx=0).stop 2024-03-04T12:27:23.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1ee589640 1 -- 172.21.15.60:0/1550128621 shutdown_connections 2024-03-04T12:27:23.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1ee589640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fd1d00a2010 0x7fd1d00a44d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1ee589640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fd1e819e600 0x7fd1e819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.428 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1ee589640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e81054e0 0x7fd1e819a280 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1ee589640 1 --2- 172.21.15.60:0/1550128621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1e8100410 0x7fd1e8199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:23.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1ee589640 1 -- 172.21.15.60:0/1550128621 >> 172.21.15.60:0/1550128621 conn(0x7fd1e8077330 msgr2=0x7fd1e80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:23.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1ee589640 1 -- 172.21.15.60:0/1550128621 shutdown_connections 2024-03-04T12:27:23.429 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:23.410+0000 7fd1ee589640 1 -- 172.21.15.60:0/1550128621 wait complete. 2024-03-04T12:27:23.430 INFO:tasks.cephfs.filesystem:scrub status for tag:84da1cba-13e6-4cf2-8d43-dc95e1b3b36f - {'path': '/', 'tag': '84da1cba-13e6-4cf2-8d43-dc95e1b3b36f', 'options': 'recursive,force'} 2024-03-04T12:27:23.531 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.269+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000002b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/MAINTAINERS auth v178 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf37080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000002b:[<0x10000000007/MAINTAINERS v178>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000002b:[<0x10000000007/MAINTAINERS v178>,<0x10000000006/xfstests-dev v88>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.532 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.270+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000236 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ auth{0=1} fragtree_t(*^3) v796 ap=1 f(v0 m2024-03-04T12:27:15.008894+0000 156=152+4) n(v1 rc2024-03-04T12:27:15.008894+0000 201=196+5) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=8 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x564e5cf37600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000236:[<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000236:[<0x10000000007/src v796>,<0x10000000006/xfstests-dev v88>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:15.008894+0000 156=152+4)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:15.008894+0000 201=196+5)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:15.008894+0000 156=152+4)","memory_value.rstat":"n(v1 rc2024-03-04T12:27:15.008894+0000 201=196+5)","error_str":""},"return_code":0} 2024-03-04T12:27:23.532 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.270+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022c [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/ auth{0=1} v338 ap=1 f(v0 m2024-03-04T12:27:13.808906+0000 6=6+0) n(v0 rc2024-03-04T12:27:13.808906+0000 7=6+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5cf37b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x1000000022c:[<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x1000000022c:[<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v88>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.808906+0000 6=6+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.808906+0000 7=6+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.808906+0000 6=6+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.808906+0000 7=6+1)","error_str":""},"return_code":0} 2024-03-04T12:27:23.532 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.271+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000230 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/ auth{0=1} v404 ap=1 f(v0 m2024-03-04T12:27:13.822906+0000 14=14+0) n(v0 rc2024-03-04T12:27:13.822906+0000 15=14+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5cf38680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000230:[<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000230:[<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v88>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.822906+0000 14=14+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.822906+0000 15=14+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.822906+0000 14=14+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.822906+0000 15=14+1)","error_str":""},"return_code":0} 2024-03-04T12:27:23.532 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.271+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000033 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.selftest auth v194 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf38100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000033:[<0x10000000007/README.selftest v194>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000033:[<0x10000000007/README.selftest v194>,<0x10000000006/xfstests-dev v88>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.532 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.272+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022e [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/ auth{0=1} v374 ap=1 f(v0 m2024-03-04T12:27:13.813906+0000 17=17+0) n(v0 rc2024-03-04T12:27:13.813906+0000 18=17+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5cf39180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x1000000022e:[<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x1000000022e:[<0x10000000007/include v374>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.813906+0000 17=17+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.813906+0000 18=17+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.813906+0000 17=17+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.813906+0000 18=17+1)","error_str":""},"return_code":0} 2024-03-04T12:27:23.532 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.273+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000002f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.config-sections auth v186 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf38c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000002f:[<0x10000000007/README.config-sections v186>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000002f:[<0x10000000007/README.config-sections v186>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.532 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.273+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000002c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/Makefile auth v180 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf39700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000002c:[<0x10000000007/Makefile v180>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000002c:[<0x10000000007/Makefile v180>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.532 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.274+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000a5 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/new auth v458 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf3c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000a5:[<0x10000000007/new v458>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000a5:[<0x10000000007/new v458>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.532 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.274+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000035 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/acinclude.m4 auth v198 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf3c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000035:[<0x10000000007/acinclude.m4 v198>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000035:[<0x10000000007/acinclude.m4 v198>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.532 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.274+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000234 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/ auth{0=1} v456 ap=1 f(v0 m2024-03-04T12:27:13.836906+0000 13=13+0) n(v0 rc2024-03-04T12:27:13.836906+0000 14=13+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5cf3cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000234:[<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000234:[<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.836906+0000 13=13+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.836906+0000 14=13+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.836906+0000 13=13+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.836906+0000 14=13+1)","error_str":""},"return_code":0} 2024-03-04T12:27:23.533 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.275+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000008 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/ auth{0=1} v786 ap=1 f(v0 m2024-03-04T12:27:13.755907+0000 12=6+6) n(v0 rc2024-03-04T12:27:13.755907+0000 b13163136 44=27+17) (inest lock dirty) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | dirtyscattered=1 importingcaps=0 dirfrag=1 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x564e5cf3d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000008:[<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000008:[<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.755907+0000 12=6+6)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.755907+0000 b13163136 44=27+17)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.755907+0000 12=6+6)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.755907+0000 b13163136 44=27+17)","error_str":""},"return_code":0} 2024-03-04T12:27:23.533 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.275+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000a6 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/randomize.awk auth v460 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf3d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000a6:[<0x10000000007/randomize.awk v460>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000a6:[<0x10000000007/randomize.awk v460>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.533 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.275+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022a [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/ auth{0=1} v324 ap=1 f(v0 m2024-03-04T12:27:13.804906+0000 3=3+0) n(v0 rc2024-03-04T12:27:13.804906+0000 4=3+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5cf3db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x1000000022a:[<0x10000000007/crash v324>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x1000000022a:[<0x10000000007/crash v324>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.804906+0000 3=3+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.804906+0000 4=3+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.804906+0000 3=3+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.804906+0000 4=3+1)","error_str":""},"return_code":0} 2024-03-04T12:27:23.533 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.275+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000002d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/Makepkgs auth v182 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf3f180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000002d:[<0x10000000007/Makepkgs v182>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000002d:[<0x10000000007/Makepkgs v182>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.533 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.276+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000090 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lsqa.pl auth v408 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf3f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000090:[<0x10000000007/lsqa.pl v408>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000090:[<0x10000000007/lsqa.pl v408>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.533 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.276+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000232 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/ auth{0=1} v428 ap=1 f(v0 m2024-03-04T12:27:13.832906+0000 9=9+0) n(v0 rc2024-03-04T12:27:13.832906+0000 10=9+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5cf3e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000232:[<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000232:[<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.832906+0000 9=9+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.832906+0000 10=9+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.832906+0000 9=9+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.832906+0000 10=9+1)","error_str":""},"return_code":0} 2024-03-04T12:27:23.533 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.276+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000030 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.device-mapper auth v188 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf48580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000030:[<0x10000000007/README.device-mapper v188>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000030:[<0x10000000007/README.device-mapper v188>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.533 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.277+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000002e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README auth v184 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf48000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000002e:[<0x10000000007/README v184>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000002e:[<0x10000000007/README v184>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.533 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.277+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000008f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/local.config.example auth v406 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf3e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000008f:[<0x10000000007/local.config.example v406>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000008f:[<0x10000000007/local.config.example v406>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.533 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.278+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000031 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.fuse auth v190 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf3ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000031:[<0x10000000007/README.fuse v190>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000031:[<0x10000000007/README.fuse v190>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.534 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.279+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000a7 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/release.sh auth v462 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf48b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000a7:[<0x10000000007/release.sh v462>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000a7:[<0x10000000007/release.sh v462>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.534 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.279+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000006a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configure.ac auth v316 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf49080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000006a:[<0x10000000007/configure.ac v316>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000006a:[<0x10000000007/configure.ac v316>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.534 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.279+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000226 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ auth{0=1} v308 ap=1 f(v0 m2024-03-04T12:27:13.783907+0000 49=49+0) n(v0 rc2024-03-04T12:27:13.783907+0000 50=49+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5cf49600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000226:[<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000226:[<0x10000000007/common v308>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.783907+0000 49=49+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.783907+0000 50=49+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.783907+0000 49=49+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.783907+0000 50=49+1)","error_str":""},"return_code":0} 2024-03-04T12:27:23.534 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.279+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000a9 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/soak auth v466 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf49b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000a9:[<0x10000000007/soak v466>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000a9:[<0x10000000007/soak v466>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.534 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.280+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000032 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.overlay auth v192 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf4a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000032:[<0x10000000007/README.overlay v192>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000032:[<0x10000000007/README.overlay v192>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.534 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.280+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000034 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/VERSION auth v196 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf4a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000034:[<0x10000000007/VERSION v196>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000034:[<0x10000000007/VERSION v196>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.534 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.280+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021e [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/LICENSES/ auth{0=1} v176 ap=1 f(v0 m2024-03-04T12:27:13.760907+0000 1=1+0) n(v0 rc2024-03-04T12:27:13.760907+0000 2=1+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5cf4ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x1000000021e:[<0x10000000007/LICENSES v176>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x1000000021e:[<0x10000000007/LICENSES v176>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.760907+0000 1=1+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.760907+0000 2=1+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.760907+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.760907+0000 2=1+1)","error_str":""},"return_code":0} 2024-03-04T12:27:23.534 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.280+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.gitignore auth v172 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf4b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021d:[<0x10000000007/.gitignore v172>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000021d:[<0x10000000007/.gitignore v172>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.534 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.281+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000228 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/ auth{0=1} v314 ap=1 f(v0 m2024-03-04T12:27:13.800907+0000 2=2+0) n(v0 rc2024-03-04T12:27:13.800907+0000 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5cf4b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000228:[<0x10000000007/configs v314>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000228:[<0x10000000007/configs v314>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.800907+0000 2=2+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.800907+0000 3=2+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.800907+0000 2=2+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.800907+0000 3=2+1)","error_str":""},"return_code":0} 2024-03-04T12:27:23.534 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.281+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000038 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/check auth v208 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf52000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000038:[<0x10000000007/check v208>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000038:[<0x10000000007/check v208>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.534 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.281+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000220 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/ auth{0=1} v778 ap=1 f(v0 m2024-03-04T12:27:13.771907+0000 3=1+2) n(v1 rc2024-03-04T12:27:13.773907+0000 8=5+3) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x564e5cf52b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000220:[<0x10000000007/build v206>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000220:[<0x10000000007/build v778>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.771907+0000 3=1+2)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.773907+0000 8=5+3)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.771907+0000 3=1+2)","memory_value.rstat":"n(v1 rc2024-03-04T12:27:13.773907+0000 8=5+3)","error_str":""},"return_code":0} 2024-03-04T12:27:23.535 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.282+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000a8 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/setup auth v464 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf52580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000a8:[<0x10000000007/setup v464>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000a8:[<0x10000000007/setup v464>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.535 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.282+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000147 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_1.c auth v330 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf53080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000147:[<0x10000000236/t_readdir_1.c v330>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000147:[<0x10000000236/t_readdir_1.c v330>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v90>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.535 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.433+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000f1 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/holes.c auth v150 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf53600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000f1:[<0x10000000236/holes.c v150>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000f1:[<0x10000000236/holes.c v150>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.535 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.434+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000e3 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2fs_check auth v122 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf54100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000e3:[<0x10000000236/fill2fs_check v122>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000e3:[<0x10000000236/fill2fs_check v122>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.535 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.434+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000014b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_rename_overwrite.c auth v338 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf54680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000014b:[<0x10000000236/t_rename_overwrite.c v338>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000014b:[<0x10000000236/t_rename_overwrite.c v338>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.535 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.434+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000157 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/usemem.c auth v362 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf54c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000157:[<0x10000000236/usemem.c v362>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000157:[<0x10000000236/usemem.c v362>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.535 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.434+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000140 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_stale_pmd.c auth v316 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf55180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000140:[<0x10000000236/t_mmap_stale_pmd.c v316>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000140:[<0x10000000236/t_mmap_stale_pmd.c v316>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.535 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.434+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000c8 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/checkpoint_journal.c auth v68 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf55700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000c8:[<0x10000000236/checkpoint_journal.c v68>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000c8:[<0x10000000236/checkpoint_journal.c v68>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.535 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.434+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000d2 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirhash_collide.c auth v88 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf5e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000d2:[<0x10000000236/dirhash_collide.c v88>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000d2:[<0x10000000236/dirhash_collide.c v88>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.535 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.435+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000119 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/randholes.c auth v238 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf5eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000119:[<0x10000000236/randholes.c v238>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000119:[<0x10000000236/randholes.c v238>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.535 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.435+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000e6 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fssum.c auth v128 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf5e000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000e6:[<0x10000000236/fssum.c v128>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000e6:[<0x10000000236/fssum.c v128>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.535 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.435+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000c9 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/chprojid_fail.c auth v70 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf5fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000c9:[<0x10000000236/chprojid_fail.c v70>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000c9:[<0x10000000236/chprojid_fail.c v70>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.543 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.435+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000bd [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/alloc.c auth v46 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf5f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000bd:[<0x10000000236/alloc.c v46>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000bd:[<0x10000000236/alloc.c v46>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.543 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.435+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000c3 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bstat.c auth v58 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf5f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000c3:[<0x10000000236/bstat.c v58>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000c3:[<0x10000000236/bstat.c v58>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.543 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.435+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000134 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ext4_dax_inline_corruption.c auth v292 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf60100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000134:[<0x10000000236/t_ext4_dax_inline_corruption.c v292>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000134:[<0x10000000236/t_ext4_dax_inline_corruption.c v292>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.544 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.435+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000108 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/nsexec.c auth v200 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf60680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000108:[<0x10000000236/nsexec.c v200>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000108:[<0x10000000236/nsexec.c v200>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.544 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.436+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000014f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_truncate_self.c auth v346 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf60c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000014f:[<0x10000000236/t_truncate_self.c v346>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000014f:[<0x10000000236/t_truncate_self.c v346>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.544 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.436+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000135 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ext4_dax_journal_corruption.c auth v294 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf61180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000135:[<0x10000000236/t_ext4_dax_journal_corruption.c v294>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000135:[<0x10000000236/t_ext4_dax_journal_corruption.c v294>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.544 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.436+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ec [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/getdevicesize.c auth v140 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf61700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ec:[<0x10000000236/getdevicesize.c v140>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000ec:[<0x10000000236/getdevicesize.c v140>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.544 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.436+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000337 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/ auth v222 ap=1 f(v0 m2024-03-04T12:27:13.897906+0000 7=7+0) n(v0 rc2024-03-04T12:27:13.897906+0000 8=7+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf68000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000337:[<0x10000000236/perf v222>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[]","memoryvalue":"(2)0x10000000337:[<0x10000000236/perf v222>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.897906+0000 7=7+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.897906+0000 8=7+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.897906+0000 7=7+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.897906+0000 8=7+1)","error_str":""},"return_code":0} 2024-03-04T12:27:23.544 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.442+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ca [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/cloner.c auth v72 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf68580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ca:[<0x10000000236/cloner.c v72>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000ca:[<0x10000000236/cloner.c v72>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.544 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.442+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000cf [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-buf-fault.c auth v82 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf6b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000cf:[<0x10000000236/dio-buf-fault.c v82>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000cf:[<0x10000000236/dio-buf-fault.c v82>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.544 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.442+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000014c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_snapshot_deleted_subvolume.c auth v340 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf74000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000014c:[<0x10000000236/t_snapshot_deleted_subvolume.c v340>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000014c:[<0x10000000236/t_snapshot_deleted_subvolume.c v340>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.544 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.442+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000d5 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dmerror auth v94 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf74580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000d5:[<0x10000000236/dmerror v94>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000d5:[<0x10000000236/dmerror v94>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.544 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.442+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000101 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/metaperf.c auth v186 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf74b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000101:[<0x10000000236/metaperf.c v186>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000101:[<0x10000000236/metaperf.c v186>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.544 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.442+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000012d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_create_long_dirs.c auth v278 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf75080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000012d:[<0x10000000236/t_create_long_dirs.c v278>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000012d:[<0x10000000236/t_create_long_dirs.c v278>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.544 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.443+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000014a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_reflink_read_race.c auth v336 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf75600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000014a:[<0x10000000236/t_reflink_read_race.c v336>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000014a:[<0x10000000236/t_reflink_read_race.c v336>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.545 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.443+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000d3 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirperf.c auth v90 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf76680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000d3:[<0x10000000236/dirperf.c v90>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000d3:[<0x10000000236/dirperf.c v90>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.545 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.443+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000e8 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fsync-err.c auth v132 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf76100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000e8:[<0x10000000236/fsync-err.c v132>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000e8:[<0x10000000236/fsync-err.c v132>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.545 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.443+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000117 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/punch-alternating.c auth v234 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf75b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000117:[<0x10000000236/punch-alternating.c v234>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000117:[<0x10000000236/punch-alternating.c v234>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.545 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.443+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000155 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/unwritten_sync.c auth v358 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf77180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000155:[<0x10000000236/unwritten_sync.c v358>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000155:[<0x10000000236/unwritten_sync.c v358>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.545 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.443+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000115 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/preallo_rw_pattern_reader.c auth v230 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf77700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000115:[<0x10000000236/preallo_rw_pattern_reader.c v230>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000115:[<0x10000000236/preallo_rw_pattern_reader.c v230>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.545 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.444+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000e4 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fs_perms.c auth v124 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf76c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000e4:[<0x10000000236/fs_perms.c v124>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000e4:[<0x10000000236/fs_perms.c v124>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.545 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.444+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000148 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_2.c auth v332 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf80000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000148:[<0x10000000236/t_readdir_2.c v332>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000148:[<0x10000000236/t_readdir_2.c v332>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.545 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.444+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000132 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_encrypted_d_revalidate.c auth v288 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf80580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000132:[<0x10000000236/t_encrypted_d_revalidate.c v288>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000132:[<0x10000000236/t_encrypted_d_revalidate.c v288>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.545 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.444+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000125 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/splice-test.c auth v262 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf80b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000125:[<0x10000000236/splice-test.c v262>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000125:[<0x10000000236/splice-test.c v262>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.545 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.448+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000123 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/seek_sanity_test.c auth v258 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf81080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000123:[<0x10000000236/seek_sanity_test.c v258>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000123:[<0x10000000236/seek_sanity_test.c v258>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.545 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.448+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000145 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ofd_locks.c auth v326 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf81600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000145:[<0x10000000236/t_ofd_locks.c v326>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000145:[<0x10000000236/t_ofd_locks.c v326>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.546 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.448+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000122 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/seek_copy_test.c auth v256 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf81b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000122:[<0x10000000236/seek_copy_test.c v256>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000122:[<0x10000000236/seek_copy_test.c v256>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.546 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.448+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000011f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/runas.c auth v250 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf82680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000011f:[<0x10000000236/runas.c v250>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000011f:[<0x10000000236/runas.c v250>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.546 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.450+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000109 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/open_by_handle.c auth v202 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf82c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000109:[<0x10000000236/open_by_handle.c v202>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000109:[<0x10000000236/open_by_handle.c v202>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.546 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.451+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000116 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/preallo_rw_pattern_writer.c auth v232 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf83180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000116:[<0x10000000236/preallo_rw_pattern_writer.c v232>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000116:[<0x10000000236/preallo_rw_pattern_writer.c v232>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.546 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.451+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000013a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_immutable.c auth v304 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf83700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000013a:[<0x10000000236/t_immutable.c v304>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000013a:[<0x10000000236/t_immutable.c v304>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.546 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.451+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000cb [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dbtest.c auth v74 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf8c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000cb:[<0x10000000236/dbtest.c v74>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000cb:[<0x10000000236/dbtest.c v74>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.546 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.451+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000104 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmap-write-concurrent.c auth v192 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf8c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000104:[<0x10000000236/mmap-write-concurrent.c v192>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000104:[<0x10000000236/mmap-write-concurrent.c v192>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.546 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.453+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000e1 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2attr auth v118 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf8cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000e1:[<0x10000000236/fill2attr v118>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000e1:[<0x10000000236/fill2attr v118>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.546 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.453+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000e9 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fsync-tester.c auth v134 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf8e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000e9:[<0x10000000236/fsync-tester.c v134>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000e9:[<0x10000000236/fsync-tester.c v134>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.546 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.453+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000dc [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/feature.c auth v108 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf8d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000dc:[<0x10000000236/feature.c v108>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000dc:[<0x10000000236/feature.c v108>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.546 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.453+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000f5 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/listxattr.c auth v158 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf8db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000f5:[<0x10000000236/listxattr.c v158>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000f5:[<0x10000000236/listxattr.c v158>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.546 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.453+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000013d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_cow_race.c auth v310 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf8d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000013d:[<0x10000000236/t_mmap_cow_race.c v310>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000013d:[<0x10000000236/t_mmap_cow_race.c v310>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.460+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000151 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/testx.c auth v350 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf8e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000151:[<0x10000000236/testx.c v350>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000151:[<0x10000000236/testx.c v350>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.460+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000fc [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/looptest.c auth v176 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf8ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000fc:[<0x10000000236/looptest.c v176>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000fc:[<0x10000000236/looptest.c v176>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.460+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000d8 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/enospc_unlink.c auth v100 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf8f180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000d8:[<0x10000000236/enospc_unlink.c v100>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000d8:[<0x10000000236/enospc_unlink.c v100>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.460+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000d1 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-invalidate-cache.c auth v86 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf8f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000d1:[<0x10000000236/dio-invalidate-cache.c v86>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000d1:[<0x10000000236/dio-invalidate-cache.c v86>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.461+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000d0 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-interleaved.c auth v84 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf98000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000d0:[<0x10000000236/dio-interleaved.c v84>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000d0:[<0x10000000236/dio-interleaved.c v84>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.461+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000d4 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirstress.c auth v92 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf98580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000d4:[<0x10000000236/dirstress.c v92>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000d4:[<0x10000000236/dirstress.c v92>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.462+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000bf [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/append_reader.c auth v50 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf98b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000bf:[<0x10000000236/append_reader.c v50>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000bf:[<0x10000000236/append_reader.c v50>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.463+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000012a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/swapon.c auth v272 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf99600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000012a:[<0x10000000236/swapon.c v272>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000012a:[<0x10000000236/swapon.c v272>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.463+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000dd [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fiemap-tester.c auth v110 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf99080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000dd:[<0x10000000236/fiemap-tester.c v110>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000dd:[<0x10000000236/fiemap-tester.c v110>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.464+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000e2 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2fs auth v120 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf99b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000e2:[<0x10000000236/fill2fs v120>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000e2:[<0x10000000236/fill2fs v120>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.465+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000126 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/splice2pipe.c auth v264 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf9a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000126:[<0x10000000236/splice2pipe.c v264>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000126:[<0x10000000236/splice2pipe.c v264>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.466+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000c6 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_unlink_test.c auth v64 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf9a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000c6:[<0x10000000236/bulkstat_unlink_test.c v64>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000c6:[<0x10000000236/bulkstat_unlink_test.c v64>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.547 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.466+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000118 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/pwrite_mmap_blocked.c auth v236 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf9ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000118:[<0x10000000236/pwrite_mmap_blocked.c v236>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000118:[<0x10000000236/pwrite_mmap_blocked.c v236>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.548 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.467+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000149 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_3.c auth v334 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf9b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000149:[<0x10000000236/t_readdir_3.c v334>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000149:[<0x10000000236/t_readdir_3.c v334>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.548 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.468+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000152 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/trunc.c auth v352 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf9b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000152:[<0x10000000236/trunc.c v352>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000152:[<0x10000000236/trunc.c v352>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.548 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.468+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000fe [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/makeextents.c auth v180 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfa6000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000fe:[<0x10000000236/makeextents.c v180>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000fe:[<0x10000000236/makeextents.c v180>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.548 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.469+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000f4 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/itrash.c auth v156 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfa6580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000f4:[<0x10000000236/itrash.c v156>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000f4:[<0x10000000236/itrash.c v156>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.548 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.470+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000127 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/stale_handle.c auth v266 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfa6b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000127:[<0x10000000236/stale_handle.c v266>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000127:[<0x10000000236/stale_handle.c v266>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.548 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.470+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ed [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/getpagesize.c auth v142 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfa7080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ed:[<0x10000000236/getpagesize.c v142>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000ed:[<0x10000000236/getpagesize.c v142>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.548 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.471+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000012b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_access_root.c auth v274 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfa7600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000012b:[<0x10000000236/t_access_root.c v274>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000012b:[<0x10000000236/t_access_root.c v274>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.548 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.471+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000011d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/resvtest.c auth v246 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfa7b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000011d:[<0x10000000236/resvtest.c v246>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000011d:[<0x10000000236/resvtest.c v246>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.548 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.472+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000133 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_enospc.c auth v290 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfa8100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000133:[<0x10000000236/t_enospc.c v290>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000133:[<0x10000000236/t_enospc.c v290>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.548 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.473+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000010a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/parse-dev-tree.awk auth v204 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfa8680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000010a:[<0x10000000236/parse-dev-tree.awk v204>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000010a:[<0x10000000236/parse-dev-tree.awk v204>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.548 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.474+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000146 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_open_tmpfiles.c auth v328 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfa8c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000146:[<0x10000000236/t_open_tmpfiles.c v328>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000146:[<0x10000000236/t_open_tmpfiles.c v328>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.548 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.474+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000011b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/rename.c auth v242 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfa9180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000011b:[<0x10000000236/rename.c v242>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000011b:[<0x10000000236/rename.c v242>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.475+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000db [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fault.c auth v106 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfa9700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000db:[<0x10000000236/fault.c v106>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000db:[<0x10000000236/fault.c v106>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.476+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000237 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/Makefile auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfb0000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000237:[<0x10000000236/Makefile v2>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000237:[<0x10000000236/Makefile v2>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.476+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000141 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_write_ro.c auth v318 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfb0580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000141:[<0x10000000236/t_mmap_write_ro.c v318>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000141:[<0x10000000236/t_mmap_write_ro.c v318>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.477+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000014d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_stripealign.c auth v342 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfb0b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000014d:[<0x10000000236/t_stripealign.c v342>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000014d:[<0x10000000236/t_stripealign.c v342>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.478+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000107 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/nametest.c auth v198 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfb1080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000107:[<0x10000000236/nametest.c v198>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000107:[<0x10000000236/nametest.c v198>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.479+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000c2 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/attr_replace_test.c auth v56 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfb1600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000c2:[<0x10000000236/attr_replace_test.c v56>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000c2:[<0x10000000236/attr_replace_test.c v56>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.479+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000128 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/stat_test.c auth v268 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfb1b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000128:[<0x10000000236/stat_test.c v268>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000128:[<0x10000000236/stat_test.c v268>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.551 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.480+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000f2 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/holetest.c auth v152 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfb2100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000f2:[<0x10000000236/holetest.c v152>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000f2:[<0x10000000236/holetest.c v152>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.551 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.481+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000cd [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/detached_mounts_propagation.c auth v78 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfb2680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000cd:[<0x10000000236/detached_mounts_propagation.c v78>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000cd:[<0x10000000236/detached_mounts_propagation.c v78>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.551 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.482+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000106 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/multi_open_unlink.c auth v196 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfb2c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000106:[<0x10000000236/multi_open_unlink.c v196>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000106:[<0x10000000236/multi_open_unlink.c v196>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.551 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.482+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000105 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmapcat.c auth v194 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfb3180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000105:[<0x10000000236/mmapcat.c v194>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000105:[<0x10000000236/mmapcat.c v194>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.551 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.483+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000011a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/readdir-while-renames.c auth v240 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfb3700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000011a:[<0x10000000236/readdir-while-renames.c v240>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000011a:[<0x10000000236/readdir-while-renames.c v240>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.551 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.484+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000112 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/permname.c auth v224 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfbe000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000112:[<0x10000000236/permname.c v224>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000112:[<0x10000000236/permname.c v224>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.551 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.485+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000f3 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/iopat.c auth v154 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfbe580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000f3:[<0x10000000236/iopat.c v154>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000f3:[<0x10000000236/iopat.c v154>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.551 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.486+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000d6 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dumpfile auth v96 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfbeb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000d6:[<0x10000000236/dumpfile v96>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000d6:[<0x10000000236/dumpfile v96>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.551 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.488+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000e0 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2.c auth v116 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfbfb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000e0:[<0x10000000236/fill2.c v116>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000e0:[<0x10000000236/fill2.c v116>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.552 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.489+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000df [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill.c auth v114 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfc0100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000df:[<0x10000000236/fill.c v114>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000df:[<0x10000000236/fill.c v114>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.552 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.490+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000f6 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/locktest.c auth v160 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfc0680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000f6:[<0x10000000236/locktest.c v160>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000f6:[<0x10000000236/locktest.c v160>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.552 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.490+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000102 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mkswap.c auth v188 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfc0c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000102:[<0x10000000236/mkswap.c v188>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000102:[<0x10000000236/mkswap.c v188>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.552 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.491+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000cc [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/deduperace.c auth v76 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfc1180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000cc:[<0x10000000236/deduperace.c v76>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000cc:[<0x10000000236/deduperace.c v76>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.552 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.492+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000143 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_writev_overlap.c auth v322 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfc1700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000143:[<0x10000000236/t_mmap_writev_overlap.c v322>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000143:[<0x10000000236/t_mmap_writev_overlap.c v322>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.552 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.492+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000139 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_holes.c auth v302 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfca000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000139:[<0x10000000236/t_holes.c v302>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000139:[<0x10000000236/t_holes.c v302>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.552 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.493+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000131 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_type.c auth v286 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfca580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000131:[<0x10000000236/t_dir_type.c v286>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000131:[<0x10000000236/t_dir_type.c v286>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.552 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.494+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000012f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_offset.c auth v282 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cfcab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000012f:[<0x10000000236/t_dir_offset.c v282>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000012f:[<0x10000000236/t_dir_offset.c v282>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.552 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.495+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000103 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmap-rw-fault.c auth v190 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf68b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000103:[<0x10000000236/mmap-rw-fault.c v190>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000103:[<0x10000000236/mmap-rw-fault.c v190>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.496+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000010b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/parse-extent-tree.awk auth v206 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0e3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000010b:[<0x10000000236/parse-extent-tree.awk v206>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000010b:[<0x10000000236/parse-extent-tree.awk v206>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.497+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ff [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/md5.c auth v182 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0e4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ff:[<0x10000000236/md5.c v182>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000ff:[<0x10000000236/md5.c v182>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.498+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000238 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/ auth v44 ap=1 f(v0 m2024-03-04T12:27:13.849906+0000 19=19+0) n(v0 rc2024-03-04T12:27:13.849906+0000 20=19+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0e4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000238:[<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[]","memoryvalue":"(2)0x10000000238:[<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.849906+0000 19=19+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.849906+0000 20=19+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.849906+0000 19=19+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.849906+0000 20=19+1)","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.498+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000114 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/popdir.pl auth v228 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0e4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000114:[<0x10000000236/popdir.pl v228>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000114:[<0x10000000236/popdir.pl v228>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.499+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000011c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/renameat2.c auth v244 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0e5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000011c:[<0x10000000236/renameat2.c v244>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000011c:[<0x10000000236/renameat2.c v244>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.500+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000129 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/statx.h auth v270 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0e5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000129:[<0x10000000236/statx.h v270>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000129:[<0x10000000236/statx.h v270>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.501+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000f0 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/helpers.h auth v148 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0ec000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000f0:[<0x10000000236/helpers.h v148>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000f0:[<0x10000000236/helpers.h v148>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.501+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000130 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_offset2.c auth v284 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0ec580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000130:[<0x10000000236/t_dir_offset2.c v284>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000130:[<0x10000000236/t_dir_offset2.c v284>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.502+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000144 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mtab.c auth v324 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0ecb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000144:[<0x10000000236/t_mtab.c v324>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000144:[<0x10000000236/t_mtab.c v324>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.503+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000da [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fake-dump-rootino.c auth v104 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0ed080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000da:[<0x10000000236/fake-dump-rootino.c v104>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000da:[<0x10000000236/fake-dump-rootino.c v104>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.504+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ee [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/global.h auth v144 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0ed600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ee:[<0x10000000236/global.h v144>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000ee:[<0x10000000236/global.h v144>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.504+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000de [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fiexchange.h auth v112 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0edb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000de:[<0x10000000236/fiexchange.h v112>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000de:[<0x10000000236/fiexchange.h v112>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.553 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.505+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000154 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/unwritten_mmap.c auth v356 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d107080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000154:[<0x10000000236/unwritten_mmap.c v356>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000154:[<0x10000000236/unwritten_mmap.c v356>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.505+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023a [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/ auth v172 ap=1 f(v0 m2024-03-04T12:27:13.886906+0000 5=5+0) n(v0 rc2024-03-04T12:27:13.886906+0000 6=5+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d107600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x1000000023a:[<0x10000000236/log-writes v172>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[]","memoryvalue":"(2)0x1000000023a:[<0x10000000236/log-writes v172>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.886906+0000 5=5+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.886906+0000 6=5+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.886906+0000 5=5+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.886906+0000 6=5+1)","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.506+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000c0 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/append_writer.c auth v52 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d107b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000c0:[<0x10000000236/append_writer.c v52>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000c0:[<0x10000000236/append_writer.c v52>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.507+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ea [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ftrunc.c auth v136 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d108100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ea:[<0x10000000236/ftrunc.c v136>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000ea:[<0x10000000236/ftrunc.c v136>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.507+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000be [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/allocstale.c auth v48 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d108680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000be:[<0x10000000236/allocstale.c v48>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000be:[<0x10000000236/allocstale.c v48>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.508+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000d7 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/e4compact.c auth v98 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d108c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000d7:[<0x10000000236/e4compact.c v98>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000d7:[<0x10000000236/e4compact.c v98>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.509+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000124 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/soak_duration.awk auth v260 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d109180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000124:[<0x10000000236/soak_duration.awk v260>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000124:[<0x10000000236/soak_duration.awk v260>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.509+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000fd [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/lstat64.c auth v178 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d109700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000fd:[<0x10000000236/lstat64.c v178>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000fd:[<0x10000000236/lstat64.c v178>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.510+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000136 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_futimens.c auth v296 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d112000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000136:[<0x10000000236/t_futimens.c v296>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000136:[<0x10000000236/t_futimens.c v296>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.510+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000113 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/popattr.py auth v226 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d112580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000113:[<0x10000000236/popattr.py v226>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000113:[<0x10000000236/popattr.py v226>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.511+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000013f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_fallocate.c auth v314 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d112b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000013f:[<0x10000000236/t_mmap_fallocate.c v314>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000013f:[<0x10000000236/t_mmap_fallocate.c v314>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.512+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000012e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_create_short_dirs.c auth v280 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d113080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000012e:[<0x10000000236/t_create_short_dirs.c v280>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000012e:[<0x10000000236/t_create_short_dirs.c v280>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.512+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000d9 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ext4_resize.c auth v102 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d113600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000d9:[<0x10000000236/ext4_resize.c v102>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000d9:[<0x10000000236/ext4_resize.c v102>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.554 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.513+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000138 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_getcwd.c auth v300 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d113b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000138:[<0x10000000236/t_getcwd.c v300>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000138:[<0x10000000236/t_getcwd.c v300>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.555 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.514+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000012c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_attr_corruption.c auth v276 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d114100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000012c:[<0x10000000236/t_attr_corruption.c v276>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000012c:[<0x10000000236/t_attr_corruption.c v276>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.555 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.514+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ef [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/godown.c auth v146 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d114680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ef:[<0x10000000236/godown.c v146>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000ef:[<0x10000000236/godown.c v146>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.555 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.515+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000156 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/uring_read_fault.c auth v360 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d114c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000156:[<0x10000000236/uring_read_fault.c v360>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000156:[<0x10000000236/uring_read_fault.c v360>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.555 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.515+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000c5 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_null_ocount.c auth v62 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d115180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000c5:[<0x10000000236/bulkstat_null_ocount.c v62>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000c5:[<0x10000000236/bulkstat_null_ocount.c v62>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.555 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.516+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000013e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_dio.c auth v312 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d115700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000013e:[<0x10000000236/t_mmap_dio.c v312>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000013e:[<0x10000000236/t_mmap_dio.c v312>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.555 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.517+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000011e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/rewinddir-test.c auth v248 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d11c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000011e:[<0x10000000236/rewinddir-test.c v248>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000011e:[<0x10000000236/rewinddir-test.c v248>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.555 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.517+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000eb [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/genhashnames.c auth v138 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d11c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000eb:[<0x10000000236/genhashnames.c v138>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000eb:[<0x10000000236/genhashnames.c v138>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.555 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.518+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000013c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_cow_memory_failure.c auth v308 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d11cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000013c:[<0x10000000236/t_mmap_cow_memory_failure.c v308>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000013c:[<0x10000000236/t_mmap_cow_memory_failure.c v308>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.555 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.519+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000158 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/uuid_ioctl.c auth v364 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d11f180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000158:[<0x10000000236/uuid_ioctl.c v364>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000158:[<0x10000000236/uuid_ioctl.c v364>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.555 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.520+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000fb [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/loggen.c auth v174 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d11f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000fb:[<0x10000000236/loggen.c v174>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000fb:[<0x10000000236/loggen.c v174>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.555 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.520+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000c4 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/btrfs_crc32c_forged_name.py auth v60 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d12c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000c4:[<0x10000000236/btrfs_crc32c_forged_name.py v60>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000c4:[<0x10000000236/btrfs_crc32c_forged_name.py v60>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.555 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.521+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ce [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/devzero.c auth v80 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d12c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ce:[<0x10000000236/devzero.c v80>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000ce:[<0x10000000236/devzero.c v80>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.556 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.522+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000e5 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fscrypt-crypt-util.c auth v126 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d12cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000e5:[<0x10000000236/fscrypt-crypt-util.c v126>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000e5:[<0x10000000236/fscrypt-crypt-util.c v126>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.556 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.523+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000100 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/md5.h auth v184 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d12d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000100:[<0x10000000236/md5.h v184>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000100:[<0x10000000236/md5.h v184>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.556 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.524+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000c1 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/attr-list-by-handle-cursor-test.c auth v54 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d12d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000c1:[<0x10000000236/attr-list-by-handle-cursor-test.c v54>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000c1:[<0x10000000236/attr-list-by-handle-cursor-test.c v54>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.556 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.524+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000142 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_writev.c auth v320 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d12db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000142:[<0x10000000236/t_mmap_writev.c v320>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000142:[<0x10000000236/t_mmap_writev.c v320>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.556 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.525+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000137 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_get_file_time.c auth v298 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d12e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000137:[<0x10000000236/t_get_file_time.c v298>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000137:[<0x10000000236/t_get_file_time.c v298>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.556 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.527+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000e7 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fstest.c auth v130 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d12e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000e7:[<0x10000000236/fstest.c v130>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000e7:[<0x10000000236/fstest.c v130>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.556 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.527+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000aa [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/af_unix.c auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d12ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000aa:[<0x10000000236/af_unix.c v4>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000aa:[<0x10000000236/af_unix.c v4>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.556 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.528+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000013b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_collision.c auth v306 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d12f180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000013b:[<0x10000000236/t_mmap_collision.c v306>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000013b:[<0x10000000236/t_mmap_collision.c v306>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.556 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.529+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000121 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/scaleread.sh auth v254 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d12f700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000121:[<0x10000000236/scaleread.sh v254>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000121:[<0x10000000236/scaleread.sh v254>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.556 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.530+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000014e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_truncate_cmtime.c auth v344 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d136000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000014e:[<0x10000000236/t_truncate_cmtime.c v344>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x1000000014e:[<0x10000000236/t_truncate_cmtime.c v344>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.556 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.531+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000120 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/scaleread.c auth v252 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d136580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000120:[<0x10000000236/scaleread.c v252>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000120:[<0x10000000236/scaleread.c v252>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.556 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.532+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000153 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/truncfile.c auth v354 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d136b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000153:[<0x10000000236/truncfile.c v354>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000153:[<0x10000000236/truncfile.c v354>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.557 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.533+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000150 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/test-nextquota.c auth v348 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d137080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000150:[<0x10000000236/test-nextquota.c v348>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x10000000150:[<0x10000000236/test-nextquota.c v348>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.557 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.534+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000c7 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_unlink_test_modified.c auth v66 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d137600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000c7:[<0x10000000236/bulkstat_unlink_test_modified.c v66>,<0x10000000007/src v784>,<0x10000000006/xfstests-dev v82>,<0x10000000005/tmp v15>,<0x100000001fb/client.0 v16>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v30>,<0x10000000001/sv_1 v52>,<0x10000000000/qa v42>,<0x1/volumes v37>]//[3]","memoryvalue":"(4)0x100000000c7:[<0x10000000236/bulkstat_unlink_test_modified.c v66>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.853 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.552+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000006d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/auxiliary-programs.txt auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d423700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000006d:[<0x1000000022c/auxiliary-programs.txt v4>,<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000006d:[<0x1000000022c/auxiliary-programs.txt v4>,<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.853 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.553+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000006f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/requirement-checking.txt auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d432000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000006f:[<0x1000000022c/requirement-checking.txt v8>,<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000006f:[<0x1000000022c/requirement-checking.txt v8>,<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.853 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.554+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000006e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/group-names.txt auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d432580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000006e:[<0x1000000022c/group-names.txt v6>,<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000006e:[<0x1000000022c/group-names.txt v6>,<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.853 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.555+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/CHANGES auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d432b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000022d:[<0x1000000022c/CHANGES v2>,<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000022d:[<0x1000000022c/CHANGES v2>,<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.853 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.557+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000071 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/xunit.xsd auth v12 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d433080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000071:[<0x1000000022c/xunit.xsd v12>,<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000071:[<0x1000000022c/xunit.xsd v12>,<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.854 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.558+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000070 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/testing-afs.txt auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d433600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000070:[<0x1000000022c/testing-afs.txt v10>,<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000070:[<0x1000000022c/testing-afs.txt v10>,<0x10000000007/doc v338>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.854 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.559+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000008c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/string_to_tokens.c auth v24 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d433b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000008c:[<0x10000000230/string_to_tokens.c v24>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000008c:[<0x10000000230/string_to_tokens.c v24>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.854 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.651+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000083 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/databin.c auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d434100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000083:[<0x10000000230/databin.c v6>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000083:[<0x10000000230/databin.c v6>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.854 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.652+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000086 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/forker.c auth v12 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d434680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000086:[<0x10000000230/forker.c v12>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000086:[<0x10000000230/forker.c v12>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.854 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.654+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000082 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/dataascii.c auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d434c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000082:[<0x10000000230/dataascii.c v4>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000082:[<0x10000000230/dataascii.c v4>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.854 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.655+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000008b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/str_to_bytes.c auth v22 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d435180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000008b:[<0x10000000230/str_to_bytes.c v22>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000008b:[<0x10000000230/str_to_bytes.c v22>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.854 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.656+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000008e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/write_log.c auth v28 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d435700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000008e:[<0x10000000230/write_log.c v28>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000008e:[<0x10000000230/write_log.c v28>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.854 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.658+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000008d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/tlibio.c auth v26 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d43e000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000008d:[<0x10000000230/tlibio.c v26>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000008d:[<0x10000000230/tlibio.c v26>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.854 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.763+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000085 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/file_lock.c auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d43e580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000085:[<0x10000000230/file_lock.c v10>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000085:[<0x10000000230/file_lock.c v10>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.854 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.764+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000008a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/random_range.c auth v20 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d43eb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000008a:[<0x10000000230/random_range.c v20>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000008a:[<0x10000000230/random_range.c v20>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.854 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.764+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000084 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/datapid.c auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d43f080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000084:[<0x10000000230/datapid.c v8>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000084:[<0x10000000230/datapid.c v8>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.854 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.765+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000231 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/Makefile auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d43f600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000231:[<0x10000000230/Makefile v2>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000231:[<0x10000000230/Makefile v2>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.855 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.766+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000087 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/open_flags.c auth v14 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d43fb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000087:[<0x10000000230/open_flags.c v14>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000087:[<0x10000000230/open_flags.c v14>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.855 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.767+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000088 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/pattern.c auth v16 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d440100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000088:[<0x10000000230/pattern.c v16>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000088:[<0x10000000230/pattern.c v16>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.855 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.768+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000089 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/random.c auth v18 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d440680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000089:[<0x10000000230/random.c v18>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000089:[<0x10000000230/random.c v18>,<0x10000000007/lib v404>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:23.855 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.768+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000078 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/file_lock.h auth v16 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d440c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000078:[<0x1000000022e/file_lock.h v16>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000078:[<0x1000000022e/file_lock.h v16>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:23 smithi086 ceph-mon[36851]: scrub queued for path: / 2024-03-04T12:27:24.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:23 smithi086 ceph-mon[36851]: scrub summary: idle+waiting paths [/] 2024-03-04T12:27:24.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:23 smithi086 ceph-mon[36851]: scrub summary: active paths [/] 2024-03-04T12:27:24.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:23 smithi086 ceph-mon[36851]: scrub summary: idle+waiting paths [/] 2024-03-04T12:27:24.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:23 smithi086 ceph-mon[36851]: scrub summary: active paths [/] 2024-03-04T12:27:24.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:23 smithi086 ceph-mon[36851]: scrub summary: idle+waiting paths [/] 2024-03-04T12:27:24.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-mon[36717]: scrub queued for path: / 2024-03-04T12:27:24.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-mon[36717]: scrub summary: idle+waiting paths [/] 2024-03-04T12:27:24.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-mon[36717]: scrub summary: active paths [/] 2024-03-04T12:27:24.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-mon[36717]: scrub summary: idle+waiting paths [/] 2024-03-04T12:27:24.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-mon[36717]: scrub summary: active paths [/] 2024-03-04T12:27:24.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-mon[36717]: scrub summary: idle+waiting paths [/] 2024-03-04T12:27:24.354 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.900+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000007c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/pattern.h auth v24 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d441180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000007c:[<0x1000000022e/pattern.h v24>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000007c:[<0x1000000022e/pattern.h v24>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.354 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.900+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000007f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/string_to_tokens.h auth v30 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d441700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000007f:[<0x1000000022e/string_to_tokens.h v30>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000007f:[<0x1000000022e/string_to_tokens.h v30>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.354 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.901+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000081 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/write_log.h auth v34 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d44a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000081:[<0x1000000022e/write_log.h v34>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000081:[<0x1000000022e/write_log.h v34>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.354 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.901+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000075 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildrules auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d44a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000075:[<0x1000000022e/buildrules v10>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000075:[<0x1000000022e/buildrules v10>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.354 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.902+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000007a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/install-sh auth v20 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d44ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000007a:[<0x1000000022e/install-sh v20>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000007a:[<0x1000000022e/install-sh v20>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.354 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.903+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/Makefile auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d44b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000022f:[<0x1000000022e/Makefile v2>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000022f:[<0x1000000022e/Makefile v2>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.354 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.903+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000007e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/str_to_bytes.h auth v28 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d44b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000007e:[<0x1000000022e/str_to_bytes.h v28>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000007e:[<0x1000000022e/str_to_bytes.h v28>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.354 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.904+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000074 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildmacros auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d44bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000074:[<0x1000000022e/buildmacros v8>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000074:[<0x1000000022e/buildmacros v8>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.354 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.905+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000077 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/databin.h auth v14 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d44c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000077:[<0x1000000022e/databin.h v14>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000077:[<0x1000000022e/databin.h v14>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.354 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.905+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000079 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/forker.h auth v18 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d44c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000079:[<0x1000000022e/forker.h v18>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000079:[<0x1000000022e/forker.h v18>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.355 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.906+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000080 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/tlibio.h auth v32 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d44cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000080:[<0x1000000022e/tlibio.h v32>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000080:[<0x1000000022e/tlibio.h v32>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.355 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.907+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000076 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/dataascii.h auth v12 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d44d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000076:[<0x1000000022e/dataascii.h v12>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000076:[<0x1000000022e/dataascii.h v12>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.355 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.907+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000007d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/random_range.h auth v26 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d44d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000007d:[<0x1000000022e/random_range.h v26>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000007d:[<0x1000000022e/random_range.h v26>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.355 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.908+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000007b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/open_flags.h auth v22 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d45a000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000007b:[<0x1000000022e/open_flags.h v22>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000007b:[<0x1000000022e/open_flags.h v22>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.355 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.909+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000072 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/builddefs.in auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d45a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000072:[<0x1000000022e/builddefs.in v4>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000072:[<0x1000000022e/builddefs.in v4>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.355 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.910+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000073 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildgrouplist auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d45ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000073:[<0x1000000022e/buildgrouplist v6>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000073:[<0x1000000022e/buildgrouplist v6>,<0x10000000007/include v374>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.355 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.910+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000a1 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_liburing.m4 auth v20 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d45b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000a1:[<0x10000000234/package_liburing.m4 v20>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000a1:[<0x10000000234/package_liburing.m4 v20>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.355 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.911+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000099 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/multilib.m4 auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d45b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000099:[<0x10000000234/multilib.m4 v4>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000099:[<0x10000000234/multilib.m4 v4>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.355 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.912+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000009e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libbtrfs.m4 auth v14 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d45bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000009e:[<0x10000000234/package_libbtrfs.m4 v14>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000009e:[<0x10000000234/package_libbtrfs.m4 v14>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.355 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.913+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000a2 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_utilies.m4 auth v22 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d45c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000a2:[<0x10000000234/package_utilies.m4 v22>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000a2:[<0x10000000234/package_utilies.m4 v22>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.355 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.913+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000009a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_acldev.m4 auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d45c680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000009a:[<0x10000000234/package_acldev.m4 v6>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000009a:[<0x10000000234/package_acldev.m4 v6>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.356 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.914+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000009f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libcap.m4 auth v16 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d45cc00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000009f:[<0x10000000234/package_libcap.m4 v16>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000009f:[<0x10000000234/package_libcap.m4 v16>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.356 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.915+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000235 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/Makefile auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d45d180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000235:[<0x10000000234/Makefile v2>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000235:[<0x10000000234/Makefile v2>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.359 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.915+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000009c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_gdbmdev.m4 auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d45d700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000009c:[<0x10000000234/package_gdbmdev.m4 v10>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000009c:[<0x10000000234/package_gdbmdev.m4 v10>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.359 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.916+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000009b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_aiodev.m4 auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d468000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000009b:[<0x10000000234/package_aiodev.m4 v8>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000009b:[<0x10000000234/package_aiodev.m4 v8>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.359 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.917+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000a0 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libcdev.m4 auth v18 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d468580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000a0:[<0x10000000234/package_libcdev.m4 v18>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000a0:[<0x10000000234/package_libcdev.m4 v18>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.359 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.917+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000a4 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_xfslibs.m4 auth v26 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d468b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000a4:[<0x10000000234/package_xfslibs.m4 v26>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000a4:[<0x10000000234/package_xfslibs.m4 v26>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.359 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.918+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000a3 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_uuiddev.m4 auth v24 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d469080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000a3:[<0x10000000234/package_uuiddev.m4 v24>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000a3:[<0x10000000234/package_uuiddev.m4 v24>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.359 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.919+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000009d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_globals.m4 auth v12 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d469600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000009d:[<0x10000000234/package_globals.m4 v12>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000009d:[<0x10000000234/package_globals.m4 v12>,<0x10000000007/m4 v456>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.359 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.919+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000002a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/index.lock auth v203 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d469b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000002a:[<0x10000000008/index.lock v203>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000002a:[<0x10000000008/index.lock v203>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.359 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.920+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000025 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/HEAD auth v170 ap=1 s=23 n(v0 rc2024-03-04T12:27:13.712907+0000 b23 1=1+0)/n(v0 b23 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25139 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d46a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000025:[<0x10000000008/HEAD v170>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000025:[<0x10000000008/HEAD v170>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.359 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.922+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000009 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/description auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d46a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000009:[<0x10000000008/description v2>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000009:[<0x10000000008/description v2>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.359 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.922+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020c [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/ auth{0=1} v209 ap=1 f(v0 m2024-03-04T12:27:11.792927+0000 2=0+2) n(v1 rc2024-03-04T12:27:13.751907+0000 b13162810 5=2+3) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x564e5d46ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x1000000020c:[<0x10000000008/objects v136>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x1000000020c:[<0x10000000008/objects v209>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:11.792927+0000 2=0+2)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.751907+0000 b13162810 5=2+3)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:11.792927+0000 2=0+2)","memory_value.rstat":"n(v1 rc2024-03-04T12:27:13.751907+0000 b13162810 5=2+3)","error_str":""},"return_code":0} 2024-03-04T12:27:24.359 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.923+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000205 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/ auth{0=1} v34 ap=1 f(v0 m2024-03-04T12:27:11.740928+0000 13=13+0) n(v0 rc2024-03-04T12:27:11.740928+0000 14=13+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5d46b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000205:[<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000205:[<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:11.740928+0000 13=13+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:11.740928+0000 14=13+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:11.740928+0000 13=13+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:11.740928+0000 14=13+1)","error_str":""},"return_code":0} 2024-03-04T12:27:24.360 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.924+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000214 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/ auth{0=1} v215 ap=1 f(v0 m2024-03-04T12:27:13.717907+0000 2=1+1) n(v2 rc2024-03-04T12:27:13.720907+0000 8=3+5) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x564e5d46b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000214:[<0x10000000008/logs v175>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000214:[<0x10000000008/logs v215>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.717907+0000 2=1+1)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.720907+0000 8=3+5)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.717907+0000 2=1+1)","memory_value.rstat":"n(v2 rc2024-03-04T12:27:13.720907+0000 8=3+5)","error_str":""},"return_code":0} 2024-03-04T12:27:24.360 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.925+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000024 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/packed-refs auth v154 ap=1 s=0 n(v0 rc2024-03-04T12:27:13.701908+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25139 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d474000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000024:[<0x10000000008/packed-refs v154>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000024:[<0x10000000008/packed-refs v154>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.360 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.926+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000029 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/config auth v198 ap=1 s=262 n(v0 rc2024-03-04T12:27:13.748907+0000 b262 1=1+0)/n(v0 b262 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25139 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d474580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000029:[<0x10000000008/config v198>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000029:[<0x10000000008/config v198>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.360 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.926+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000203 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/info/ auth{0=1} v6 ap=1 f(v0 m2024-03-04T12:27:11.737928+0000 1=1+0) n(v0 rc2024-03-04T12:27:11.737928+0000 2=1+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5d474b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000203:[<0x10000000008/info v6>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000203:[<0x10000000008/info v6>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:11.737928+0000 1=1+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:11.737928+0000 2=1+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:11.737928+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:11.737928+0000 2=1+1)","error_str":""},"return_code":0} 2024-03-04T12:27:24.360 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.927+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000208 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/ auth{0=1} v213 ap=1 f(v0 m2024-03-04T12:27:13.703908+0000 3=0+3) n(v1 rc2024-03-04T12:27:13.721907+0000 7=2+5) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x564e5d475080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000208:[<0x10000000008/refs v160>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000208:[<0x10000000008/refs v213>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.703908+0000 3=0+3)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.721907+0000 7=2+5)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.703908+0000 3=0+3)","memory_value.rstat":"n(v1 rc2024-03-04T12:27:13.721907+0000 7=2+5)","error_str":""},"return_code":0} 2024-03-04T12:27:24.360 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.928+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000022 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/shallow auth v142 ap=1 s=41 n(v0 rc2024-03-04T12:27:13.688908+0000 b41 1=1+0)/n(v0 b41 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25139 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d475600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000022:[<0x10000000008/shallow v142>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000022:[<0x10000000008/shallow v142>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.360 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.928+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000207 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/branches/ auth v36 ap=1 f() n(v0 1=0+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/AsLsXsFsx@2},l=25139 | importingcaps=0 dirfrag=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d475b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000207:[<0x10000000008/branches v36>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000207:[<0x10000000008/branches v36>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 1=0+1)","error_str":""},"return_code":0} 2024-03-04T12:27:24.360 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.929+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000006c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/xfscrash auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4dd600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000006c:[<0x1000000022a/xfscrash v6>,<0x10000000007/crash v324>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000006c:[<0x1000000022a/xfscrash v6>,<0x10000000007/crash v324>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.360 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.930+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000006b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/rc.sysinit auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4ddb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000006b:[<0x1000000022a/rc.sysinit v4>,<0x10000000007/crash v324>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000006b:[<0x1000000022a/rc.sysinit v4>,<0x10000000007/crash v324>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.360 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.931+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/README auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4de100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000022b:[<0x1000000022a/README v2>,<0x10000000007/crash v324>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000022b:[<0x1000000022a/README v2>,<0x10000000007/crash v324>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.360 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.932+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000097 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/iogen.c auth v16 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4de680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000097:[<0x10000000232/iogen.c v16>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000097:[<0x10000000232/iogen.c v16>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.933+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000092 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/doio.c auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4dec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000092:[<0x10000000232/doio.c v6>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000092:[<0x10000000232/doio.c v6>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.933+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000096 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/growfiles.c auth v14 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4df180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000096:[<0x10000000232/growfiles.c v14>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000096:[<0x10000000232/growfiles.c v14>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.934+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000098 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/rwtest.sh auth v18 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4df700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000098:[<0x10000000232/rwtest.sh v18>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000098:[<0x10000000232/rwtest.sh v18>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.934+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000233 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/Makefile auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4f2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000233:[<0x10000000232/Makefile v2>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000233:[<0x10000000232/Makefile v2>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.935+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000095 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/fsx.c auth v12 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4f2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000095:[<0x10000000232/fsx.c v12>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000095:[<0x10000000232/fsx.c v12>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.936+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000094 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/fsstress.c auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4f2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000094:[<0x10000000232/fsstress.c v10>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000094:[<0x10000000232/fsstress.c v10>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.937+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000093 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/doio.h auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4f3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000093:[<0x10000000232/doio.h v8>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000093:[<0x10000000232/doio.h v8>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.938+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000091 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/aio-stress.c auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4f3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000091:[<0x10000000232/aio-stress.c v4>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000091:[<0x10000000232/aio-stress.c v4>,<0x10000000007/ltp v428>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.938+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000060 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/repair auth v82 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4f3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000060:[<0x10000000226/repair v82>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000060:[<0x10000000226/repair v82>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.939+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000004e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filter.btrfs auth v46 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4f4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000004e:[<0x10000000226/filter.btrfs v46>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000004e:[<0x10000000226/filter.btrfs v46>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.940+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000061 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/report auth v84 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4f4680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000061:[<0x10000000226/report v84>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000061:[<0x10000000226/report v84>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.941+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000045 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmlogwrites auth v28 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4f4c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000045:[<0x10000000226/dmlogwrites v28>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000045:[<0x10000000226/dmlogwrites v28>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.361 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.941+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000003b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/casefold auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4f5180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000003b:[<0x10000000226/casefold v8>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000003b:[<0x10000000226/casefold v8>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.362 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.942+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000058 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/perf auth v66 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4f5700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000058:[<0x10000000226/perf v66>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000058:[<0x10000000226/perf v66>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.362 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.943+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000004b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/fail_make_request auth v40 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4fe000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000004b:[<0x10000000226/fail_make_request v40>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000004b:[<0x10000000226/fail_make_request v40>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.362 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.943+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000042 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmerror auth v22 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4fe580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000042:[<0x10000000226/dmerror v22>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000042:[<0x10000000226/dmerror v22>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.362 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.944+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000062 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/scsi_debug auth v86 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4feb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000062:[<0x10000000226/scsi_debug v86>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000062:[<0x10000000226/scsi_debug v86>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.362 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.945+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000050 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/gcov auth v50 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4ff080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000050:[<0x10000000226/gcov v50>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000050:[<0x10000000226/gcov v50>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.362 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.946+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000044 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmhugedisk auth v26 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4ff600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000044:[<0x10000000226/dmhugedisk v26>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000044:[<0x10000000226/dmhugedisk v26>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.362 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.946+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000066 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/verity auth v94 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4ffb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000066:[<0x10000000226/verity v94>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000066:[<0x10000000226/verity v94>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.362 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.947+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000067 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/xfs auth v96 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d500100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000067:[<0x10000000226/xfs v96>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000067:[<0x10000000226/xfs v96>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.362 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.948+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000052 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/locktest auth v54 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d500680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000052:[<0x10000000226/locktest v54>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000052:[<0x10000000226/locktest v54>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.362 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.949+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000004c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filestreams auth v42 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d500c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000004c:[<0x10000000226/filestreams v42>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000004c:[<0x10000000226/filestreams v42>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.363 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.950+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000049 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ext4 auth v36 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d501180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000049:[<0x10000000226/ext4 v36>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000049:[<0x10000000226/ext4 v36>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.363 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.950+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000048 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/encrypt auth v34 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d501700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000048:[<0x10000000226/encrypt v34>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000048:[<0x10000000226/encrypt v34>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.363 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.951+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000005a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/preamble auth v70 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d506000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000005a:[<0x10000000226/preamble v70>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000005a:[<0x10000000226/preamble v70>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.363 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.952+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000003e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/config auth v14 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d506580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000003e:[<0x10000000226/config v14>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000003e:[<0x10000000226/config v14>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.363 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.952+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000005d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/rc auth v76 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d506b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000005d:[<0x10000000226/rc v76>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000005d:[<0x10000000226/rc v76>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.363 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.953+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000047 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dump auth v32 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d507080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000047:[<0x10000000226/dump v32>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000047:[<0x10000000226/dump v32>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.363 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.954+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000003d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/cgroup2 auth v12 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d507600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000003d:[<0x10000000226/cgroup2 v12>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000003d:[<0x10000000226/cgroup2 v12>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.363 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.955+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000068 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/zoned auth v98 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d507b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000068:[<0x10000000226/zoned v98>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000068:[<0x10000000226/zoned v98>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.363 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.955+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000005f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/renameat2 auth v80 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d508100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000005f:[<0x10000000226/renameat2 v80>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000005f:[<0x10000000226/renameat2 v80>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.363 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.956+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000227 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/Makefile auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d508680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000227:[<0x10000000226/Makefile v2>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000227:[<0x10000000226/Makefile v2>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.367 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.957+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000054 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/metadump auth v58 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d508c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000054:[<0x10000000226/metadump v58>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000054:[<0x10000000226/metadump v58>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.367 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.958+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000003c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ceph auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d509180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000003c:[<0x10000000226/ceph v10>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000003c:[<0x10000000226/ceph v10>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.367 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.958+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000005b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/punch auth v72 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d509700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000005b:[<0x10000000226/punch v72>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000005b:[<0x10000000226/punch v72>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.367 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.959+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000056 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/nfs auth v62 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d514000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000056:[<0x10000000226/nfs v62>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000056:[<0x10000000226/nfs v62>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.367 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.960+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000065 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ubifs auth v92 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d514580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000065:[<0x10000000226/ubifs v92>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000065:[<0x10000000226/ubifs v92>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.367 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.961+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000043 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmflakey auth v24 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d514b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000043:[<0x10000000226/dmflakey v24>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000043:[<0x10000000226/dmflakey v24>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.367 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.962+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000004d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filter auth v44 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d515080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000004d:[<0x10000000226/filter v44>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000004d:[<0x10000000226/filter v44>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.367 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.963+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000040 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmdelay auth v18 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d515600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000040:[<0x10000000226/dmdelay v18>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000040:[<0x10000000226/dmdelay v18>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.368 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.963+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000055 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/module auth v60 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d515b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000055:[<0x10000000226/module v60>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000055:[<0x10000000226/module v60>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.368 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.964+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000003a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/btrfs auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d516100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000003a:[<0x10000000226/btrfs v6>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000003a:[<0x10000000226/btrfs v6>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.368 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.965+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000005e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/reflink auth v78 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d516680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000005e:[<0x10000000226/reflink v78>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000005e:[<0x10000000226/reflink v78>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.368 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.965+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000004f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/fuzzy auth v48 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d516c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000004f:[<0x10000000226/fuzzy v48>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000004f:[<0x10000000226/fuzzy v48>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.368 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.966+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000063 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/test_names auth v88 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d517180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000063:[<0x10000000226/test_names v88>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000063:[<0x10000000226/test_names v88>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.368 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.967+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000041 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmdust auth v20 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d517700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000041:[<0x10000000226/dmdust v20>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000041:[<0x10000000226/dmdust v20>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.368 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.968+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000039 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/attr auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d524000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000039:[<0x10000000226/attr v4>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000039:[<0x10000000226/attr v4>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.368 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.968+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000005c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/quota auth v74 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d524580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000005c:[<0x10000000226/quota v74>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000005c:[<0x10000000226/quota v74>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.368 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.969+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000046 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmthin auth v30 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d524b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000046:[<0x10000000226/dmthin v30>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000046:[<0x10000000226/dmthin v30>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.368 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.970+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000057 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/overlay auth v64 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d525080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000057:[<0x10000000226/overlay v64>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000057:[<0x10000000226/overlay v64>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.368 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.971+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000053 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/log auth v56 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d525600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000053:[<0x10000000226/log v56>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000053:[<0x10000000226/log v56>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.369 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.972+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000051 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/inject auth v52 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d525b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000051:[<0x10000000226/inject v52>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000051:[<0x10000000226/inject v52>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.371 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.973+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000059 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/populate auth v68 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d526100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000059:[<0x10000000226/populate v68>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000059:[<0x10000000226/populate v68>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.372 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.974+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000004a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/f2fs auth v38 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d526680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000004a:[<0x10000000226/f2fs v38>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000004a:[<0x10000000226/f2fs v38>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.372 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.976+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000064 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/tracing auth v90 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d526c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000064:[<0x10000000226/tracing v90>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000064:[<0x10000000226/tracing v90>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.372 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.976+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000003f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/defrag auth v16 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d527180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000003f:[<0x10000000226/defrag v16>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000003f:[<0x10000000226/defrag v16>,<0x10000000007/common v308>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.372 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.977+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/LICENSES/GPL-2.0 auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d527700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021f:[<0x1000000021e/GPL-2.0 v2>,<0x10000000007/LICENSES v176>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000021f:[<0x1000000021e/GPL-2.0 v2>,<0x10000000007/LICENSES v176>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.372 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.977+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000229 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/example.config auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d530000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000229:[<0x10000000228/example.config v2>,<0x10000000007/configs v314>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000229:[<0x10000000228/example.config v2>,<0x10000000007/configs v314>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.372 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.978+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000069 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/localhost.config auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d530580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000069:[<0x10000000228/localhost.config v4>,<0x10000000007/configs v314>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000069:[<0x10000000228/localhost.config v4>,<0x10000000007/configs v314>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.372 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.979+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000224 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/tar/ auth v14 ap=1 f(v0 m2024-03-04T12:27:13.773907+0000 1=1+0) n(v0 rc2024-03-04T12:27:13.773907+0000 2=1+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d530b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000224:[<0x10000000220/tar v14>,<0x10000000007/build v206>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000224:[<0x10000000220/tar v14>,<0x10000000007/build v778>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.773907+0000 1=1+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.773907+0000 2=1+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.773907+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.773907+0000 2=1+1)","error_str":""},"return_code":0} 2024-03-04T12:27:24.372 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.980+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000221 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/Makefile auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d531080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000221:[<0x10000000220/Makefile v2>,<0x10000000007/build v206>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000221:[<0x10000000220/Makefile v2>,<0x10000000007/build v778>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.372 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.980+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000222 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/ auth v10 ap=1 f(v0 m2024-03-04T12:27:13.771907+0000 3=3+0) n(v0 rc2024-03-04T12:27:13.771907+0000 4=3+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d531600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000222:[<0x10000000220/rpm v10>,<0x10000000007/build v206>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000222:[<0x10000000220/rpm v10>,<0x10000000007/build v778>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.771907+0000 3=3+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.771907+0000 4=3+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.771907+0000 3=3+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.771907+0000 4=3+1)","error_str":""},"return_code":0} 2024-03-04T12:27:24.372 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.981+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000010e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/ResultData.py auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf69080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000010e:[<0x10000000337/ResultData.py v8>,<0x10000000236/perf v222>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000010e:[<0x10000000337/ResultData.py v8>,<0x10000000236/perf v222>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.372 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.982+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000110 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/fio-results.sql auth v12 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf69600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000110:[<0x10000000337/fio-results.sql v12>,<0x10000000236/perf v222>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000110:[<0x10000000337/fio-results.sql v12>,<0x10000000236/perf v222>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.373 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.983+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000111 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/generate-schema.py auth v14 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf69b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000111:[<0x10000000337/generate-schema.py v14>,<0x10000000236/perf v222>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000111:[<0x10000000337/generate-schema.py v14>,<0x10000000236/perf v222>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.373 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.983+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000010d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/Makefile auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf6a100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000010d:[<0x10000000337/Makefile v6>,<0x10000000236/perf v222>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000010d:[<0x10000000337/Makefile v6>,<0x10000000236/perf v222>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.373 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.984+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000010f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/fio-insert-and-compare.py auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf6a680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000010f:[<0x10000000337/fio-insert-and-compare.py v10>,<0x10000000236/perf v222>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000010f:[<0x10000000337/fio-insert-and-compare.py v10>,<0x10000000236/perf v222>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.373 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.985+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000338 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/FioCompare.py auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf6ac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000338:[<0x10000000337/FioCompare.py v2>,<0x10000000236/perf v222>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000338:[<0x10000000337/FioCompare.py v2>,<0x10000000236/perf v222>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.373 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.985+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000010c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/FioResultDecoder.py auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5cf6b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000010c:[<0x10000000337/FioResultDecoder.py v4>,<0x10000000236/perf v222>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000010c:[<0x10000000337/FioResultDecoder.py v4>,<0x10000000236/perf v222>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.373 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.996+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ae [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-cow-race.c auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0ee100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ae:[<0x10000000238/aio-dio-cow-race.c v10>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000ae:[<0x10000000238/aio-dio-cow-race.c v10>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.373 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.996+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000b3 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-hole-filling-race.c auth v20 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0ee680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000b3:[<0x10000000238/aio-dio-hole-filling-race.c v20>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000b3:[<0x10000000238/aio-dio-hole-filling-race.c v20>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.373 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.997+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000bb [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aiocp.c auth v36 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0eec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000bb:[<0x10000000238/aiocp.c v36>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000bb:[<0x10000000238/aiocp.c v36>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.373 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.998+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000bc [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aiodio_sparse2.c auth v38 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0ef180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000bc:[<0x10000000238/aiodio_sparse2.c v38>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000bc:[<0x10000000238/aiodio_sparse2.c v38>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.373 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.998+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000b0 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-eof-race.c auth v14 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0ef700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000b0:[<0x10000000238/aio-dio-eof-race.c v14>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000b0:[<0x10000000238/aio-dio-eof-race.c v14>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.373 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:23.999+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000239 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/Makefile auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0f8000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000239:[<0x10000000238/Makefile v2>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000239:[<0x10000000238/Makefile v2>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.373 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.000+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000b8 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-free-ring-with-bogus-nr-pages.c auth v30 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0f8580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000b8:[<0x10000000238/aio-free-ring-with-bogus-nr-pages.c v30>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000b8:[<0x10000000238/aio-free-ring-with-bogus-nr-pages.c v30>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.001+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ba [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-last-ref-held-by-io.c auth v34 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0f8b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ba:[<0x10000000238/aio-last-ref-held-by-io.c v34>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000ba:[<0x10000000238/aio-last-ref-held-by-io.c v34>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.001+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000b6 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-subblock-eof-read.c auth v26 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0f9080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000b6:[<0x10000000238/aio-dio-subblock-eof-read.c v26>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000b6:[<0x10000000238/aio-dio-subblock-eof-read.c v26>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.002+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000b2 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-fcntl-race.c auth v18 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0f9600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000b2:[<0x10000000238/aio-dio-fcntl-race.c v18>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000b2:[<0x10000000238/aio-dio-fcntl-race.c v18>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.003+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000b9 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-io-setup-with-nonwritable-context-pointer.c auth v32 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0f9b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000b9:[<0x10000000238/aio-io-setup-with-nonwritable-context-pointer.c v32>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000b9:[<0x10000000238/aio-io-setup-with-nonwritable-context-pointer.c v32>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.004+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ad [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-append-write-read-race.c auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0fa100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ad:[<0x10000000238/aio-dio-append-write-read-race.c v8>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000ad:[<0x10000000238/aio-dio-append-write-read-race.c v8>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.004+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000b7 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-write-verify.c auth v28 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0fa680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000b7:[<0x10000000238/aio-dio-write-verify.c v28>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000b7:[<0x10000000238/aio-dio-write-verify.c v28>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.005+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000af [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-cycle-write.c auth v12 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0fac00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000af:[<0x10000000238/aio-dio-cycle-write.c v12>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000af:[<0x10000000238/aio-dio-cycle-write.c v12>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.006+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000b1 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-extend-stat.c auth v16 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0fb180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000b1:[<0x10000000238/aio-dio-extend-stat.c v16>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000b1:[<0x10000000238/aio-dio-extend-stat.c v16>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.006+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000b4 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-invalidate-failure.c auth v22 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d0fb700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000b4:[<0x10000000238/aio-dio-invalidate-failure.c v22>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000b4:[<0x10000000238/aio-dio-invalidate-failure.c v22>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.007+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ab [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/SOURCE auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d106000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ab:[<0x10000000238/SOURCE v4>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000ab:[<0x10000000238/SOURCE v4>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.008+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000b5 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-invalidate-readahead.c auth v24 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d106580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000b5:[<0x10000000238/aio-dio-invalidate-readahead.c v24>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000b5:[<0x10000000238/aio-dio-invalidate-readahead.c v24>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.009+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000ac [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-append-write-fallocate-race.c auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d106b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000ac:[<0x10000000238/aio-dio-append-write-fallocate-race.c v6>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000ac:[<0x10000000238/aio-dio-append-write-fallocate-race.c v6>,<0x10000000236/aio-dio-regress v44>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.374 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.009+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000fa [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/replay-log.c auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d11d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000fa:[<0x1000000023a/replay-log.c v10>,<0x10000000236/log-writes v172>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000fa:[<0x1000000023a/replay-log.c v10>,<0x10000000236/log-writes v172>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.375 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.010+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/Makefile auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d11db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000023b:[<0x1000000023a/Makefile v2>,<0x10000000236/log-writes v172>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000023b:[<0x1000000023a/Makefile v2>,<0x10000000236/log-writes v172>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.375 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.011+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000f9 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/log-writes.h auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d11e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000f9:[<0x1000000023a/log-writes.h v8>,<0x10000000236/log-writes v172>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000f9:[<0x1000000023a/log-writes.h v8>,<0x10000000236/log-writes v172>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.375 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.011+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000f8 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/log-writes.c auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d11e680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000f8:[<0x1000000023a/log-writes.c v6>,<0x10000000236/log-writes v172>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000f8:[<0x1000000023a/log-writes.c v6>,<0x10000000236/log-writes v172>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.375 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:24.012+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000000f7 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/SOURCE auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d11ec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000000f7:[<0x1000000023a/SOURCE v4>,<0x10000000236/log-writes v172>,<0x10000000007/src v770>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x100000000f7:[<0x1000000023a/SOURCE v4>,<0x10000000236/log-writes v172>,<0x10000000007/src v796>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:24.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:23 smithi060 ceph-mon[32745]: scrub queued for path: / 2024-03-04T12:27:24.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:23 smithi060 ceph-mon[32745]: scrub summary: idle+waiting paths [/] 2024-03-04T12:27:24.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:23 smithi060 ceph-mon[32745]: scrub summary: active paths [/] 2024-03-04T12:27:24.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:23 smithi060 ceph-mon[32745]: scrub summary: idle+waiting paths [/] 2024-03-04T12:27:24.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:23 smithi060 ceph-mon[32745]: scrub summary: active paths [/] 2024-03-04T12:27:24.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:23 smithi060 ceph-mon[32745]: scrub summary: idle+waiting paths [/] 2024-03-04T12:27:25.338 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: pgmap v49: 129 pgs: 129 active+clean; 16 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-03-04T12:27:25.338 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: scrub summary: active paths [/] 2024-03-04T12:27:25.338 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000002b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/MAINTAINERS) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.339 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000236 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.339 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000022c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.339 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000230 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.339 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000033 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.selftest) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.339 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000022e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.339 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000002f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.config-sections) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.339 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000002c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.339 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000a5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/new) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.339 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000035 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/acinclude.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.339 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000234 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.339 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000008 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000a6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/randomize.awk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000022a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000002d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/Makepkgs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000090 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lsqa.pl) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000232 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000030 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.device-mapper) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000002e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000008f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/local.config.example) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000031 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.fuse) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000a7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/release.sh) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000006a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configure.ac) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.340 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000a9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/soak) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.341 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000032 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.overlay) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.341 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000034 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/VERSION) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.341 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000021e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/LICENSES) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.341 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000021d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.gitignore) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.341 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.341 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000038 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/check) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.341 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000220 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.341 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000a8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/setup) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.341 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000147 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_1.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.341 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000f1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/holes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.341 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000e3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2fs_check) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.343 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000014b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_rename_overwrite.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.343 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000157 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/usemem.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.343 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000140 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_stale_pmd.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.343 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000c8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/checkpoint_journal.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.343 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000d2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirhash_collide.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.343 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000119 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/randholes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.343 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000e6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fssum.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.343 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000c9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/chprojid_fail.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.343 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000bd (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/alloc.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.343 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000c3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bstat.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000134 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ext4_dax_inline_corruption.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000108 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/nsexec.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000014f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_truncate_self.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000135 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ext4_dax_journal_corruption.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ec (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/getdevicesize.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000337 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ca (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/cloner.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000cf (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-buf-fault.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000014c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_snapshot_deleted_subvolume.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000d5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dmerror) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000101 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/metaperf.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000012d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_create_long_dirs.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.344 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000014a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_reflink_read_race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000d3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirperf.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000e8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fsync-err.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000117 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/punch-alternating.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000155 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/unwritten_sync.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000115 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/preallo_rw_pattern_reader.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000e4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fs_perms.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000148 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000132 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_encrypted_d_revalidate.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000125 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/splice-test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000123 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/seek_sanity_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000145 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ofd_locks.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000122 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/seek_copy_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.345 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000011f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/runas.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000109 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/open_by_handle.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000116 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/preallo_rw_pattern_writer.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000013a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_immutable.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000cb (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dbtest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000104 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmap-write-concurrent.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000e1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2attr) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000e9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fsync-tester.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000dc (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/feature.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000f5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/listxattr.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000013d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_cow_race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000151 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/testx.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000fc (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/looptest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000d8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/enospc_unlink.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000d1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-invalidate-cache.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.346 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000d0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-interleaved.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000d4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirstress.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000bf (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/append_reader.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000012a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/swapon.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000dd (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fiemap-tester.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000e2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2fs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000126 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/splice2pipe.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000c6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_unlink_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000118 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/pwrite_mmap_blocked.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000149 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_3.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000152 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/trunc.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000fe (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/makeextents.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000f4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/itrash.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000127 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/stale_handle.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.347 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ed (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/getpagesize.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000012b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_access_root.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000011d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/resvtest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000133 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_enospc.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000010a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/parse-dev-tree.awk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000146 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_open_tmpfiles.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000011b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/rename.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000db (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fault.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000237 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000141 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_write_ro.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000014d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_stripealign.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000107 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/nametest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000c2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/attr_replace_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.348 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000128 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/stat_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000f2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/holetest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000cd (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/detached_mounts_propagation.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000106 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/multi_open_unlink.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000105 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmapcat.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000011a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/readdir-while-renames.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000112 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/permname.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000f3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/iopat.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000d6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dumpfile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000e0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.349 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000df (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000f6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/locktest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000102 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mkswap.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000cc (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/deduperace.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000143 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_writev_overlap.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000139 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_holes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000131 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_type.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000012f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_offset.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000103 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmap-rw-fault.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000010b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/parse-extent-tree.awk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ff (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/md5.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000238 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000114 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/popdir.pl) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000011c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/renameat2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.350 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000129 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/statx.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000f0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/helpers.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000130 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_offset2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000144 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mtab.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000da (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fake-dump-rootino.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ee (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/global.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000de (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fiexchange.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000154 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/unwritten_mmap.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000023a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000c0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/append_writer.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ea (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ftrunc.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000be (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/allocstale.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000d7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/e4compact.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000124 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/soak_duration.awk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000fd (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/lstat64.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000136 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_futimens.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000113 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/popattr.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000013f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_fallocate.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000012e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_create_short_dirs.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000d9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ext4_resize.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000138 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_getcwd.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000012c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_attr_corruption.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ef (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/godown.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000156 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/uring_read_fault.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000c5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_null_ocount.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000013e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_dio.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000011e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/rewinddir-test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000eb (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/genhashnames.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000013c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_cow_memory_failure.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000158 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/uuid_ioctl.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000fb (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/loggen.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000c4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/btrfs_crc32c_forged_name.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ce (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/devzero.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000e5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fscrypt-crypt-util.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000100 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/md5.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000c1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/attr-list-by-handle-cursor-test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000142 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_writev.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000137 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_get_file_time.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000e7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fstest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000aa (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/af_unix.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000013b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_collision.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000121 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/scaleread.sh) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000014e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_truncate_cmtime.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000120 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/scaleread.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000153 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/truncfile.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000150 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/test-nextquota.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000c7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_unlink_test_modified.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000006d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/auxiliary-programs.txt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000006f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/requirement-checking.txt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000006e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/group-names.txt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000022d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/CHANGES) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000071 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/xunit.xsd) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000070 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/testing-afs.txt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000008c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/string_to_tokens.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000083 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/databin.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000086 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/forker.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000082 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/dataascii.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000008b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/str_to_bytes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000008e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/write_log.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000008d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/tlibio.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000085 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/file_lock.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000008a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/random_range.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000084 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/datapid.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000231 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000087 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/open_flags.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000088 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/pattern.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000089 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/random.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000078 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/file_lock.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000007c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/pattern.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000007f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/string_to_tokens.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000081 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/write_log.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000075 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildrules) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000007a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/install-sh) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000022f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000007e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/str_to_bytes.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000074 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildmacros) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000077 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/databin.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000079 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/forker.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000080 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/tlibio.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000076 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/dataascii.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000007d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/random_range.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.359 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000007b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/open_flags.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.359 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000072 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/builddefs.in) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.359 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000073 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildgrouplist) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.359 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000a1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_liburing.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.359 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000099 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/multilib.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.359 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000009e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libbtrfs.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.359 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000a2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_utilies.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.359 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000009a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_acldev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.359 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000009f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libcap.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.359 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000235 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.359 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000009c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_gdbmdev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.359 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000009b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_aiodev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.360 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000a0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libcdev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.360 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000a4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_xfslibs.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.360 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000a3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_uuiddev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.360 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000009d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_globals.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.360 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000002a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/index.lock) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.360 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000025 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/HEAD) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.360 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000009 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/description) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.360 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000020c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.360 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000205 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.360 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000214 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.360 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000024 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/packed-refs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000029 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/config) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000203 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/info) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000208 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000022 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/shallow) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000207 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/branches) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000006c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/xfscrash) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000006b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/rc.sysinit) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000022b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/README) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000097 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/iogen.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000092 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/doio.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000096 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/growfiles.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000098 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/rwtest.sh) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000233 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.361 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000095 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/fsx.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000094 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/fsstress.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000093 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/doio.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000091 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/aio-stress.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000060 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/repair) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000004e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filter.btrfs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000061 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/report) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000045 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmlogwrites) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000003b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/casefold) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000058 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/perf) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000004b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/fail_make_request) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000042 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmerror) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000062 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/scsi_debug) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.362 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000050 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/gcov) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000044 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmhugedisk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000066 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/verity) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000067 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/xfs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000052 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/locktest) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000004c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filestreams) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000049 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ext4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000048 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/encrypt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000005a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/preamble) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000003e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/config) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000005d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/rc) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000047 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dump) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000003d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/cgroup2) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000068 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/zoned) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000005f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/renameat2) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.363 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.364 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000054 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/metadump) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.364 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000003c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ceph) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.364 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000005b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/punch) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.364 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000056 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/nfs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.364 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000065 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ubifs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.364 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000043 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmflakey) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.364 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000004d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filter) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.364 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000040 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmdelay) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.364 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000055 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/module) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.364 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000003a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/btrfs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.364 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000005e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/reflink) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.364 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000004f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/fuzzy) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000063 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/test_names) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000041 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmdust) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000039 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/attr) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000005c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/quota) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000046 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmthin) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000057 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/overlay) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000053 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/log) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000051 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/inject) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000059 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/populate) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: mgrmap e27: x(active, since 90s), standbys: y, z 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000004a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/f2fs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000064 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/tracing) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000003f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/defrag) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.365 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000021f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/LICENSES/GPL-2.0) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000229 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/example.config) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000069 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/localhost.config) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000224 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/tar) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000221 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000222 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000010e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/ResultData.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000110 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/fio-results.sql) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000111 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/generate-schema.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000010d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000010f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/fio-insert-and-compare.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000338 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/FioCompare.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000010c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/FioResultDecoder.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ae (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-cow-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.366 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000b3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-hole-filling-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000bb (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aiocp.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000bc (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aiodio_sparse2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000b0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-eof-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000239 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000b8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-free-ring-with-bogus-nr-pages.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ba (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-last-ref-held-by-io.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000b6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-subblock-eof-read.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000b2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-fcntl-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000b9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-io-setup-with-nonwritable-context-pointer.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ad (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-append-write-read-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000b7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-write-verify.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000af (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-cycle-write.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000b1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-extend-stat.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.368 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000b4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-invalidate-failure.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.368 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ab (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/SOURCE) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.368 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000b5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-invalidate-readahead.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.368 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000ac (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-append-write-fallocate-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.368 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000fa (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/replay-log.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.368 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000023b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.368 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000f9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/log-writes.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.368 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000f8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/log-writes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.368 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:24 smithi063 ceph-mon[36717]: Scrub error on inode 0x100000000f7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/SOURCE) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.370 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: pgmap v49: 129 pgs: 129 active+clean; 16 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-03-04T12:27:25.370 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: scrub summary: active paths [/] 2024-03-04T12:27:25.370 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000002b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/MAINTAINERS) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.370 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000236 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.370 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000022c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.370 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000230 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.370 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000033 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.selftest) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.370 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000022e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000002f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.config-sections) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000002c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000a5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/new) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000035 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/acinclude.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000234 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000008 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000a6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/randomize.awk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000022a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000002d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/Makepkgs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000090 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lsqa.pl) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000232 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000030 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.device-mapper) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000002e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.371 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000008f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/local.config.example) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000031 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.fuse) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000a7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/release.sh) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000006a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configure.ac) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000a9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/soak) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000032 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.overlay) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000034 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/VERSION) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000021e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/LICENSES) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000021d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.gitignore) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000038 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/check) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000220 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.372 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000a8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/setup) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000147 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_1.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000f1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/holes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000e3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2fs_check) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000014b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_rename_overwrite.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000157 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/usemem.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000140 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_stale_pmd.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000c8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/checkpoint_journal.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000d2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirhash_collide.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000119 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/randholes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000e6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fssum.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000c9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/chprojid_fail.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000bd (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/alloc.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000c3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bstat.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.373 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000134 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ext4_dax_inline_corruption.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000108 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/nsexec.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000014f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_truncate_self.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000135 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ext4_dax_journal_corruption.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ec (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/getdevicesize.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000337 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ca (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/cloner.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000cf (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-buf-fault.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000014c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_snapshot_deleted_subvolume.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000d5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dmerror) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000101 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/metaperf.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000012d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_create_long_dirs.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000014a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_reflink_read_race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000d3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirperf.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000e8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fsync-err.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.374 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000117 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/punch-alternating.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.375 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000155 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/unwritten_sync.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.375 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000115 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/preallo_rw_pattern_reader.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.375 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000e4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fs_perms.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.375 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000148 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.375 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000132 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_encrypted_d_revalidate.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.375 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000125 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/splice-test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.375 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000123 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/seek_sanity_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.375 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000145 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ofd_locks.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.375 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000122 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/seek_copy_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.375 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000011f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/runas.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.375 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000109 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/open_by_handle.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.375 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000116 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/preallo_rw_pattern_writer.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000013a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_immutable.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000cb (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dbtest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000104 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmap-write-concurrent.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000e1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2attr) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000e9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fsync-tester.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000dc (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/feature.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000f5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/listxattr.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000013d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_cow_race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000151 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/testx.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000fc (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/looptest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000d8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/enospc_unlink.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000d1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-invalidate-cache.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.376 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000d0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-interleaved.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000d4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirstress.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000bf (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/append_reader.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000012a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/swapon.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000dd (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fiemap-tester.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000e2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2fs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000126 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/splice2pipe.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000c6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_unlink_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000118 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/pwrite_mmap_blocked.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000149 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_3.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000152 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/trunc.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000fe (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/makeextents.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000f4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/itrash.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000127 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/stale_handle.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ed (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/getpagesize.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.377 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000012b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_access_root.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000011d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/resvtest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000133 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_enospc.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000010a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/parse-dev-tree.awk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000146 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_open_tmpfiles.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000011b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/rename.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000db (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fault.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000237 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000141 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_write_ro.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000014d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_stripealign.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000107 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/nametest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000c2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/attr_replace_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000128 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/stat_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000f2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/holetest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000cd (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/detached_mounts_propagation.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.378 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000106 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/multi_open_unlink.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000105 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmapcat.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000011a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/readdir-while-renames.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000112 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/permname.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000f3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/iopat.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000d6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dumpfile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000e0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000df (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000f6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/locktest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000102 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mkswap.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000cc (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/deduperace.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000143 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_writev_overlap.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000139 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_holes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000131 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_type.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000012f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_offset.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000103 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmap-rw-fault.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000010b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/parse-extent-tree.awk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ff (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/md5.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000238 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000114 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/popdir.pl) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000011c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/renameat2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000129 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/statx.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000f0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/helpers.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000130 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_offset2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000144 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mtab.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000da (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fake-dump-rootino.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ee (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/global.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000de (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fiexchange.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.381 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000154 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/unwritten_mmap.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.381 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000023a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.381 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000c0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/append_writer.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.381 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ea (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ftrunc.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.381 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000be (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/allocstale.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.381 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000d7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/e4compact.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.381 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000124 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/soak_duration.awk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.381 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000fd (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/lstat64.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.381 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000136 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_futimens.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.381 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000113 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/popattr.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.381 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000013f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_fallocate.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.381 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000012e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_create_short_dirs.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000d9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ext4_resize.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000138 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_getcwd.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000012c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_attr_corruption.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ef (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/godown.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000156 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/uring_read_fault.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000c5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_null_ocount.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000013e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_dio.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000011e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/rewinddir-test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000eb (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/genhashnames.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000013c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_cow_memory_failure.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000158 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/uuid_ioctl.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000fb (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/loggen.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000c4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/btrfs_crc32c_forged_name.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.382 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ce (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/devzero.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000e5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fscrypt-crypt-util.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000100 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/md5.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000c1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/attr-list-by-handle-cursor-test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000142 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_writev.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000137 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_get_file_time.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000e7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fstest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000aa (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/af_unix.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000013b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_collision.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000121 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/scaleread.sh) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000014e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_truncate_cmtime.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000120 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/scaleread.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000153 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/truncfile.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000150 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/test-nextquota.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.383 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000c7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_unlink_test_modified.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000006d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/auxiliary-programs.txt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000006f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/requirement-checking.txt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000006e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/group-names.txt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000022d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/CHANGES) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000071 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/xunit.xsd) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000070 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/testing-afs.txt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000008c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/string_to_tokens.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000083 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/databin.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000086 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/forker.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000082 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/dataascii.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000008b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/str_to_bytes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000008e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/write_log.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000008d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/tlibio.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.384 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000085 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/file_lock.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000008a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/random_range.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000084 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/datapid.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000231 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000087 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/open_flags.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000088 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/pattern.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000089 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/random.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000078 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/file_lock.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000007c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/pattern.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000007f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/string_to_tokens.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000081 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/write_log.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000075 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildrules) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000007a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/install-sh) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000022f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000007e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/str_to_bytes.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000074 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildmacros) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.385 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000077 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/databin.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000079 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/forker.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000080 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/tlibio.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000076 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/dataascii.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000007d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/random_range.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000007b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/open_flags.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000072 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/builddefs.in) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000073 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildgrouplist) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000a1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_liburing.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000099 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/multilib.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000009e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libbtrfs.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000a2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_utilies.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000009a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_acldev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000009f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libcap.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000235 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000009c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_gdbmdev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.386 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000009b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_aiodev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000a0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libcdev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000a4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_xfslibs.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000a3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_uuiddev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000009d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_globals.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000002a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/index.lock) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000025 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/HEAD) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000009 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/description) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000020c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000205 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000214 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000024 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/packed-refs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000029 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/config) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000203 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/info) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000208 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000022 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/shallow) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.387 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000207 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/branches) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000006c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/xfscrash) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000006b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/rc.sysinit) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000022b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/README) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000097 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/iogen.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000092 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/doio.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000096 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/growfiles.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000098 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/rwtest.sh) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000233 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000095 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/fsx.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000094 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/fsstress.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000093 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/doio.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000091 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/aio-stress.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000060 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/repair) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.388 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000004e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filter.btrfs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000061 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/report) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000045 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmlogwrites) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000003b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/casefold) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000058 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/perf) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000004b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/fail_make_request) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000042 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmerror) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000062 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/scsi_debug) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000050 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/gcov) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000044 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmhugedisk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000066 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/verity) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000067 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/xfs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000052 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/locktest) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000004c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filestreams) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.389 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000049 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ext4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000048 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/encrypt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000005a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/preamble) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000003e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/config) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000005d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/rc) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000047 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dump) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000003d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/cgroup2) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000068 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/zoned) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000005f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/renameat2) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000054 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/metadump) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000003c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ceph) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000005b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/punch) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000056 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/nfs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.390 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000065 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ubifs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.391 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000043 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmflakey) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.391 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000004d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filter) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.391 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000040 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmdelay) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.391 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000055 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/module) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.391 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000003a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/btrfs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.391 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000005e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/reflink) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.391 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000004f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/fuzzy) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.391 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000063 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/test_names) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.391 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000041 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmdust) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.391 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000039 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/attr) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.391 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000005c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/quota) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000046 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmthin) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000057 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/overlay) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000053 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/log) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000051 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/inject) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000059 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/populate) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: mgrmap e27: x(active, since 90s), standbys: y, z 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000004a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/f2fs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000064 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/tracing) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000003f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/defrag) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000021f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/LICENSES/GPL-2.0) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000229 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/example.config) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000069 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/localhost.config) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.392 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000224 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/tar) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000221 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000222 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000010e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/ResultData.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000110 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/fio-results.sql) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000111 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/generate-schema.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000010d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000010f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/fio-insert-and-compare.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000338 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/FioCompare.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000010c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/FioResultDecoder.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ae (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-cow-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000b3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-hole-filling-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000bb (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aiocp.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000bc (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aiodio_sparse2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.393 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000b0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-eof-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000239 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000b8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-free-ring-with-bogus-nr-pages.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ba (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-last-ref-held-by-io.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000b6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-subblock-eof-read.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000b2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-fcntl-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000b9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-io-setup-with-nonwritable-context-pointer.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ad (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-append-write-read-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000b7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-write-verify.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000af (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-cycle-write.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000b1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-extend-stat.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000b4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-invalidate-failure.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ab (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/SOURCE) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000b5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-invalidate-readahead.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000ac (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-append-write-fallocate-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000fa (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/replay-log.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.394 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000023b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.395 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000f9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/log-writes.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.395 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000f8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/log-writes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.395 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:24 smithi086 ceph-mon[36851]: Scrub error on inode 0x100000000f7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/SOURCE) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: pgmap v49: 129 pgs: 129 active+clean; 16 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 1.6 MiB/s wr, 147 op/s 2024-03-04T12:27:25.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: scrub summary: active paths [/] 2024-03-04T12:27:25.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000002b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/MAINTAINERS) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000236 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000022c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000230 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000033 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.selftest) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000022e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000002f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.config-sections) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000002c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000a5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/new) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000035 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/acinclude.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000234 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000008 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000a6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/randomize.awk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000022a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000002d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/Makepkgs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000090 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lsqa.pl) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000232 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000030 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.device-mapper) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000002e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000008f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/local.config.example) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000031 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.fuse) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000a7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/release.sh) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000006a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configure.ac) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000a9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/soak) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000032 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.overlay) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000034 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/VERSION) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000021e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/LICENSES) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000021d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.gitignore) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000038 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/check) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000220 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000a8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/setup) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000147 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_1.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000f1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/holes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000e3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2fs_check) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000014b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_rename_overwrite.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000157 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/usemem.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000140 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_stale_pmd.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000c8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/checkpoint_journal.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000d2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirhash_collide.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000119 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/randholes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000e6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fssum.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000c9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/chprojid_fail.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000bd (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/alloc.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000c3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bstat.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.433 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000134 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ext4_dax_inline_corruption.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000108 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/nsexec.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000014f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_truncate_self.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000135 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ext4_dax_journal_corruption.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ec (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/getdevicesize.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000337 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ca (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/cloner.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000cf (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-buf-fault.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000014c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_snapshot_deleted_subvolume.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000d5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dmerror) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000101 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/metaperf.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000012d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_create_long_dirs.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000014a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_reflink_read_race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000d3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirperf.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.434 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000e8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fsync-err.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000117 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/punch-alternating.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000155 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/unwritten_sync.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000115 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/preallo_rw_pattern_reader.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000e4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fs_perms.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000148 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000132 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_encrypted_d_revalidate.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000125 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/splice-test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000123 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/seek_sanity_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000145 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ofd_locks.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000122 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/seek_copy_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000011f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/runas.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000109 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/open_by_handle.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000116 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/preallo_rw_pattern_writer.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.435 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000013a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_immutable.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000cb (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dbtest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000104 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmap-write-concurrent.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000e1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2attr) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000e9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fsync-tester.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000dc (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/feature.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000f5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/listxattr.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000013d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_cow_race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000151 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/testx.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000fc (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/looptest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000d8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/enospc_unlink.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000d1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-invalidate-cache.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000d0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-interleaved.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.436 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000d4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirstress.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.437 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000bf (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/append_reader.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.439 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000012a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/swapon.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.439 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000dd (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fiemap-tester.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.439 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000e2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2fs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.439 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000126 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/splice2pipe.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.439 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000c6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_unlink_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.439 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000118 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/pwrite_mmap_blocked.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.439 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000149 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_3.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.439 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000152 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/trunc.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.439 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000fe (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/makeextents.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.439 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000f4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/itrash.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.439 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000127 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/stale_handle.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ed (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/getpagesize.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000012b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_access_root.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000011d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/resvtest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000133 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_enospc.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000010a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/parse-dev-tree.awk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000146 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_open_tmpfiles.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000011b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/rename.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000db (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fault.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000237 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000141 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_write_ro.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000014d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_stripealign.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000107 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/nametest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.440 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000c2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/attr_replace_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000128 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/stat_test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000f2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/holetest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000cd (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/detached_mounts_propagation.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000106 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/multi_open_unlink.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000105 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmapcat.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000011a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/readdir-while-renames.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000112 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/permname.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000f3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/iopat.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000d6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dumpfile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000e0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000df (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000f6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/locktest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000102 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mkswap.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.441 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000cc (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/deduperace.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000143 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_writev_overlap.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000139 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_holes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000131 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_type.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000012f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_offset.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000103 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmap-rw-fault.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000010b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/parse-extent-tree.awk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ff (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/md5.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000238 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000114 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/popdir.pl) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000011c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/renameat2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000129 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/statx.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000f0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/helpers.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.443 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000130 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_offset2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.443 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000144 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mtab.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.443 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000da (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fake-dump-rootino.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.443 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ee (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/global.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.443 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000de (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fiexchange.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.443 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000154 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/unwritten_mmap.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.443 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000023a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.443 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000c0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/append_writer.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.443 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ea (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ftrunc.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.443 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000be (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/allocstale.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000d7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/e4compact.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000124 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/soak_duration.awk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000fd (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/lstat64.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000136 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_futimens.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000113 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/popattr.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000013f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_fallocate.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000012e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_create_short_dirs.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000d9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ext4_resize.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000138 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_getcwd.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000012c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_attr_corruption.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ef (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/godown.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000156 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/uring_read_fault.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000c5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_null_ocount.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.444 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000013e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_dio.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000011e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/rewinddir-test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000eb (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/genhashnames.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000013c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_cow_memory_failure.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000158 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/uuid_ioctl.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000fb (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/loggen.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000c4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/btrfs_crc32c_forged_name.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ce (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/devzero.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000e5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fscrypt-crypt-util.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000100 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/md5.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000c1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/attr-list-by-handle-cursor-test.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000142 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_writev.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000137 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_get_file_time.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000e7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fstest.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.445 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000aa (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/af_unix.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000013b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_collision.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000121 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/scaleread.sh) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000014e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_truncate_cmtime.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000120 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/scaleread.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000153 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/truncfile.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000150 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/test-nextquota.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000c7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_unlink_test_modified.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000006d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/auxiliary-programs.txt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000006f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/requirement-checking.txt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000006e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/group-names.txt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000022d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/CHANGES) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000071 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/xunit.xsd) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000070 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/testing-afs.txt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000008c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/string_to_tokens.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.446 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000083 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/databin.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000086 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/forker.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000082 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/dataascii.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000008b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/str_to_bytes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000008e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/write_log.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000008d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/tlibio.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000085 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/file_lock.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000008a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/random_range.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000084 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/datapid.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000231 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000087 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/open_flags.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000088 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/pattern.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000089 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/random.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.447 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000078 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/file_lock.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000007c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/pattern.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000007f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/string_to_tokens.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000081 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/write_log.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000075 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildrules) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000007a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/install-sh) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000022f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000007e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/str_to_bytes.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000074 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildmacros) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000077 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/databin.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000079 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/forker.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000080 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/tlibio.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000076 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/dataascii.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000007d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/random_range.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.448 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000007b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/open_flags.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000072 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/builddefs.in) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000073 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildgrouplist) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000a1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_liburing.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000099 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/multilib.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000009e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libbtrfs.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000a2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_utilies.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000009a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_acldev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000009f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libcap.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000235 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000009c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_gdbmdev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000009b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_aiodev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000a0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libcdev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000a4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_xfslibs.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000a3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_uuiddev.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.449 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000009d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_globals.m4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000002a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/index.lock) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000025 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/HEAD) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000009 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/description) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000020c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000205 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000214 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000024 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/packed-refs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000029 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/config) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000203 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/info) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000208 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000022 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/shallow) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000207 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/branches) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000006c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/xfscrash) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.450 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000006b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/rc.sysinit) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000022b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/README) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000097 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/iogen.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000092 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/doio.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000096 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/growfiles.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000098 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/rwtest.sh) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000233 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000095 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/fsx.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000094 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/fsstress.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000093 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/doio.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000091 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/aio-stress.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000060 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/repair) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000004e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filter.btrfs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.451 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000061 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/report) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000045 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmlogwrites) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000003b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/casefold) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000058 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/perf) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000004b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/fail_make_request) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000042 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmerror) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000062 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/scsi_debug) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000050 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/gcov) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000044 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmhugedisk) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000066 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/verity) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000067 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/xfs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000052 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/locktest) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000004c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filestreams) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.452 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000049 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ext4) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.453 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000048 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/encrypt) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.453 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000005a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/preamble) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.453 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000003e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/config) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.453 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000005d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/rc) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.453 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000047 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dump) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.453 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000003d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/cgroup2) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.453 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000068 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/zoned) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.453 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000005f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/renameat2) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.453 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.453 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000054 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/metadump) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.453 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000003c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ceph) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000005b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/punch) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000056 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/nfs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000065 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ubifs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000043 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmflakey) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000004d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filter) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000040 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmdelay) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000055 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/module) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000003a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/btrfs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000005e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/reflink) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000004f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/fuzzy) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000063 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/test_names) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000041 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmdust) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000039 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/attr) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.454 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000005c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/quota) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000046 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmthin) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000057 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/overlay) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000053 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/log) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000051 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/inject) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000059 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/populate) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: mgrmap e27: x(active, since 90s), standbys: y, z 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000004a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/f2fs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000064 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/tracing) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000003f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/defrag) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000021f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/LICENSES/GPL-2.0) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000229 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/example.config) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000069 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/localhost.config) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.455 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000224 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/tar) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000221 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000222 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000010e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/ResultData.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000110 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/fio-results.sql) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000111 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/generate-schema.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000010d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000010f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/fio-insert-and-compare.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000338 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/FioCompare.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000010c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/FioResultDecoder.py) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ae (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-cow-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000b3 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-hole-filling-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000bb (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aiocp.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.456 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000bc (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aiodio_sparse2.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000b0 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-eof-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000239 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000b8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-free-ring-with-bogus-nr-pages.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ba (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-last-ref-held-by-io.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000b6 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-subblock-eof-read.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000b2 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-fcntl-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000b9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-io-setup-with-nonwritable-context-pointer.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ad (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-append-write-read-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000b7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-write-verify.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000af (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-cycle-write.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000b1 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-extend-stat.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000b4 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-invalidate-failure.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.457 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ab (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/SOURCE) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.458 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000b5 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-invalidate-readahead.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.458 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000ac (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-append-write-fallocate-race.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.458 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000fa (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/replay-log.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.458 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000023b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.458 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000f9 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/log-writes.h) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.458 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000f8 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/log-writes.c) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.458 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:24 smithi060 ceph-mon[32745]: Scrub error on inode 0x100000000f7 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/SOURCE) see mds.h log and `damage ls` output for details 2024-03-04T12:27:25.605 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.330+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020e [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/info/ auth v4 ap=1 f() n(v0 1=0+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFs@2},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d476100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x1000000020e:[<0x1000000020c/info v4>,<0x10000000008/objects v136>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x1000000020e:[<0x1000000020c/info v4>,<0x10000000008/objects v209>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 1=0+1)","error_str":""},"return_code":0} 2024-03-04T12:27:25.605 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.331+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020d [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/ auth{0=1} v44 ap=1 f(v0 m2024-03-04T12:27:13.751907+0000 2=2+0) n(v0 rc2024-03-04T12:27:13.751907+0000 b13162810 3=2+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5d476680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x1000000020d:[<0x1000000020c/pack v44>,<0x10000000008/objects v136>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x1000000020d:[<0x1000000020c/pack v44>,<0x10000000008/objects v209>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.751907+0000 2=2+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.751907+0000 b13162810 3=2+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.751907+0000 2=2+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.751907+0000 b13162810 3=2+1)","error_str":""},"return_code":0} 2024-03-04T12:27:25.605 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.332+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000015 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-merge-commit.sample auth v26 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d477700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000015:[<0x10000000205/pre-merge-commit.sample v26>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000015:[<0x10000000205/pre-merge-commit.sample v26>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.605 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.333+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000206 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-commit.sample auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4bc000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000206:[<0x10000000205/pre-commit.sample v2>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000206:[<0x10000000205/pre-commit.sample v2>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.605 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.333+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000000c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/fsmonitor-watchman.sample auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4bc580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000000c:[<0x10000000205/fsmonitor-watchman.sample v8>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000000c:[<0x10000000205/fsmonitor-watchman.sample v8>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.605 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.334+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000013 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/commit-msg.sample auth v22 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4bcb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000013:[<0x10000000205/commit-msg.sample v22>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000013:[<0x10000000205/commit-msg.sample v22>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.605 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.335+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000000d [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-rebase.sample auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4bd080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000000d:[<0x10000000205/pre-rebase.sample v10>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000000d:[<0x10000000205/pre-rebase.sample v10>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.605 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.336+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000000f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-receive.sample auth v14 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4bd600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000000f:[<0x10000000205/pre-receive.sample v14>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000000f:[<0x10000000205/pre-receive.sample v14>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.605 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.336+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000000a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/prepare-commit-msg.sample auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4bdb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000000a:[<0x10000000205/prepare-commit-msg.sample v4>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000000a:[<0x10000000205/prepare-commit-msg.sample v4>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.606 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.337+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000011 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/update.sample auth v18 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4be100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000011:[<0x10000000205/update.sample v18>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000011:[<0x10000000205/update.sample v18>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.606 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.338+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000012 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-push.sample auth v20 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4be680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000012:[<0x10000000205/pre-push.sample v20>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000012:[<0x10000000205/pre-push.sample v20>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.606 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.338+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000014 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-applypatch.sample auth v24 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4bec00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000014:[<0x10000000205/pre-applypatch.sample v24>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000014:[<0x10000000205/pre-applypatch.sample v24>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.606 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.339+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000000b [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/push-to-checkout.sample auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4bf180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000000b:[<0x10000000205/push-to-checkout.sample v6>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000000b:[<0x10000000205/push-to-checkout.sample v6>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.606 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.340+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000010 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/post-update.sample auth v16 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4bf700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000010:[<0x10000000205/post-update.sample v16>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000010:[<0x10000000205/post-update.sample v16>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.606 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.341+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000000e [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/applypatch-msg.sample auth v12 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4ce000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000000e:[<0x10000000205/applypatch-msg.sample v12>,<0x10000000008/hooks v34>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000000e:[<0x10000000205/applypatch-msg.sample v12>,<0x10000000008/hooks v34>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.606 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.341+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021a [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/HEAD auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4ce580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021a:[<0x10000000214/HEAD v6>,<0x10000000008/logs v175>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000021a:[<0x10000000214/HEAD v6>,<0x10000000008/logs v215>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.606 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.342+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/ auth{0=1} v15 ap=1 f(v0 m2024-03-04T12:27:13.718907+0000 2=0+2) n(v1 rc2024-03-04T12:27:13.720907+0000 6=2+4) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x564e5d4ceb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000214/refs v8>,<0x10000000008/logs v175>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000214/refs v15>,<0x10000000008/logs v215>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.718907+0000 2=0+2)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.720907+0000 6=2+4)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.718907+0000 2=0+2)","memory_value.rstat":"n(v1 rc2024-03-04T12:27:13.720907+0000 6=2+4)","error_str":""},"return_code":0} 2024-03-04T12:27:25.606 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.343+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000204 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/info/exclude auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4d0c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000204:[<0x10000000203/exclude v2>,<0x10000000008/info v6>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000204:[<0x10000000203/exclude v2>,<0x10000000008/info v6>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.606 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.344+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020a [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/tags/ auth v4 ap=1 f() n(v0 1=0+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@2},l=25139 | importingcaps=0 dirfrag=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4d1180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x1000000020a:[<0x10000000208/tags v4>,<0x10000000008/refs v160>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x1000000020a:[<0x10000000208/tags v4>,<0x10000000008/refs v213>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 1=0+1)","error_str":""},"return_code":0} 2024-03-04T12:27:25.606 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.344+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000209 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/heads/ auth{0=1} v14 ap=1 f(v0 m2024-03-04T12:27:13.721907+0000 1=1+0) n(v0 rc2024-03-04T12:27:13.721907+0000 2=1+1) (inest lock dirty) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | dirtyscattered=1 importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5d4d1700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000209:[<0x10000000208/heads v14>,<0x10000000008/refs v160>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000209:[<0x10000000208/heads v14>,<0x10000000008/refs v213>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.721907+0000 1=1+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.721907+0000 2=1+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.721907+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.721907+0000 2=1+1)","error_str":""},"return_code":0} 2024-03-04T12:27:25.606 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.345+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000211 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/ auth{0=1} v16 ap=1 f(v0 m2024-03-04T12:27:13.704908+0000 1=0+1) n(v1 rc2024-03-04T12:27:13.711907+0000 3=1+2) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x564e5d4dc000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000211:[<0x10000000208/remotes v8>,<0x10000000008/refs v160>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000211:[<0x10000000208/remotes v16>,<0x10000000008/refs v213>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.704908+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.711907+0000 3=1+2)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.704908+0000 1=0+1)","memory_value.rstat":"n(v1 rc2024-03-04T12:27:13.711907+0000 3=1+2)","error_str":""},"return_code":0} 2024-03-04T12:27:25.607 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.346+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000225 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/tar/Makefile auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d531b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000225:[<0x10000000224/Makefile v2>,<0x10000000220/tar v14>,<0x10000000007/build v206>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000225:[<0x10000000224/Makefile v2>,<0x10000000220/tar v14>,<0x10000000007/build v778>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.607 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.346+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000223 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/Makefile auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d532100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000223:[<0x10000000222/Makefile v2>,<0x10000000220/rpm v10>,<0x10000000007/build v206>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000223:[<0x10000000222/Makefile v2>,<0x10000000220/rpm v10>,<0x10000000007/build v778>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.607 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.347+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000036 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/macros.template auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d532680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000036:[<0x10000000222/macros.template v4>,<0x10000000220/rpm v10>,<0x10000000007/build v206>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000036:[<0x10000000222/macros.template v4>,<0x10000000220/rpm v10>,<0x10000000007/build v778>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.607 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.348+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000037 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/xfstests.spec.in auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d532c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000037:[<0x10000000222/xfstests.spec.in v6>,<0x10000000220/rpm v10>,<0x10000000007/build v206>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000037:[<0x10000000222/xfstests.spec.in v6>,<0x10000000220/rpm v10>,<0x10000000007/build v778>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.607 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.349+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000210 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.idx auth v130 ap=1 s=1305816 n(v0 rc2024-03-04T12:27:13.669908+0000 b1305816 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d476c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000210:[<0x1000000020d/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.idx v130>,<0x1000000020c/pack v44>,<0x10000000008/objects v136>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000210:[<0x1000000020d/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.idx v130>,<0x1000000020c/pack v44>,<0x10000000008/objects v209>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.607 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.349+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020f [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.pack auth{0=1} v127 ap=1 s=11856994 n(v0 rc2024-03-04T12:27:13.665908+0000 b11856994 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-25165824@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25139 | importingcaps=0 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5d477180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000020f:[<0x1000000020d/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.pack v127>,<0x1000000020c/pack v44>,<0x10000000008/objects v136>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000020f:[<0x1000000020d/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.pack v127>,<0x1000000020c/pack v44>,<0x10000000008/objects v209>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.607 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.350+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021b [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/heads/ auth{0=1} v8 ap=1 f(v0 m2024-03-04T12:27:13.720907+0000 1=1+0) n(v0 rc2024-03-04T12:27:13.720907+0000 2=1+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5d4cf080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x1000000021b:[<0x10000000215/heads v8>,<0x10000000214/refs v8>,<0x10000000008/logs v175>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x1000000021b:[<0x10000000215/heads v8>,<0x10000000214/refs v15>,<0x10000000008/logs v215>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.720907+0000 1=1+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.720907+0000 2=1+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.720907+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.720907+0000 2=1+1)","error_str":""},"return_code":0} 2024-03-04T12:27:25.607 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.351+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000216 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/ auth{0=1} v10 ap=1 f(v0 m2024-03-04T12:27:13.709907+0000 1=0+1) n(v1 rc2024-03-04T12:27:13.710907+0000 3=1+2) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x564e5d4cf600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000216:[<0x10000000215/remotes v4>,<0x10000000214/refs v8>,<0x10000000008/logs v175>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000216:[<0x10000000215/remotes v10>,<0x10000000214/refs v15>,<0x10000000008/logs v215>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.709907+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.710907+0000 3=1+2)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.709907+0000 1=0+1)","memory_value.rstat":"n(v1 rc2024-03-04T12:27:13.710907+0000 3=1+2)","error_str":""},"return_code":0} 2024-03-04T12:27:25.607 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.352+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000219 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/heads/master auth v4 ap=1 s=0 n(v0 rc2024-03-04T12:27:13.721907+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25139 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4dc580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000219:[<0x10000000209/master v4>,<0x10000000208/heads v14>,<0x10000000008/refs v160>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000219:[<0x10000000209/master v4>,<0x10000000208/heads v14>,<0x10000000008/refs v213>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.607 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.352+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000212 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/origin/ auth{0=1} v8 ap=1 f(v0 m2024-03-04T12:27:13.711907+0000 1=1+0) n(v0 rc2024-03-04T12:27:13.711907+0000 2=1+1) (inest lock dirty) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | dirtyscattered=1 importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5d4dcb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000212:[<0x10000000211/origin v8>,<0x10000000208/remotes v8>,<0x10000000008/refs v160>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000212:[<0x10000000211/origin v8>,<0x10000000208/remotes v16>,<0x10000000008/refs v213>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.711907+0000 1=1+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.711907+0000 2=1+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.711907+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.711907+0000 2=1+1)","error_str":""},"return_code":0} 2024-03-04T12:27:25.607 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.353+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021c [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/heads/master auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4cfb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021c:[<0x1000000021b/master v2>,<0x10000000215/heads v8>,<0x10000000214/refs v8>,<0x10000000008/logs v175>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x1000000021c:[<0x1000000021b/master v2>,<0x10000000215/heads v8>,<0x10000000214/refs v15>,<0x10000000008/logs v215>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.608 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.354+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000217 [...4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/origin/ auth{0=1} v4 ap=1 f(v0 m2024-03-04T12:27:13.710907+0000 1=1+0) n(v0 rc2024-03-04T12:27:13.710907+0000 2=1+1) (inest lock) (ifile excl) (iversion lock) caps={25139=pAsLsXsFsx/pAsLsXsFsxcral@1},l=25139 | importingcaps=0 dirfrag=1 caps=1 replicated=1 authpin=1 scrubqueue=0 0x564e5d4d0100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000217:[<0x10000000216/origin v4>,<0x10000000215/remotes v4>,<0x10000000214/refs v8>,<0x10000000008/logs v175>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[]","memoryvalue":"(2)0x10000000217:[<0x10000000216/origin v4>,<0x10000000215/remotes v10>,<0x10000000214/refs v15>,<0x10000000008/logs v215>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-03-04T12:27:13.710907+0000 1=1+0)","ondisk_value.rstat":"n(v0 rc2024-03-04T12:27:13.710907+0000 2=1+1)","memory_value.dirstat":"f(v0 m2024-03-04T12:27:13.710907+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-03-04T12:27:13.710907+0000 2=1+1)","error_str":""},"return_code":0} 2024-03-04T12:27:25.608 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.354+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000213 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/origin/HEAD auth v4 ap=1 s=0 n(v0 rc2024-03-04T12:27:13.711907+0000 1=1+0)/n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25139 | importingcaps=0 caps=1 dirtyrstat=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4dd080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000213:[<0x10000000212/HEAD v4>,<0x10000000211/origin v8>,<0x10000000208/remotes v8>,<0x10000000008/refs v160>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000213:[<0x10000000212/HEAD v4>,<0x10000000211/origin v8>,<0x10000000208/remotes v16>,<0x10000000008/refs v213>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:25.608 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:27:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:27:25.355+0000 7feb7d62f700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000218 [4,head] /volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/origin/HEAD auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25139=0-4194304@3} caps={25139=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@1},l=25139 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x564e5d4d0680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000218:[<0x10000000217/HEAD v2>,<0x10000000216/origin v4>,<0x10000000215/remotes v4>,<0x10000000214/refs v8>,<0x10000000008/logs v175>,<0x10000000007/.git v170>,<0x10000000006/xfstests-dev v70>,<0x10000000005/tmp v6>,<0x100000001fb/client.0 v10>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v26>,<0x10000000001/sv_1 v50>,<0x10000000000/qa v40>,<0x1/volumes v35>]//[3]","memoryvalue":"(4)0x10000000218:[<0x10000000217/HEAD v2>,<0x10000000216/origin v4>,<0x10000000215/remotes v10>,<0x10000000214/refs v15>,<0x10000000008/logs v215>,<0x10000000007/.git v786>,<0x10000000006/xfstests-dev v92>,<0x10000000005/tmp v20>,<0x100000001fb/client.0 v20>,<0x100000001fa/5936704d-a11e-4f77-91f6-4c6a307f08e1 v35>,<0x10000000001/sv_1 v56>,<0x10000000000/qa v29>,<0x1/volumes v29>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-04T12:27:26.138 INFO:tasks.workunit.client.0.smithi060.stderr:Updating files: 7% (321/4261) Updating files: 8% (341/4261) Updating files: 9% (384/4261) Updating files: 10% (427/4261) Updating files: 11% (469/4261) Updating files: 12% (512/4261) Updating files: 13% (554/4261) Updating files: 14% (597/4261) Updating files: 15% (640/4261) Updating files: 16% (682/4261) Updating files: 17% (725/4261) Updating files: 17% (741/4261) Updating files: 18% (767/4261) Updating files: 19% (810/4261) Updating files: 20% (853/4261) Updating files: 21% (895/4261) Updating files: 22% (938/4261) Updating files: 23% (981/4261) Updating files: 24% (1023/4261) Updating files: 25% (1066/4261) Updating files: 26% (1108/4261) Updating files: 26% (1111/4261) Updating files: 27% (1151/4261) Updating files: 28% (1194/4261) Updating files: 29% (1236/4261) Updating files: 30% (1279/4261) Updating files: 31% (1321/4261) Updating files: 32% (1364/4261) Updating files: 33% (1407/4261) Updating files: 34% (1449/4261) Updating files: 35% (1492/4261) Updating files: 36% (1534/4261) Updating files: 37% (1577/4261) Updating files: 37% (1615/4261) Updating files: 38% (1620/4261) Updating files: 39% (1662/4261) Updating files: 40% (1705/4261) Updating files: 41% (1748/4261) Updating files: 42% (1790/4261) Updating files: 43% (1833/4261) Updating files: 44% (1875/4261) Updating files: 45% (1918/4261) Updating files: 46% (1961/4261) Updating files: 47% (2003/4261) Updating files: 48% (2046/4261) Updating files: 49% (2088/4261) Updating files: 49% (2118/4261) Updating files: 50% (2131/4261) Updating files: 51% (2174/4261) Updating files: 52% (2216/4261) Updating files: 53% (2259/4261) Updating files: 54% (2301/4261) Updating files: 55% (2344/4261) Updating files: 56% (2387/4261) Updating files: 57% (2429/4261) Updating files: 58% (2472/4261) Updating files: 59% (2514/4261) Updating files: 60% (2557/4261) Updating files: 61% (2600/4261) Updating files: 62% (2642/4261) Updating files: 63% (2685/4261) Updating files: 64% (2728/4261) Updating files: 65% (2770/4261) Updating files: 66% (2813/4261) Updating files: 67% (2855/4261) Updating files: 68% (2898/4261) Updating files: 69% (2941/4261) Updating files: 70% (2983/4261) Updating files: 71% (3026/4261) Updating files: 72% (3068/4261) Updating files: 73% (3111/4261) Updating files: 73% (3132/4261) Updating files: 74% (3154/4261) Updating files: 75% (3196/4261) Updating files: 76% (3239/4261) Updating files: 77% (3281/4261) Updating files: 78% (3324/4261) Updating files: 79% (3367/4261) Updating files: 80% (3409/4261) Updating files: 81% (3452/4261) Updating files: 82% (3495/4261) Updating files: 83% (3537/4261) Updating files: 84% (3580/4261) Updating files: 85% (3622/4261) Updating files: 86% (3665/4261) Updating files: 87% (3708/4261) Updating files: 88% (3750/4261) Updating files: 89% (3793/4261) Updating files: 90% (3835/4261) Updating files: 91% (3878/4261) Updating files: 92% (3921/4261) Updating files: 93% (3963/4261) Updating files: 94% (4006/4261) Updating files: 95% (4048/4261) Updating files: 96% (4091/4261) Updating files: 97% (4134/4261) Updating files: 98% (4176/4261) Updating files: 99% (4219/4261) Updating files: 99% (4249/4261) Updating files: 100% (4261/4261) Updating files: 100% (4261/4261), done. 2024-03-04T12:27:26.183 INFO:tasks.workunit.client.0.smithi060.stderr:+ cd xfstests-dev 2024-03-04T12:27:26.189 INFO:tasks.workunit.client.0.smithi060.stderr:+ git checkout 12973fc04fd10d4af086901e10ffa8e48866b735 2024-03-04T12:27:27.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: pgmap v50: 129 pgs: 129 active+clean; 20 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-03-04T12:27:27.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000020e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/info) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000020d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000015 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-merge-commit.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000206 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-commit.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000000c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/fsmonitor-watchman.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000013 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/commit-msg.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000000d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-rebase.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-receive.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000000a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/prepare-commit-msg.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000011 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/update.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000012 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-push.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000014 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-applypatch.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000000b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/push-to-checkout.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000010 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/post-update.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000000e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/applypatch-msg.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000021a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/HEAD) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000204 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/info/exclude) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000020a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/tags) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000209 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/heads) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000211 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000225 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/tar/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000223 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000036 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/macros.template) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000037 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/xfstests.spec.in) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000210 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.idx) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.pack) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000021b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/heads) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000216 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000219 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/heads/master) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000212 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/origin) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x1000000021c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/heads/master) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000217 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/origin) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000213 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/origin/HEAD) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: Scrub error on inode 0x10000000218 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/origin/HEAD) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: scrub summary: idle+waiting paths [/] 2024-03-04T12:27:27.353 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:26 smithi086 ceph-mon[36851]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:27.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: pgmap v50: 129 pgs: 129 active+clean; 20 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-03-04T12:27:27.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000020e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/info) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000020d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000015 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-merge-commit.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000206 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-commit.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000000c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/fsmonitor-watchman.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000013 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/commit-msg.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000000d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-rebase.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-receive.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000000a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/prepare-commit-msg.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000011 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/update.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000012 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-push.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000014 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-applypatch.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000000b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/push-to-checkout.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000010 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/post-update.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000000e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/applypatch-msg.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000021a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/HEAD) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000204 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/info/exclude) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000020a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/tags) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000209 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/heads) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000211 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000225 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/tar/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000223 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000036 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/macros.template) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000037 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/xfstests.spec.in) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000210 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.idx) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.pack) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000021b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/heads) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000216 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000219 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/heads/master) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000212 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/origin) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x1000000021c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/heads/master) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000217 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/origin) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000213 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/origin/HEAD) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: Scrub error on inode 0x10000000218 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/origin/HEAD) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.356 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: scrub summary: idle+waiting paths [/] 2024-03-04T12:27:27.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:26 smithi063 ceph-mon[36717]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:27.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: pgmap v50: 129 pgs: 129 active+clean; 20 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-03-04T12:27:27.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000020e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/info) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000020d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000015 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-merge-commit.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000206 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-commit.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000000c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/fsmonitor-watchman.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000013 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/commit-msg.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000000d (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-rebase.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000000f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-receive.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000000a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/prepare-commit-msg.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000011 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/update.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000012 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-push.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000014 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-applypatch.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000000b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/push-to-checkout.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000010 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/post-update.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000000e (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/applypatch-msg.sample) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000021a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/HEAD) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000204 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/info/exclude) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000020a (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/tags) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000209 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/heads) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000211 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000225 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/tar/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000223 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/Makefile) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000036 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/macros.template) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000037 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/xfstests.spec.in) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000210 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.idx) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000020f (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.pack) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.431 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000021b (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/heads) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000216 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000219 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/heads/master) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000212 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/origin) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x1000000021c (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/heads/master) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000217 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/origin) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000213 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/origin/HEAD) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: Scrub error on inode 0x10000000218 (/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/origin/HEAD) see mds.h log and `damage ls` output for details 2024-03-04T12:27:27.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: scrub summary: idle+waiting paths [/] 2024-03-04T12:27:27.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:26 smithi060 ceph-mon[32745]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.131+0000 7f0851a61640 1 -- 172.21.15.60:0/1695927324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f084c101f00 msgr2=0x7f084c105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.131+0000 7f0851a61640 1 --2- 172.21.15.60:0/1695927324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f084c101f00 0x7f084c105f40 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f0834009900 tx=0x7f083402bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.131+0000 7f0851a61640 1 -- 172.21.15.60:0/1695927324 shutdown_connections 2024-03-04T12:27:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.131+0000 7f0851a61640 1 --2- 172.21.15.60:0/1695927324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f084c106620 0x7f084c10e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.131+0000 7f0851a61640 1 --2- 172.21.15.60:0/1695927324 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f084c101f00 0x7f084c105f40 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.140 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.131+0000 7f0851a61640 1 --2- 172.21.15.60:0/1695927324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f084c101550 0x7f084c101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.131+0000 7f0851a61640 1 -- 172.21.15.60:0/1695927324 >> 172.21.15.60:0/1695927324 conn(0x7f084c0fcdb0 msgr2=0x7f084c0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.131+0000 7f0851a61640 1 -- 172.21.15.60:0/1695927324 shutdown_connections 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.131+0000 7f0851a61640 1 -- 172.21.15.60:0/1695927324 wait complete. 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.131+0000 7f0851a61640 1 Processor -- start 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0851a61640 1 -- start start 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0851a61640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f084c101550 0x7f084c073630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0851a61640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f084c106620 0x7f084c073b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0851a61640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f084c0740b0 0x7f084c1a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0851a61640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c111690 con 0x7f084c101550 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0851a61640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c111510 con 0x7f084c106620 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0851a61640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c111810 con 0x7f084c0740b0 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f084affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f084c101550 0x7f084c073630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f084affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f084c101550 0x7f084c073630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51484/0 (socket says 172.21.15.60:51484) 2024-03-04T12:27:28.141 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0843fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f084c106620 0x7f084c073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f084affd640 1 -- 172.21.15.60:0/1200808147 learned_addr learned my addr 172.21.15.60:0/1200808147 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f084b7fe640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f084c0740b0 0x7f084c1a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0843fff640 1 -- 172.21.15.60:0/1200808147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f084c0740b0 msgr2=0x7f084c1a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0843fff640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f084c0740b0 0x7f084c1a5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0843fff640 1 -- 172.21.15.60:0/1200808147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f084c101550 msgr2=0x7f084c073630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0843fff640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f084c101550 0x7f084c073630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0843fff640 1 -- 172.21.15.60:0/1200808147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c1a64e0 con 0x7f084c106620 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0843fff640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f084c106620 0x7f084c073b70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0834036040 tx=0x7f083402cb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.132+0000 7f0850a5f640 1 -- 172.21.15.60:0/1200808147 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834041070 con 0x7f084c106620 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.133+0000 7f0851a61640 1 -- 172.21.15.60:0/1200808147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084c1a6770 con 0x7f084c106620 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.133+0000 7f0850a5f640 1 -- 172.21.15.60:0/1200808147 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0834004450 con 0x7f084c106620 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.133+0000 7f0850a5f640 1 -- 172.21.15.60:0/1200808147 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834003d90 con 0x7f084c106620 2024-03-04T12:27:28.142 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.133+0000 7f0851a61640 1 -- 172.21.15.60:0/1200808147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084c1a6c60 con 0x7f084c106620 2024-03-04T12:27:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.135+0000 7f0850a5f640 1 -- 172.21.15.60:0/1200808147 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f0834044070 con 0x7f084c106620 2024-03-04T12:27:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.135+0000 7f085463e740 1 Processor -- start 2024-03-04T12:27:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.135+0000 7f085463e740 1 -- start start 2024-03-04T12:27:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.135+0000 7f085463e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18bfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.135+0000 7f085463e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.135+0000 7f085463e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.135+0000 7f085463e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff18245f0 con 0x563ff18c10d0 2024-03-04T12:27:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.135+0000 7f085463e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff17cd930 con 0x563ff18c98e0 2024-03-04T12:27:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.135+0000 7f085463e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff1820690 con 0x563ff18c0360 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18bfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18bfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51494/0 (socket says 172.21.15.60:51494) 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084b7fe640 1 -- 172.21.15.60:0/3979291363 learned_addr learned my addr 172.21.15.60:0/3979291363 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084affd640 1 --2- 172.21.15.60:0/3979291363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f0843fff640 1 --2- 172.21.15.60:0/3979291363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f0850a5f640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f08380a1f10 0x7f08380a43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084b7fe640 1 -- 172.21.15.60:0/3979291363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 msgr2=0x563ff18c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084b7fe640 1 --2- 172.21.15.60:0/3979291363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084b7fe640 1 -- 172.21.15.60:0/3979291363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 msgr2=0x563ff18cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084b7fe640 1 --2- 172.21.15.60:0/3979291363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18cbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084b7fe640 1 -- 172.21.15.60:0/3979291363 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ff1539210 con 0x563ff18c10d0 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f0850a5f640 1 -- 172.21.15.60:0/1200808147 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f08340efe00 con 0x7f084c106620 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084affd640 1 --2- 172.21.15.60:0/3979291363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084b7fe640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f08380a1f10 0x7f08380a43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084b7fe640 1 --2- 172.21.15.60:0/3979291363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18bfe20 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0844011590 tx=0x7f0844011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c029bbed673def1b server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f0841ffb640 1 -- 172.21.15.60:0/3979291363 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844017050 con 0x563ff18c10d0 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f0841ffb640 1 -- 172.21.15.60:0/3979291363 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f084401bb90 con 0x563ff18c10d0 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f0841ffb640 1 -- 172.21.15.60:0/3979291363 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084400edc0 con 0x563ff18c10d0 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.136+0000 7f084b7fe640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f08380a1f10 0x7f08380a43d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0844022000 tx=0x7f0844023000 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.137+0000 7f085463e740 1 -- 172.21.15.60:0/3979291363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 msgr2=0x563ff18bfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.137+0000 7f085463e740 1 --2- 172.21.15.60:0/3979291363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18bfe20 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0844011590 tx=0x7f0844011a50 comp rx=0 tx=0).stop 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.137+0000 7f085463e740 1 -- 172.21.15.60:0/3979291363 shutdown_connections 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.137+0000 7f085463e740 1 --2- 172.21.15.60:0/3979291363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18cbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.137+0000 7f085463e740 1 --2- 172.21.15.60:0/3979291363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.137+0000 7f085463e740 1 --2- 172.21.15.60:0/3979291363 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18bfe20 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.137+0000 7f085463e740 1 -- 172.21.15.60:0/3979291363 >> 172.21.15.60:0/3979291363 conn(0x563ff1887b90 msgr2=0x563ff18bb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.137+0000 7f085463e740 1 -- 172.21.15.60:0/3979291363 shutdown_connections 2024-03-04T12:27:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.137+0000 7f085463e740 1 -- 172.21.15.60:0/3979291363 wait complete. 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.137+0000 7f085463e740 1 -- start start 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f085463e740 1 Processor -- start 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f085463e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f085463e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c10d0 0x563ff18c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f085463e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c98e0 0x563ff18c7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f085463e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff18cc610 con 0x563ff18c98e0 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f085463e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff17e0d90 con 0x563ff18c10d0 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f085463e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ff18cc790 con 0x563ff18c0360 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f085463e740 10 client.0 _ll_register_callbacks cb 0x563ff1538a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f084b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c98e0 0x563ff18c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f0843fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c10d0 0x563ff18c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f084affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f084b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c98e0 0x563ff18c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51506/0 (socket says 172.21.15.60:51506) 2024-03-04T12:27:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f0843fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c10d0 0x563ff18c6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:39230/0 (socket says 172.21.15.60:39230) 2024-03-04T12:27:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f084b7fe640 1 -- 172.21.15.60:0/4110060346 learned_addr learned my addr 172.21.15.60:0/4110060346 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f084b7fe640 1 -- 172.21.15.60:0/4110060346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 msgr2=0x563ff18c6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f084b7fe640 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f084b7fe640 1 -- 172.21.15.60:0/4110060346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c10d0 msgr2=0x563ff18c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f084b7fe640 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c10d0 0x563ff18c6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f084b7fe640 1 -- 172.21.15.60:0/4110060346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ff1605620 con 0x563ff18c98e0 2024-03-04T12:27:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f0843fff640 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c10d0 0x563ff18c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f084affd640 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f084b7fe640 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c98e0 0x563ff18c7400 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0844005c10 tx=0x7f084400f420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7466549202dce7d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.138+0000 7f082affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f082affd640 1 -- 172.21.15.60:0/4110060346 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0844004bd0 con 0x563ff18c98e0 2024-03-04T12:27:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f082affd640 1 -- 172.21.15.60:0/4110060346 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f0844004d70 con 0x563ff18c98e0 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f085463e740 1 -- 172.21.15.60:0/4110060346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f084401bb90 con 0x563ff18c98e0 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f082affd640 1 -- 172.21.15.60:0/4110060346 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0844017020 con 0x563ff18c98e0 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f085463e740 10 client.16395 fetch_fsmap learned FSMap version 35 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f085463e740 1 -- 172.21.15.60:0/4110060346 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x563ff18d8950 con 0x563ff18c98e0 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f082affd640 1 -- 172.21.15.60:0/4110060346 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f084401bb90 con 0x563ff18c98e0 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f082affd640 1 -- 172.21.15.60:0/4110060346 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 35) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0844035030 con 0x563ff18c98e0 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f085463e740 10 client.16395 fetch_fsmap finished waiting for FSMap version 35 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f085463e740 10 client.16395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f085463e740 20 client.16395 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f085463e740 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x563ff18d9db0 0x563ff18dc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f085463e740 4 client.16395 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.139+0000 7f085463e740 1 -- 172.21.15.60:0/4110060346 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ff1831b30 con 0x563ff18d9db0 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.140+0000 7f084affd640 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x563ff18d9db0 0x563ff18dc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.140+0000 7f082affd640 10 client.16395 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.140+0000 7f084affd640 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x563ff18d9db0 0x563ff18dc1a0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c1a5c1edcc33ee9 server_cookie=631cad7a5311412c in_seq=0 out_seq=0 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.140+0000 7f082affd640 1 -- 172.21.15.60:0/4110060346 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563ff1831b30 con 0x563ff18d9db0 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.140+0000 7f082affd640 10 client.16395 handle_command_reply: tid=0 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f085463e740 1 client.16395 shutdown 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f085463e740 1 -- 172.21.15.60:0/4110060346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c98e0 msgr2=0x563ff18c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f085463e740 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c98e0 0x563ff18c7400 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0844005c10 tx=0x7f084400f420 comp rx=0 tx=0).stop 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f085463e740 1 -- 172.21.15.60:0/4110060346 shutdown_connections 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f085463e740 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x563ff18d9db0 0x563ff18dc1a0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f085463e740 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c98e0 0x563ff18c7400 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f085463e740 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c10d0 0x563ff18c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f082affd640 0 client.16395 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f085463e740 1 --2- 172.21.15.60:0/4110060346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f085463e740 1 -- 172.21.15.60:0/4110060346 >> 172.21.15.60:0/4110060346 conn(0x563ff1887b90 msgr2=0x563ff18bcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f085463e740 1 -- 172.21.15.60:0/4110060346 shutdown_connections 2024-03-04T12:27:28.155 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f085463e740 1 -- 172.21.15.60:0/4110060346 wait complete. 2024-03-04T12:27:28.156 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.141+0000 7f085463e740 20 client.16395 trim_cache size 0 max 16384 2024-03-04T12:27:28.167 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.155+0000 7f085463e740 1 Processor -- start 2024-03-04T12:27:28.167 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.155+0000 7f085463e740 1 -- start start 2024-03-04T12:27:28.167 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.155+0000 7f085463e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.167 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.155+0000 7f085463e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.155+0000 7f085463e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18c4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.155+0000 7f085463e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c1b0650 con 0x563ff18c10d0 2024-03-04T12:27:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.155+0000 7f085463e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c1b2030 con 0x563ff18c98e0 2024-03-04T12:27:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.155+0000 7f085463e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c1b0cb0 con 0x563ff18c0360 2024-03-04T12:27:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.155+0000 7f0843fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.155+0000 7f0843fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18c3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51522/0 (socket says 172.21.15.60:51522) 2024-03-04T12:27:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.155+0000 7f084affd640 1 --2- 172.21.15.60:0/3631657895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.155+0000 7f0843fff640 1 -- 172.21.15.60:0/3631657895 learned_addr learned my addr 172.21.15.60:0/3631657895 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.156+0000 7f084b7fe640 1 --2- 172.21.15.60:0/3631657895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18c4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.156+0000 7f0843fff640 1 -- 172.21.15.60:0/3631657895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 msgr2=0x563ff18c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.156+0000 7f0843fff640 1 --2- 172.21.15.60:0/3631657895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.168 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.156+0000 7f0843fff640 1 -- 172.21.15.60:0/3631657895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 msgr2=0x563ff18c4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.156+0000 7f0843fff640 1 --2- 172.21.15.60:0/3631657895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18c4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.156+0000 7f0843fff640 1 -- 172.21.15.60:0/3631657895 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ff18cc780 con 0x563ff18c10d0 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.156+0000 7f084b7fe640 1 --2- 172.21.15.60:0/3631657895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18c4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.156+0000 7f084affd640 1 --2- 172.21.15.60:0/3631657895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.156+0000 7f0843fff640 1 --2- 172.21.15.60:0/3631657895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18c3be0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f08340b8f80 tx=0x7f08340a7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915f833b65c5c25c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.156+0000 7f08417fa640 1 -- 172.21.15.60:0/3631657895 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083404b040 con 0x563ff18c10d0 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.156+0000 7f08417fa640 1 -- 172.21.15.60:0/3631657895 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0834004c80 con 0x563ff18c10d0 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.156+0000 7f08417fa640 1 -- 172.21.15.60:0/3631657895 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08340bc5b0 con 0x563ff18c10d0 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 -- 172.21.15.60:0/3631657895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 msgr2=0x563ff18c3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 --2- 172.21.15.60:0/3631657895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18c3be0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f08340b8f80 tx=0x7f08340a7040 comp rx=0 tx=0).stop 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 -- 172.21.15.60:0/3631657895 shutdown_connections 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 --2- 172.21.15.60:0/3631657895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18c4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.169 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 --2- 172.21.15.60:0/3631657895 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18c3be0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 --2- 172.21.15.60:0/3631657895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 -- 172.21.15.60:0/3631657895 >> 172.21.15.60:0/3631657895 conn(0x563ff1887b90 msgr2=0x563ff18c5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 -- 172.21.15.60:0/3631657895 shutdown_connections 2024-03-04T12:27:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 -- 172.21.15.60:0/3631657895 wait complete. 2024-03-04T12:27:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 -- start start 2024-03-04T12:27:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 Processor -- start 2024-03-04T12:27:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c1b19d0 con 0x563ff18c10d0 2024-03-04T12:27:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.157+0000 7f085463e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c1b1310 con 0x563ff18c98e0 2024-03-04T12:27:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f085463e740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c1ae8f0 con 0x563ff18c0360 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f085463e740 10 client.0 _ll_register_callbacks cb 0x7f0844011f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f0843fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f084affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f0843fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18cfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:51530/0 (socket says 172.21.15.60:51530) 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f084affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:52448/0 (socket says 172.21.15.60:52448) 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f0843fff640 1 -- 172.21.15.60:0/1260133074 learned_addr learned my addr 172.21.15.60:0/1260133074 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f0843fff640 1 -- 172.21.15.60:0/1260133074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 msgr2=0x563ff18cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f0843fff640 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f0843fff640 1 -- 172.21.15.60:0/1260133074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 msgr2=0x563ff18cffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f084b7fe640 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f0843fff640 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18cffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f0843fff640 1 -- 172.21.15.60:0/1260133074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ff18d5a20 con 0x563ff18c10d0 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f084affd640 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f0843fff640 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18cfa70 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f08340bc540 tx=0x7f08340ed1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437fb0a25a4d40a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f082affd640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f082affd640 1 -- 172.21.15.60:0/1260133074 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083404b040 con 0x563ff18c10d0 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f082affd640 1 -- 172.21.15.60:0/1260133074 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f08340bca10 con 0x563ff18c10d0 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f085463e740 1 -- 172.21.15.60:0/1260133074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0834004c80 con 0x563ff18c10d0 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f082affd640 1 -- 172.21.15.60:0/1260133074 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08340c4ca0 con 0x563ff18c10d0 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f085463e740 10 client.16401 fetch_fsmap learned FSMap version 35 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.158+0000 7f085463e740 1 -- 172.21.15.60:0/1260133074 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x563ff18bf630 con 0x563ff18c10d0 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.159+0000 7f082affd640 1 -- 172.21.15.60:0/1260133074 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0834004c80 con 0x563ff18c10d0 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.159+0000 7f082affd640 1 -- 172.21.15.60:0/1260133074 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 35) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f08340bba00 con 0x563ff18c10d0 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.159+0000 7f085463e740 10 client.16401 fetch_fsmap finished waiting for FSMap version 35 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.159+0000 7f085463e740 10 client.16401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.159+0000 7f085463e740 20 client.16401 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.159+0000 7f085463e740 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x563ff19316c0 0x563ff1933ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.159+0000 7f085463e740 4 client.16401 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T12:27:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.159+0000 7f085463e740 1 -- 172.21.15.60:0/1260133074 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563ff182be80 con 0x563ff19316c0 2024-03-04T12:27:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.159+0000 7f084affd640 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x563ff19316c0 0x563ff1933ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.159+0000 7f082affd640 10 client.16401 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.159+0000 7f084affd640 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x563ff19316c0 0x563ff1933ae0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad03e27c8007ec26 server_cookie=c6c1ef76b19c381e in_seq=0 out_seq=0 2024-03-04T12:27:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f082affd640 1 -- 172.21.15.60:0/1260133074 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563ff182be80 con 0x563ff19316c0 2024-03-04T12:27:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f082affd640 10 client.16401 handle_command_reply: tid=0 2024-03-04T12:27:28.181 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f085463e740 1 client.16401 shutdown 2024-03-04T12:27:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f085463e740 1 -- 172.21.15.60:0/1260133074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 msgr2=0x563ff18cfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f085463e740 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18cfa70 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f08340bc540 tx=0x7f08340ed1d0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f085463e740 1 -- 172.21.15.60:0/1260133074 shutdown_connections 2024-03-04T12:27:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f085463e740 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x563ff19316c0 0x563ff1933ae0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f085463e740 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ff18c98e0 0x563ff18cffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f085463e740 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x563ff18c10d0 0x563ff18cfa70 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f085463e740 1 --2- 172.21.15.60:0/1260133074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563ff18c0360 0x563ff18cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f082affd640 0 client.16401 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f085463e740 1 -- 172.21.15.60:0/1260133074 >> 172.21.15.60:0/1260133074 conn(0x563ff1887b90 msgr2=0x563ff18bc420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f085463e740 1 -- 172.21.15.60:0/1260133074 shutdown_connections 2024-03-04T12:27:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.174+0000 7f085463e740 1 -- 172.21.15.60:0/1260133074 wait complete. 2024-03-04T12:27:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f085463e740 20 client.16401 trim_cache size 0 max 16384 2024-03-04T12:27:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:{ 2024-03-04T12:27:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "message": "", 2024-03-04T12:27:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "return_code": 0 2024-03-04T12:27:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:} 2024-03-04T12:27:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f0851a61640 1 -- 172.21.15.60:0/1200808147 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f08380a1f10 msgr2=0x7f08380a43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f0851a61640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f08380a1f10 0x7f08380a43d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0844022000 tx=0x7f0844023000 comp rx=0 tx=0).stop 2024-03-04T12:27:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f0851a61640 1 -- 172.21.15.60:0/1200808147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f084c106620 msgr2=0x7f084c073b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f0851a61640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f084c106620 0x7f084c073b70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0834036040 tx=0x7f083402cb20 comp rx=0 tx=0).stop 2024-03-04T12:27:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f0851a61640 1 -- 172.21.15.60:0/1200808147 shutdown_connections 2024-03-04T12:27:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f0851a61640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f084c0740b0 0x7f084c1a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f0851a61640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f084c106620 0x7f084c073b70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f0851a61640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f08380a1f10 0x7f08380a43d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f0851a61640 1 --2- 172.21.15.60:0/1200808147 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f084c101550 0x7f084c073630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:28.184 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f0851a61640 1 -- 172.21.15.60:0/1200808147 >> 172.21.15.60:0/1200808147 conn(0x7f084c0fcdb0 msgr2=0x7f084c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f0851a61640 1 -- 172.21.15.60:0/1200808147 shutdown_connections 2024-03-04T12:27:28.185 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:28.175+0000 7f0851a61640 1 -- 172.21.15.60:0/1200808147 wait complete. 2024-03-04T12:27:28.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:27 smithi086 ceph-mon[36851]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:28.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:27 smithi063 ceph-mon[36717]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:28.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:27 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a[32741]: 2024-03-04T12:27:27.979+0000 7f4240087700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-04T12:27:28.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:27 smithi060 ceph-mon[32745]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:29.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:28 smithi086 ceph-mon[36851]: pgmap v51: 129 pgs: 129 active+clean; 20 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 587 KiB/s wr, 159 op/s 2024-03-04T12:27:29.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:28 smithi086 ceph-mon[36851]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-04T12:27:29.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:28 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] up:active 2024-03-04T12:27:29.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:28 smithi086 ceph-mon[36851]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:29.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:28 smithi063 ceph-mon[36717]: pgmap v51: 129 pgs: 129 active+clean; 20 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 587 KiB/s wr, 159 op/s 2024-03-04T12:27:29.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:28 smithi063 ceph-mon[36717]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-04T12:27:29.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:28 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] up:active 2024-03-04T12:27:29.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:28 smithi063 ceph-mon[36717]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:29.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:28 smithi060 ceph-mon[32745]: pgmap v51: 129 pgs: 129 active+clean; 20 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 587 KiB/s wr, 159 op/s 2024-03-04T12:27:29.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:28 smithi060 ceph-mon[32745]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-04T12:27:29.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:28 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] up:active 2024-03-04T12:27:29.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:28 smithi060 ceph-mon[32745]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:31.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:30 smithi086 ceph-mon[36851]: pgmap v52: 129 pgs: 129 active+clean; 32 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 221 op/s 2024-03-04T12:27:31.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:30 smithi063 ceph-mon[36717]: pgmap v52: 129 pgs: 129 active+clean; 32 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 221 op/s 2024-03-04T12:27:31.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:30 smithi060 ceph-mon[32745]: pgmap v52: 129 pgs: 129 active+clean; 32 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.8 MiB/s wr, 221 op/s 2024-03-04T12:27:32.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:31 smithi086 ceph-mon[36851]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:32.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:31 smithi063 ceph-mon[36717]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:32.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:31 smithi060 ceph-mon[32745]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:27:33.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:32 smithi086 ceph-mon[36851]: pgmap v53: 129 pgs: 129 active+clean; 32 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 105 op/s 2024-03-04T12:27:33.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:32 smithi063 ceph-mon[36717]: pgmap v53: 129 pgs: 129 active+clean; 32 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 105 op/s 2024-03-04T12:27:33.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:32 smithi060 ceph-mon[32745]: pgmap v53: 129 pgs: 129 active+clean; 32 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 105 op/s 2024-03-04T12:27:35.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:35 smithi086 ceph-mon[36851]: pgmap v54: 129 pgs: 129 active+clean; 32 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 105 op/s 2024-03-04T12:27:35.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:35 smithi063 ceph-mon[36717]: pgmap v54: 129 pgs: 129 active+clean; 32 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 105 op/s 2024-03-04T12:27:35.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:35 smithi060 ceph-mon[32745]: pgmap v54: 129 pgs: 129 active+clean; 32 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 105 op/s 2024-03-04T12:27:37.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:37 smithi086 ceph-mon[36851]: pgmap v55: 129 pgs: 129 active+clean; 56 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 551 op/s 2024-03-04T12:27:37.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:37 smithi086 ceph-mon[36851]: scrub summary: idle 2024-03-04T12:27:37.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:37 smithi063 ceph-mon[36717]: pgmap v55: 129 pgs: 129 active+clean; 56 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 551 op/s 2024-03-04T12:27:37.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:37 smithi063 ceph-mon[36717]: scrub summary: idle 2024-03-04T12:27:37.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:37 smithi060 ceph-mon[32745]: pgmap v55: 129 pgs: 129 active+clean; 56 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.9 MiB/s wr, 551 op/s 2024-03-04T12:27:37.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:37 smithi060 ceph-mon[32745]: scrub summary: idle 2024-03-04T12:27:38.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:38 smithi086 ceph-mon[36851]: scrub summary: idle 2024-03-04T12:27:38.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:38 smithi063 ceph-mon[36717]: scrub summary: idle 2024-03-04T12:27:38.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:38 smithi060 ceph-mon[32745]: scrub summary: idle 2024-03-04T12:27:39.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:39 smithi086 ceph-mon[36851]: pgmap v56: 129 pgs: 129 active+clean; 56 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 508 op/s 2024-03-04T12:27:39.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:39 smithi063 ceph-mon[36717]: pgmap v56: 129 pgs: 129 active+clean; 56 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 508 op/s 2024-03-04T12:27:39.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:39 smithi060 ceph-mon[32745]: pgmap v56: 129 pgs: 129 active+clean; 56 MiB data, 552 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.6 MiB/s wr, 508 op/s 2024-03-04T12:27:39.661 INFO:tasks.workunit.client.0.smithi060.stderr:Note: switching to '12973fc04fd10d4af086901e10ffa8e48866b735'. 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr:state without impacting any branches by switching back to a branch. 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr: git switch -c 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr:Or undo this operation with: 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr: git switch - 2024-03-04T12:27:39.662 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:39.663 INFO:tasks.workunit.client.0.smithi060.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-04T12:27:39.663 INFO:tasks.workunit.client.0.smithi060.stderr: 2024-03-04T12:27:39.663 INFO:tasks.workunit.client.0.smithi060.stderr:HEAD is now at 12973fc0 btrfs/022: Correct seq number in golden output 2024-03-04T12:27:39.663 INFO:tasks.workunit.client.0.smithi060.stderr:+ make -j4 2024-03-04T12:27:39.671 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize -cfi 2024-03-04T12:27:40.091 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize: putting auxiliary files in '.'. 2024-03-04T12:27:40.091 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize: copying file './config.guess' 2024-03-04T12:27:40.103 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize: copying file './config.sub' 2024-03-04T12:27:40.114 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize: copying file './install-sh' 2024-03-04T12:27:40.126 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize: copying file './ltmain.sh' 2024-03-04T12:27:40.153 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-03-04T12:27:40.153 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-03-04T12:27:40.159 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-03-04T12:27:40.163 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-03-04T12:27:40.167 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-03-04T12:27:40.172 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-03-04T12:27:40.178 INFO:tasks.workunit.client.0.smithi060.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-03-04T12:27:40.179 INFO:tasks.workunit.client.0.smithi060.stdout:cp include/install-sh . 2024-03-04T12:27:40.186 INFO:tasks.workunit.client.0.smithi060.stdout:aclocal -I m4 2024-03-04T12:27:41.271 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.262+0000 7ff717436640 1 -- 172.21.15.60:0/1500336691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff710105510 msgr2=0x7ff71010d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.262+0000 7ff717436640 1 --2- 172.21.15.60:0/1500336691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff710105510 0x7ff71010d0e0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7ff708009ca0 tx=0x7ff70802c0f0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.262+0000 7ff717436640 1 -- 172.21.15.60:0/1500336691 shutdown_connections 2024-03-04T12:27:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.262+0000 7ff717436640 1 --2- 172.21.15.60:0/1500336691 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff710105510 0x7ff71010d0e0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.262+0000 7ff717436640 1 --2- 172.21.15.60:0/1500336691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff710100df0 0x7ff710104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.262+0000 7ff717436640 1 --2- 172.21.15.60:0/1500336691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff710100440 0x7ff710100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.262+0000 7ff717436640 1 -- 172.21.15.60:0/1500336691 >> 172.21.15.60:0/1500336691 conn(0x7ff710077320 msgr2=0x7ff710077730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.262+0000 7ff717436640 1 -- 172.21.15.60:0/1500336691 shutdown_connections 2024-03-04T12:27:41.272 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.262+0000 7ff717436640 1 -- 172.21.15.60:0/1500336691 wait complete. 2024-03-04T12:27:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff717436640 1 Processor -- start 2024-03-04T12:27:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff717436640 1 -- start start 2024-03-04T12:27:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff717436640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff710100440 0x7ff710101ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff717436640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff710105510 0x7ff710102410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff717436640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff710102950 0x7ff7101a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff717436640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7101105d0 con 0x7ff710105510 2024-03-04T12:27:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff717436640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff710110450 con 0x7ff710100440 2024-03-04T12:27:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff717436640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff710110750 con 0x7ff710102950 2024-03-04T12:27:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff7149d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff710105510 0x7ff710102410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff7149d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff710105510 0x7ff710102410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45874/0 (socket says 172.21.15.60:45874) 2024-03-04T12:27:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff7149d1640 1 -- 172.21.15.60:0/628942910 learned_addr learned my addr 172.21.15.60:0/628942910 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:41.273 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff7151d2640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff710100440 0x7ff710101ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff7159d3640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff710102950 0x7ff7101a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff7149d1640 1 -- 172.21.15.60:0/628942910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff710102950 msgr2=0x7ff7101a5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff7149d1640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff710102950 0x7ff7101a5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff7149d1640 1 -- 172.21.15.60:0/628942910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff710100440 msgr2=0x7ff710101ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff7149d1640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff710100440 0x7ff710101ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff7149d1640 1 -- 172.21.15.60:0/628942910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7101a6450 con 0x7ff710105510 2024-03-04T12:27:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.263+0000 7ff7151d2640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff710100440 0x7ff710101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.264+0000 7ff7149d1640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff710105510 0x7ff710102410 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff70000e9c0 tx=0x7ff70000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.264+0000 7ff7067fc640 1 -- 172.21.15.60:0/628942910 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70000cd80 con 0x7ff710105510 2024-03-04T12:27:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.264+0000 7ff7067fc640 1 -- 172.21.15.60:0/628942910 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff700005500 con 0x7ff710105510 2024-03-04T12:27:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.264+0000 7ff717436640 1 -- 172.21.15.60:0/628942910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7101a6740 con 0x7ff710105510 2024-03-04T12:27:41.274 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.264+0000 7ff7067fc640 1 -- 172.21.15.60:0/628942910 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff700010690 con 0x7ff710105510 2024-03-04T12:27:41.275 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.264+0000 7ff717436640 1 -- 172.21.15.60:0/628942910 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7101a6ce0 con 0x7ff710105510 2024-03-04T12:27:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.266+0000 7ff7067fc640 1 -- 172.21.15.60:0/628942910 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7ff7000046c0 con 0x7ff710105510 2024-03-04T12:27:41.281 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff71a018740 1 Processor -- start 2024-03-04T12:27:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff71a018740 1 -- start start 2024-03-04T12:27:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff71a018740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4d0d0 0x55ae43d4be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff71a018740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4c360 0x55ae43d4c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff71a018740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d558e0 0x55ae43d57cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff71a018740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae43cb05f0 con 0x55ae43d558e0 2024-03-04T12:27:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff71a018740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae43c59930 con 0x55ae43d4d0d0 2024-03-04T12:27:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff71a018740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae43cac690 con 0x55ae43d4c360 2024-03-04T12:27:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff7067fc640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff6fc0a1f90 0x7ff6fc0a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff7151d2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4c360 0x55ae43d4c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff7149d1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d558e0 0x55ae43d57cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff7159d3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4d0d0 0x55ae43d4be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.282 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff7151d2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4c360 0x55ae43d4c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:44634/0 (socket says 172.21.15.60:44634) 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff7151d2640 1 -- 172.21.15.60:0/3166822985 learned_addr learned my addr 172.21.15.60:0/3166822985 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff7159d3640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff6fc0a1f90 0x7ff6fc0a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff7067fc640 1 -- 172.21.15.60:0/628942910 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7ff7000ce7c0 con 0x7ff710105510 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff7151d2640 1 -- 172.21.15.60:0/3166822985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4d0d0 msgr2=0x55ae43d4be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff7151d2640 1 --2- 172.21.15.60:0/3166822985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4d0d0 0x55ae43d4be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff7151d2640 1 -- 172.21.15.60:0/3166822985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d558e0 msgr2=0x55ae43d57cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.267+0000 7ff7151d2640 1 --2- 172.21.15.60:0/3166822985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d558e0 0x55ae43d57cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.268+0000 7ff7151d2640 1 -- 172.21.15.60:0/3166822985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae439c5210 con 0x55ae43d4c360 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.268+0000 7ff7159d3640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff6fc0a1f90 0x7ff6fc0a4450 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff708004af0 tx=0x7ff708005c50 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.268+0000 7ff7151d2640 1 --2- 172.21.15.60:0/3166822985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4c360 0x55ae43d4c7e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff70c0115c0 tx=0x7ff70c011a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=508d88a6b5d80cda server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.268+0000 7ff6fb7fe640 1 -- 172.21.15.60:0/3166822985 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c00f640 con 0x55ae43d4c360 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.268+0000 7ff6fb7fe640 1 -- 172.21.15.60:0/3166822985 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff70c00fcc0 con 0x55ae43d4c360 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.268+0000 7ff6fb7fe640 1 -- 172.21.15.60:0/3166822985 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c019d20 con 0x55ae43d4c360 2024-03-04T12:27:41.283 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 -- 172.21.15.60:0/3166822985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4c360 msgr2=0x55ae43d4c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 --2- 172.21.15.60:0/3166822985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4c360 0x55ae43d4c7e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff70c0115c0 tx=0x7ff70c011a80 comp rx=0 tx=0).stop 2024-03-04T12:27:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 -- 172.21.15.60:0/3166822985 shutdown_connections 2024-03-04T12:27:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 --2- 172.21.15.60:0/3166822985 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d558e0 0x55ae43d57cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 --2- 172.21.15.60:0/3166822985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4c360 0x55ae43d4c7e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 --2- 172.21.15.60:0/3166822985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4d0d0 0x55ae43d4be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 -- 172.21.15.60:0/3166822985 >> 172.21.15.60:0/3166822985 conn(0x55ae43d13b90 msgr2=0x55ae43d47050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 -- 172.21.15.60:0/3166822985 shutdown_connections 2024-03-04T12:27:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 -- 172.21.15.60:0/3166822985 wait complete. 2024-03-04T12:27:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 -- start start 2024-03-04T12:27:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 Processor -- start 2024-03-04T12:27:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 0x55ae43d52940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.284 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4d0d0 0x55ae43d52e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.269+0000 7ff71a018740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d558e0 0x55ae43d533c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff71a018740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae43d58610 con 0x55ae43d4c360 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff71a018740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae43c6cd90 con 0x55ae43d558e0 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff71a018740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae43d58790 con 0x55ae43d4d0d0 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff71a018740 10 client.0 _ll_register_callbacks cb 0x55ae439c4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff7151d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 0x55ae43d52940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff7151d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 0x55ae43d52940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45904/0 (socket says 172.21.15.60:45904) 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff7151d2640 1 -- 172.21.15.60:0/1918004817 learned_addr learned my addr 172.21.15.60:0/1918004817 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff7149d1640 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4d0d0 0x55ae43d52e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff7159d3640 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d558e0 0x55ae43d533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff7151d2640 1 -- 172.21.15.60:0/1918004817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4d0d0 msgr2=0x55ae43d52e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff7151d2640 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4d0d0 0x55ae43d52e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.285 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff7151d2640 1 -- 172.21.15.60:0/1918004817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d558e0 msgr2=0x55ae43d533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff7151d2640 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d558e0 0x55ae43d533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff7151d2640 1 -- 172.21.15.60:0/1918004817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ae43a91620 con 0x55ae43d4c360 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff7159d3640 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d558e0 0x55ae43d533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff7151d2640 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 0x55ae43d52940 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff70c005df0 tx=0x7ff70c0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a53efb5b01afbd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff6f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff6f8ff9640 1 -- 172.21.15.60:0/1918004817 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c026020 con 0x55ae43d4c360 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff6f8ff9640 1 -- 172.21.15.60:0/1918004817 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7ff70c019cf0 con 0x55ae43d4c360 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff71a018740 1 -- 172.21.15.60:0/1918004817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff70c00fcc0 con 0x55ae43d4c360 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff6f8ff9640 1 -- 172.21.15.60:0/1918004817 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff70c031050 con 0x55ae43d4c360 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.270+0000 7ff71a018740 10 client.16413 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.271+0000 7ff71a018740 1 -- 172.21.15.60:0/1918004817 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55ae43d64910 con 0x55ae43d4c360 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.271+0000 7ff6f8ff9640 1 -- 172.21.15.60:0/1918004817 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff70c00fcc0 con 0x55ae43d4c360 2024-03-04T12:27:41.286 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.271+0000 7ff6f8ff9640 1 -- 172.21.15.60:0/1918004817 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7ff70c003680 con 0x55ae43d4c360 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.271+0000 7ff71a018740 10 client.16413 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.271+0000 7ff71a018740 10 client.16413 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.271+0000 7ff71a018740 20 client.16413 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.271+0000 7ff71a018740 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ae43d65d70 0x55ae43d68160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.271+0000 7ff71a018740 4 client.16413 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.271+0000 7ff71a018740 1 -- 172.21.15.60:0/1918004817 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ae43cbdb30 con 0x55ae43d65d70 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.271+0000 7ff7149d1640 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ae43d65d70 0x55ae43d68160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.272+0000 7ff7149d1640 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ae43d65d70 0x55ae43d68160 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2741541f46eed986 server_cookie=5ec222f43c029ff9 in_seq=0 out_seq=0 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.272+0000 7ff6f8ff9640 10 client.16413 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.272+0000 7ff6f8ff9640 1 -- 172.21.15.60:0/1918004817 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ae43cbdb30 con 0x55ae43d65d70 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.272+0000 7ff6f8ff9640 10 client.16413 handle_command_reply: tid=0 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.272+0000 7ff71a018740 1 client.16413 shutdown 2024-03-04T12:27:41.287 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.272+0000 7ff71a018740 1 -- 172.21.15.60:0/1918004817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 msgr2=0x55ae43d52940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.272+0000 7ff71a018740 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 0x55ae43d52940 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff70c005df0 tx=0x7ff70c0056d0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.273+0000 7ff71a018740 1 -- 172.21.15.60:0/1918004817 shutdown_connections 2024-03-04T12:27:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.273+0000 7ff71a018740 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ae43d65d70 0x55ae43d68160 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.273+0000 7ff71a018740 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d558e0 0x55ae43d533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.273+0000 7ff6f8ff9640 0 client.16413 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.273+0000 7ff71a018740 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4d0d0 0x55ae43d52e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.273+0000 7ff71a018740 1 --2- 172.21.15.60:0/1918004817 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 0x55ae43d52940 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.273+0000 7ff71a018740 1 -- 172.21.15.60:0/1918004817 >> 172.21.15.60:0/1918004817 conn(0x55ae43d13b90 msgr2=0x55ae43d48da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.273+0000 7ff71a018740 1 -- 172.21.15.60:0/1918004817 shutdown_connections 2024-03-04T12:27:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.273+0000 7ff71a018740 1 -- 172.21.15.60:0/1918004817 wait complete. 2024-03-04T12:27:41.288 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.273+0000 7ff71a018740 20 client.16413 trim_cache size 0 max 16384 2024-03-04T12:27:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.286+0000 7ff71a018740 1 Processor -- start 2024-03-04T12:27:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.286+0000 7ff71a018740 1 -- start start 2024-03-04T12:27:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff71a018740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4c360 0x55ae43d5ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff71a018740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4d0d0 0x55ae43d5cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff71a018740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d558e0 0x55ae43d51230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.298 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff71a018740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7101b0690 con 0x55ae43d558e0 2024-03-04T12:27:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff71a018740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7101b2070 con 0x55ae43d4c360 2024-03-04T12:27:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff71a018740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7101b0cf0 con 0x55ae43d4d0d0 2024-03-04T12:27:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff7159d3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d558e0 0x55ae43d51230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff7159d3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d558e0 0x55ae43d51230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45908/0 (socket says 172.21.15.60:45908) 2024-03-04T12:27:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff7159d3640 1 -- 172.21.15.60:0/791671589 learned_addr learned my addr 172.21.15.60:0/791671589 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff7151d2640 1 --2- 172.21.15.60:0/791671589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4c360 0x55ae43d5ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff7149d1640 1 --2- 172.21.15.60:0/791671589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4d0d0 0x55ae43d5cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff7159d3640 1 -- 172.21.15.60:0/791671589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4d0d0 msgr2=0x55ae43d5cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff7159d3640 1 --2- 172.21.15.60:0/791671589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4d0d0 0x55ae43d5cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff7159d3640 1 -- 172.21.15.60:0/791671589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4c360 msgr2=0x55ae43d5ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:41.299 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff7159d3640 1 --2- 172.21.15.60:0/791671589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4c360 0x55ae43d5ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff7159d3640 1 -- 172.21.15.60:0/791671589 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae43d6d1c0 con 0x55ae43d558e0 2024-03-04T12:27:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff7159d3640 1 --2- 172.21.15.60:0/791671589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d558e0 0x55ae43d51230 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff708036000 tx=0x7ff70802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f634de443924cf server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff6faffd640 1 -- 172.21.15.60:0/791671589 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff708039040 con 0x55ae43d558e0 2024-03-04T12:27:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff6faffd640 1 -- 172.21.15.60:0/791671589 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff708003e50 con 0x55ae43d558e0 2024-03-04T12:27:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.287+0000 7ff6faffd640 1 -- 172.21.15.60:0/791671589 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7080043a0 con 0x55ae43d558e0 2024-03-04T12:27:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.288+0000 7ff71a018740 1 -- 172.21.15.60:0/791671589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d558e0 msgr2=0x55ae43d51230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.288+0000 7ff71a018740 1 --2- 172.21.15.60:0/791671589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d558e0 0x55ae43d51230 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff708036000 tx=0x7ff70802e040 comp rx=0 tx=0).stop 2024-03-04T12:27:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.288+0000 7ff71a018740 1 -- 172.21.15.60:0/791671589 shutdown_connections 2024-03-04T12:27:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.288+0000 7ff71a018740 1 --2- 172.21.15.60:0/791671589 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d558e0 0x55ae43d51230 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.288+0000 7ff71a018740 1 --2- 172.21.15.60:0/791671589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d4d0d0 0x55ae43d5cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.288+0000 7ff71a018740 1 --2- 172.21.15.60:0/791671589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4c360 0x55ae43d5ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.300 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.288+0000 7ff71a018740 1 -- 172.21.15.60:0/791671589 >> 172.21.15.60:0/791671589 conn(0x55ae43d13b90 msgr2=0x55ae43d46b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.288+0000 7ff71a018740 1 -- 172.21.15.60:0/791671589 shutdown_connections 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.288+0000 7ff71a018740 1 -- 172.21.15.60:0/791671589 wait complete. 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.288+0000 7ff71a018740 1 -- start start 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff71a018740 1 Processor -- start 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff71a018740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 0x55ae43d5ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff71a018740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4d0d0 0x55ae43d4f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff71a018740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d558e0 0x55ae43d4fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff71a018740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7101b1a10 con 0x55ae43d4c360 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff71a018740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7101b1350 con 0x55ae43d4d0d0 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff71a018740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7101ae930 con 0x55ae43d558e0 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff71a018740 10 client.0 _ll_register_callbacks cb 0x7ff700009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff7151d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 0x55ae43d5ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.301 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff7151d2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 0x55ae43d5ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:45914/0 (socket says 172.21.15.60:45914) 2024-03-04T12:27:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff7151d2640 1 -- 172.21.15.60:0/2697518078 learned_addr learned my addr 172.21.15.60:0/2697518078 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff7159d3640 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d558e0 0x55ae43d4fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff7149d1640 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4d0d0 0x55ae43d4f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff7151d2640 1 -- 172.21.15.60:0/2697518078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d558e0 msgr2=0x55ae43d4fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff7151d2640 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d558e0 0x55ae43d4fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff7151d2640 1 -- 172.21.15.60:0/2697518078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4d0d0 msgr2=0x55ae43d4f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff7151d2640 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4d0d0 0x55ae43d4f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff7151d2640 1 -- 172.21.15.60:0/2697518078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ae43d50210 con 0x55ae43d4c360 2024-03-04T12:27:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.289+0000 7ff7149d1640 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4d0d0 0x55ae43d4f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff7151d2640 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 0x55ae43d5ce30 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff70c003a20 tx=0x7ff70c003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c168ecd29987833a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff6f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:41.302 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff6f8ff9640 1 -- 172.21.15.60:0/2697518078 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70c026040 con 0x55ae43d4c360 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff6f8ff9640 1 -- 172.21.15.60:0/2697518078 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7ff70c011230 con 0x55ae43d4c360 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff71a018740 1 -- 172.21.15.60:0/2697518078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff708003e50 con 0x55ae43d4c360 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff6f8ff9640 1 -- 172.21.15.60:0/2697518078 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff70c011430 con 0x55ae43d4c360 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff71a018740 10 client.16419 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff71a018740 1 -- 172.21.15.60:0/2697518078 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55ae43dbd1f0 con 0x55ae43d4c360 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff6f8ff9640 1 -- 172.21.15.60:0/2697518078 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff708003e50 con 0x55ae43d4c360 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff6f8ff9640 1 -- 172.21.15.60:0/2697518078 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7ff70c00f800 con 0x55ae43d4c360 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff71a018740 10 client.16419 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff71a018740 10 client.16419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff71a018740 20 client.16419 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff71a018740 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ae43d65d70 0x55ae43dbf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff71a018740 4 client.16419 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.290+0000 7ff71a018740 1 -- 172.21.15.60:0/2697518078 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ae43cb7e80 con 0x55ae43d65d70 2024-03-04T12:27:41.303 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.291+0000 7ff7149d1640 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ae43d65d70 0x55ae43dbf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.291+0000 7ff6f8ff9640 10 client.16419 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:41.304 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.291+0000 7ff7149d1640 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ae43d65d70 0x55ae43dbf9e0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b6f229d6cd85079 server_cookie=b7dc54a9e50640ed in_seq=0 out_seq=0 2024-03-04T12:27:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.307+0000 7ff6f8ff9640 1 -- 172.21.15.60:0/2697518078 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ae43cb7e80 con 0x55ae43d65d70 2024-03-04T12:27:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.307+0000 7ff6f8ff9640 10 client.16419 handle_command_reply: tid=0 2024-03-04T12:27:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.307+0000 7ff71a018740 1 client.16419 shutdown 2024-03-04T12:27:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.307+0000 7ff71a018740 1 -- 172.21.15.60:0/2697518078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 msgr2=0x55ae43d5ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.307+0000 7ff71a018740 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 0x55ae43d5ce30 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff70c003a20 tx=0x7ff70c003e20 comp rx=0 tx=0).stop 2024-03-04T12:27:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.307+0000 7ff71a018740 1 -- 172.21.15.60:0/2697518078 shutdown_connections 2024-03-04T12:27:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.307+0000 7ff71a018740 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55ae43d65d70 0x55ae43dbf9e0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.307+0000 7ff71a018740 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ae43d558e0 0x55ae43d4fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.307+0000 7ff6f8ff9640 0 client.16419 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.307+0000 7ff71a018740 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae43d4d0d0 0x55ae43d4f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.307+0000 7ff71a018740 1 --2- 172.21.15.60:0/2697518078 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55ae43d4c360 0x55ae43d5ce30 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.307+0000 7ff71a018740 1 -- 172.21.15.60:0/2697518078 >> 172.21.15.60:0/2697518078 conn(0x55ae43d13b90 msgr2=0x55ae43d510a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.308+0000 7ff71a018740 1 -- 172.21.15.60:0/2697518078 shutdown_connections 2024-03-04T12:27:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.308+0000 7ff71a018740 1 -- 172.21.15.60:0/2697518078 wait complete. 2024-03-04T12:27:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.308+0000 7ff71a018740 20 client.16419 trim_cache size 0 max 16384 2024-03-04T12:27:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:{ 2024-03-04T12:27:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "message": "", 2024-03-04T12:27:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "return_code": 0 2024-03-04T12:27:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:} 2024-03-04T12:27:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.308+0000 7ff717436640 1 -- 172.21.15.60:0/628942910 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff6fc0a1f90 msgr2=0x7ff6fc0a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.308+0000 7ff717436640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff6fc0a1f90 0x7ff6fc0a4450 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff708004af0 tx=0x7ff708005c50 comp rx=0 tx=0).stop 2024-03-04T12:27:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.308+0000 7ff717436640 1 -- 172.21.15.60:0/628942910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff710105510 msgr2=0x7ff710102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.308+0000 7ff717436640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff710105510 0x7ff710102410 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff70000e9c0 tx=0x7ff70000ee80 comp rx=0 tx=0).stop 2024-03-04T12:27:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.308+0000 7ff717436640 1 -- 172.21.15.60:0/628942910 shutdown_connections 2024-03-04T12:27:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.308+0000 7ff717436640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff710102950 0x7ff7101a5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.308+0000 7ff717436640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ff6fc0a1f90 0x7ff6fc0a4450 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.308+0000 7ff717436640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ff710105510 0x7ff710102410 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.308+0000 7ff717436640 1 --2- 172.21.15.60:0/628942910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff710100440 0x7ff710101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.309+0000 7ff717436640 1 -- 172.21.15.60:0/628942910 >> 172.21.15.60:0/628942910 conn(0x7ff710077320 msgr2=0x7ff710068e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.309+0000 7ff717436640 1 -- 172.21.15.60:0/628942910 shutdown_connections 2024-03-04T12:27:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:41.309+0000 7ff717436640 1 -- 172.21.15.60:0/628942910 wait complete. 2024-03-04T12:27:41.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:41 smithi086 ceph-mon[36851]: pgmap v57: 129 pgs: 129 active+clean; 63 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.6 MiB/s wr, 720 op/s 2024-03-04T12:27:41.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:41 smithi063 ceph-mon[36717]: pgmap v57: 129 pgs: 129 active+clean; 63 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.6 MiB/s wr, 720 op/s 2024-03-04T12:27:41.365 INFO:tasks.workunit.client.0.smithi060.stdout:autoheader 2024-03-04T12:27:41.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:41 smithi060 ceph-mon[32745]: pgmap v57: 129 pgs: 129 active+clean; 63 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.6 MiB/s wr, 720 op/s 2024-03-04T12:27:41.821 INFO:tasks.workunit.client.0.smithi060.stdout:autoconf 2024-03-04T12:27:42.009 INFO:tasks.workunit.client.0.smithi060.stdout:./configure \ 2024-03-04T12:27:42.009 INFO:tasks.workunit.client.0.smithi060.stdout: --libexecdir=/usr/lib \ 2024-03-04T12:27:42.010 INFO:tasks.workunit.client.0.smithi060.stdout: --exec_prefix=/var/lib 2024-03-04T12:27:42.169 INFO:tasks.workunit.client.0.smithi060.stdout:checking build system type... x86_64-pc-linux-gnu 2024-03-04T12:27:42.169 INFO:tasks.workunit.client.0.smithi060.stdout:checking host system type... x86_64-pc-linux-gnu 2024-03-04T12:27:42.170 INFO:tasks.workunit.client.0.smithi060.stdout:checking how to print strings... printf 2024-03-04T12:27:42.170 INFO:tasks.workunit.client.0.smithi060.stdout:checking for gcc... gcc 2024-03-04T12:27:42.284 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether the C compiler works... yes 2024-03-04T12:27:42.285 INFO:tasks.workunit.client.0.smithi060.stdout:checking for C compiler default output file name... a.out 2024-03-04T12:27:42.336 INFO:tasks.workunit.client.0.smithi060.stdout:checking for suffix of executables... 2024-03-04T12:27:42.382 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether we are cross compiling... no 2024-03-04T12:27:42.429 INFO:tasks.workunit.client.0.smithi060.stdout:checking for suffix of object files... o 2024-03-04T12:27:42.462 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether we are using the GNU C compiler... yes 2024-03-04T12:27:42.494 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether gcc accepts -g... yes 2024-03-04T12:27:42.586 INFO:tasks.workunit.client.0.smithi060.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-04T12:27:42.589 INFO:tasks.workunit.client.0.smithi060.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-03-04T12:27:42.592 INFO:tasks.workunit.client.0.smithi060.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-04T12:27:42.593 INFO:tasks.workunit.client.0.smithi060.stdout:checking for egrep... /usr/bin/grep -E 2024-03-04T12:27:42.594 INFO:tasks.workunit.client.0.smithi060.stdout:checking for fgrep... /usr/bin/grep -F 2024-03-04T12:27:42.606 INFO:tasks.workunit.client.0.smithi060.stdout:checking for ld used by gcc... /usr/bin/ld 2024-03-04T12:27:42.608 INFO:tasks.workunit.client.0.smithi060.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-03-04T12:27:42.622 INFO:tasks.workunit.client.0.smithi060.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-03-04T12:27:43.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:43 smithi086 ceph-mon[36851]: pgmap v58: 129 pgs: 129 active+clean; 63 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 658 op/s 2024-03-04T12:27:43.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:43 smithi063 ceph-mon[36717]: pgmap v58: 129 pgs: 129 active+clean; 63 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 658 op/s 2024-03-04T12:27:43.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:43 smithi060 ceph-mon[32745]: pgmap v58: 129 pgs: 129 active+clean; 63 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 658 op/s 2024-03-04T12:27:43.605 INFO:tasks.workunit.client.0.smithi060.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-03-04T12:27:43.605 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether ln -s works... yes 2024-03-04T12:27:43.626 INFO:tasks.workunit.client.0.smithi060.stdout:checking the maximum length of command line arguments... 1572864 2024-03-04T12:27:43.630 INFO:tasks.workunit.client.0.smithi060.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-03-04T12:27:43.630 INFO:tasks.workunit.client.0.smithi060.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-03-04T12:27:43.630 INFO:tasks.workunit.client.0.smithi060.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-03-04T12:27:43.630 INFO:tasks.workunit.client.0.smithi060.stdout:checking for objdump... objdump 2024-03-04T12:27:43.630 INFO:tasks.workunit.client.0.smithi060.stdout:checking how to recognize dependent libraries... pass_all 2024-03-04T12:27:43.630 INFO:tasks.workunit.client.0.smithi060.stdout:checking for dlltool... no 2024-03-04T12:27:43.630 INFO:tasks.workunit.client.0.smithi060.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-03-04T12:27:43.631 INFO:tasks.workunit.client.0.smithi060.stdout:checking for ar... ar 2024-03-04T12:27:43.737 INFO:tasks.workunit.client.0.smithi060.stdout:checking for archiver @FILE support... @ 2024-03-04T12:27:43.737 INFO:tasks.workunit.client.0.smithi060.stdout:checking for strip... strip 2024-03-04T12:27:43.737 INFO:tasks.workunit.client.0.smithi060.stdout:checking for ranlib... ranlib 2024-03-04T12:27:43.737 INFO:tasks.workunit.client.0.smithi060.stdout:checking for gawk... gawk 2024-03-04T12:27:43.867 INFO:tasks.workunit.client.0.smithi060.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-03-04T12:27:43.875 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sysroot... no 2024-03-04T12:27:43.896 INFO:tasks.workunit.client.0.smithi060.stdout:checking for a working dd... /usr/bin/dd 2024-03-04T12:27:43.910 INFO:tasks.workunit.client.0.smithi060.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-03-04T12:27:43.948 INFO:tasks.workunit.client.0.smithi060.stdout:checking for mt... no 2024-03-04T12:27:43.962 INFO:tasks.workunit.client.0.smithi060.stdout:checking if : is a manifest tool... no 2024-03-04T12:27:44.058 INFO:tasks.workunit.client.0.smithi060.stdout:checking how to run the C preprocessor... gcc -E 2024-03-04T12:27:44.304 INFO:tasks.workunit.client.0.smithi060.stdout:checking for ANSI C header files... yes 2024-03-04T12:27:44.346 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/types.h... yes 2024-03-04T12:27:44.396 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/stat.h... yes 2024-03-04T12:27:44.443 INFO:tasks.workunit.client.0.smithi060.stdout:checking for stdlib.h... yes 2024-03-04T12:27:44.490 INFO:tasks.workunit.client.0.smithi060.stdout:checking for string.h... yes 2024-03-04T12:27:44.549 INFO:tasks.workunit.client.0.smithi060.stdout:checking for memory.h... yes 2024-03-04T12:27:44.599 INFO:tasks.workunit.client.0.smithi060.stdout:checking for strings.h... yes 2024-03-04T12:27:44.654 INFO:tasks.workunit.client.0.smithi060.stdout:checking for inttypes.h... yes 2024-03-04T12:27:44.702 INFO:tasks.workunit.client.0.smithi060.stdout:checking for stdint.h... yes 2024-03-04T12:27:44.756 INFO:tasks.workunit.client.0.smithi060.stdout:checking for unistd.h... yes 2024-03-04T12:27:44.815 INFO:tasks.workunit.client.0.smithi060.stdout:checking for dlfcn.h... yes 2024-03-04T12:27:44.820 INFO:tasks.workunit.client.0.smithi060.stdout:checking for objdir... .libs 2024-03-04T12:27:44.968 INFO:tasks.workunit.client.0.smithi060.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-03-04T12:27:44.968 INFO:tasks.workunit.client.0.smithi060.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-03-04T12:27:45.013 INFO:tasks.workunit.client.0.smithi060.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-03-04T12:27:45.077 INFO:tasks.workunit.client.0.smithi060.stdout:checking if gcc static flag -static works... no 2024-03-04T12:27:45.144 INFO:tasks.workunit.client.0.smithi060.stdout:checking if gcc supports -c -o file.o... yes 2024-03-04T12:27:45.144 INFO:tasks.workunit.client.0.smithi060.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-03-04T12:27:45.150 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-03-04T12:27:45.211 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether -lc should be explicitly linked in... no 2024-03-04T12:27:45.305 INFO:tasks.workunit.client.0.smithi060.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-03-04T12:27:45.305 INFO:tasks.workunit.client.0.smithi060.stdout:checking how to hardcode library paths into programs... immediate 2024-03-04T12:27:45.306 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether stripping libraries is possible... yes 2024-03-04T12:27:45.306 INFO:tasks.workunit.client.0.smithi060.stdout:checking if libtool supports shared libraries... yes 2024-03-04T12:27:45.306 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether to build shared libraries... yes 2024-03-04T12:27:45.306 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether to build static libraries... yes 2024-03-04T12:27:45.312 INFO:tasks.workunit.client.0.smithi060.stdout:checking for gcc... (cached) gcc 2024-03-04T12:27:45.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:45 smithi086 ceph-mon[36851]: pgmap v59: 129 pgs: 129 active+clean; 63 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 658 op/s 2024-03-04T12:27:45.406 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:45 smithi063 ceph-mon[36717]: pgmap v59: 129 pgs: 129 active+clean; 63 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 658 op/s 2024-03-04T12:27:45.407 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-04T12:27:45.407 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-04T12:27:45.407 INFO:tasks.workunit.client.0.smithi060.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-04T12:27:45.407 INFO:tasks.workunit.client.0.smithi060.stdout:checking for gmake... /usr/bin/gmake 2024-03-04T12:27:45.407 INFO:tasks.workunit.client.0.smithi060.stderr:./configure: line 11968: test: $(SHELL): binary operator expected 2024-03-04T12:27:45.408 INFO:tasks.workunit.client.0.smithi060.stdout:checking for tar... /bin/tar 2024-03-04T12:27:45.409 INFO:tasks.workunit.client.0.smithi060.stdout:checking for gzip... /bin/gzip 2024-03-04T12:27:45.409 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether gcc -MM is supported... yes 2024-03-04T12:27:45.409 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sort... /bin/sort 2024-03-04T12:27:45.409 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether ln -s works... yes 2024-03-04T12:27:45.409 INFO:tasks.workunit.client.0.smithi060.stdout:checking for rpm... /bin/rpm 2024-03-04T12:27:45.412 INFO:tasks.workunit.client.0.smithi060.stdout:checking for rpmbuild... no 2024-03-04T12:27:45.413 INFO:tasks.workunit.client.0.smithi060.stdout:checking for ANSI C header files... (cached) yes 2024-03-04T12:27:45.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:45 smithi060 ceph-mon[32745]: pgmap v59: 129 pgs: 129 active+clean; 63 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 658 op/s 2024-03-04T12:27:45.456 INFO:tasks.workunit.client.0.smithi060.stdout:checking assert.h usability... yes 2024-03-04T12:27:45.471 INFO:tasks.workunit.client.0.smithi060.stdout:checking assert.h presence... yes 2024-03-04T12:27:45.472 INFO:tasks.workunit.client.0.smithi060.stdout:checking for assert.h... yes 2024-03-04T12:27:45.529 INFO:tasks.workunit.client.0.smithi060.stdout:checking bstring.h usability... no 2024-03-04T12:27:45.555 INFO:tasks.workunit.client.0.smithi060.stdout:checking bstring.h presence... no 2024-03-04T12:27:45.555 INFO:tasks.workunit.client.0.smithi060.stdout:checking for bstring.h... no 2024-03-04T12:27:45.613 INFO:tasks.workunit.client.0.smithi060.stdout:checking libgen.h usability... yes 2024-03-04T12:27:45.635 INFO:tasks.workunit.client.0.smithi060.stdout:checking libgen.h presence... yes 2024-03-04T12:27:45.635 INFO:tasks.workunit.client.0.smithi060.stdout:checking for libgen.h... yes 2024-03-04T12:27:45.695 INFO:tasks.workunit.client.0.smithi060.stdout:checking dirent.h usability... yes 2024-03-04T12:27:45.711 INFO:tasks.workunit.client.0.smithi060.stdout:checking dirent.h presence... yes 2024-03-04T12:27:45.712 INFO:tasks.workunit.client.0.smithi060.stdout:checking for dirent.h... yes 2024-03-04T12:27:45.765 INFO:tasks.workunit.client.0.smithi060.stdout:checking errno.h usability... yes 2024-03-04T12:27:45.781 INFO:tasks.workunit.client.0.smithi060.stdout:checking errno.h presence... yes 2024-03-04T12:27:45.781 INFO:tasks.workunit.client.0.smithi060.stdout:checking for errno.h... yes 2024-03-04T12:27:45.844 INFO:tasks.workunit.client.0.smithi060.stdout:checking malloc.h usability... yes 2024-03-04T12:27:45.861 INFO:tasks.workunit.client.0.smithi060.stdout:checking malloc.h presence... yes 2024-03-04T12:27:45.861 INFO:tasks.workunit.client.0.smithi060.stdout:checking for malloc.h... yes 2024-03-04T12:27:45.918 INFO:tasks.workunit.client.0.smithi060.stdout:checking uuid.h usability... no 2024-03-04T12:27:45.944 INFO:tasks.workunit.client.0.smithi060.stdout:checking uuid.h presence... no 2024-03-04T12:27:45.944 INFO:tasks.workunit.client.0.smithi060.stdout:checking for uuid.h... no 2024-03-04T12:27:46.010 INFO:tasks.workunit.client.0.smithi060.stdout:checking uuid/uuid.h usability... yes 2024-03-04T12:27:46.026 INFO:tasks.workunit.client.0.smithi060.stdout:checking uuid/uuid.h presence... yes 2024-03-04T12:27:46.026 INFO:tasks.workunit.client.0.smithi060.stdout:checking for uuid/uuid.h... yes 2024-03-04T12:27:46.086 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/uuid.h usability... no 2024-03-04T12:27:46.120 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/uuid.h presence... no 2024-03-04T12:27:46.120 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/uuid.h... no 2024-03-04T12:27:46.178 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/file.h usability... yes 2024-03-04T12:27:46.194 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/file.h presence... yes 2024-03-04T12:27:46.194 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/file.h... yes 2024-03-04T12:27:46.375 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/fcntl.h usability... yes 2024-03-04T12:27:46.392 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/fcntl.h presence... yes 2024-03-04T12:27:46.392 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/fcntl.h... yes 2024-03-04T12:27:46.571 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/syssgi.h usability... no 2024-03-04T12:27:46.598 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/syssgi.h presence... no 2024-03-04T12:27:46.598 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/syssgi.h... no 2024-03-04T12:27:46.808 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/param.h usability... yes 2024-03-04T12:27:46.827 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/param.h presence... yes 2024-03-04T12:27:46.827 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/param.h... yes 2024-03-04T12:27:46.845 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/stat.h... (cached) yes 2024-03-04T12:27:46.889 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/statvfs.h usability... yes 2024-03-04T12:27:46.905 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/statvfs.h presence... yes 2024-03-04T12:27:46.905 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/statvfs.h... yes 2024-03-04T12:27:46.963 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/time.h usability... yes 2024-03-04T12:27:46.980 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/time.h presence... yes 2024-03-04T12:27:46.980 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/time.h... yes 2024-03-04T12:27:47.035 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/ioctl.h usability... yes 2024-03-04T12:27:47.051 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/ioctl.h presence... yes 2024-03-04T12:27:47.052 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/ioctl.h... yes 2024-03-04T12:27:47.114 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/wait.h usability... yes 2024-03-04T12:27:47.132 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/wait.h presence... yes 2024-03-04T12:27:47.132 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/wait.h... yes 2024-03-04T12:27:47.147 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/types.h... (cached) yes 2024-03-04T12:27:47.151 INFO:tasks.workunit.client.0.smithi060.stdout:checking for strings.h... (cached) yes 2024-03-04T12:27:47.196 INFO:tasks.workunit.client.0.smithi060.stdout:checking err.h usability... yes 2024-03-04T12:27:47.211 INFO:tasks.workunit.client.0.smithi060.stdout:checking err.h presence... yes 2024-03-04T12:27:47.211 INFO:tasks.workunit.client.0.smithi060.stdout:checking for err.h... yes 2024-03-04T12:27:47.265 INFO:tasks.workunit.client.0.smithi060.stdout:checking linux/falloc.h usability... yes 2024-03-04T12:27:47.279 INFO:tasks.workunit.client.0.smithi060.stdout:checking linux/falloc.h presence... yes 2024-03-04T12:27:47.279 INFO:tasks.workunit.client.0.smithi060.stdout:checking for linux/falloc.h... yes 2024-03-04T12:27:47.337 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-03-04T12:27:47.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:47 smithi086 ceph-mon[36851]: pgmap v60: 129 pgs: 129 active+clean; 66 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 734 op/s 2024-03-04T12:27:47.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:47 smithi063 ceph-mon[36717]: pgmap v60: 129 pgs: 129 active+clean; 66 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 734 op/s 2024-03-04T12:27:47.363 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-03-04T12:27:47.363 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/fs/xfs_fsops.h... no 2024-03-04T12:27:47.415 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/fs/xfs_itable.h usability... no 2024-03-04T12:27:47.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:47 smithi060 ceph-mon[32745]: pgmap v60: 129 pgs: 129 active+clean; 66 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 734 op/s 2024-03-04T12:27:47.441 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/fs/xfs_itable.h presence... no 2024-03-04T12:27:47.442 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/fs/xfs_itable.h... no 2024-03-04T12:27:47.495 INFO:tasks.workunit.client.0.smithi060.stdout:checking xfs/platform_defs.h usability... no 2024-03-04T12:27:47.524 INFO:tasks.workunit.client.0.smithi060.stdout:checking xfs/platform_defs.h presence... no 2024-03-04T12:27:47.524 INFO:tasks.workunit.client.0.smithi060.stdout:checking for xfs/platform_defs.h... no 2024-03-04T12:27:47.583 INFO:tasks.workunit.client.0.smithi060.stdout:checking btrfs/ioctl.h usability... no 2024-03-04T12:27:47.615 INFO:tasks.workunit.client.0.smithi060.stdout:checking btrfs/ioctl.h presence... no 2024-03-04T12:27:47.615 INFO:tasks.workunit.client.0.smithi060.stdout:checking for btrfs/ioctl.h... no 2024-03-04T12:27:47.674 INFO:tasks.workunit.client.0.smithi060.stdout:checking cifs/ioctl.h usability... no 2024-03-04T12:27:47.705 INFO:tasks.workunit.client.0.smithi060.stdout:checking cifs/ioctl.h presence... no 2024-03-04T12:27:47.705 INFO:tasks.workunit.client.0.smithi060.stdout:checking for cifs/ioctl.h... no 2024-03-04T12:27:47.755 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/mman.h usability... yes 2024-03-04T12:27:47.770 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/mman.h presence... yes 2024-03-04T12:27:47.770 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/mman.h... yes 2024-03-04T12:27:47.816 INFO:tasks.workunit.client.0.smithi060.stdout:checking for xfs/xfs_log_format.h... no 2024-03-04T12:27:47.879 INFO:tasks.workunit.client.0.smithi060.stdout:checking for uuid_compare... no 2024-03-04T12:27:47.926 INFO:tasks.workunit.client.0.smithi060.stdout:checking for uuid_compare in -luuid... yes 2024-03-04T12:27:47.998 INFO:tasks.workunit.client.0.smithi060.stdout:checking for xfs/xfs.h... yes 2024-03-04T12:27:48.035 INFO:tasks.workunit.client.0.smithi060.stdout:checking for xfs/libxfs.h... no 2024-03-04T12:27:48.074 INFO:tasks.workunit.client.0.smithi060.stdout:checking whether xlog_assign_lsn is declared... no 2024-03-04T12:27:48.119 INFO:tasks.workunit.client.0.smithi060.stdout:checking for xfs/xqm.h... yes 2024-03-04T12:27:48.180 INFO:tasks.workunit.client.0.smithi060.stdout:checking xfsctl from xfs/xfs.h... ok 2024-03-04T12:27:48.224 INFO:tasks.workunit.client.0.smithi060.stdout:checking xfs/handle.h usability... yes 2024-03-04T12:27:48.241 INFO:tasks.workunit.client.0.smithi060.stdout:checking xfs/handle.h presence... yes 2024-03-04T12:27:48.241 INFO:tasks.workunit.client.0.smithi060.stdout:checking for xfs/handle.h... yes 2024-03-04T12:27:48.298 INFO:tasks.workunit.client.0.smithi060.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-03-04T12:27:48.359 INFO:tasks.workunit.client.0.smithi060.stdout:checking attr/attributes.h usability... yes 2024-03-04T12:27:48.374 INFO:tasks.workunit.client.0.smithi060.stdout:checking attr/attributes.h presence... yes 2024-03-04T12:27:48.375 INFO:tasks.workunit.client.0.smithi060.stdout:checking for attr/attributes.h... yes 2024-03-04T12:27:48.437 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/attributes.h usability... no 2024-03-04T12:27:48.464 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/attributes.h presence... no 2024-03-04T12:27:48.464 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/attributes.h... no 2024-03-04T12:27:48.511 INFO:tasks.workunit.client.0.smithi060.stdout:checking for attr_list in -lattr... yes 2024-03-04T12:27:48.559 INFO:tasks.workunit.client.0.smithi060.stdout:checking for attr_set in -lattr... yes 2024-03-04T12:27:48.599 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/acl.h usability... yes 2024-03-04T12:27:48.616 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/acl.h presence... yes 2024-03-04T12:27:48.616 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/acl.h... yes 2024-03-04T12:27:48.659 INFO:tasks.workunit.client.0.smithi060.stdout:checking acl/libacl.h usability... yes 2024-03-04T12:27:48.676 INFO:tasks.workunit.client.0.smithi060.stdout:checking acl/libacl.h presence... yes 2024-03-04T12:27:48.676 INFO:tasks.workunit.client.0.smithi060.stdout:checking for acl/libacl.h... yes 2024-03-04T12:27:48.727 INFO:tasks.workunit.client.0.smithi060.stdout:checking for acl_init in -lacl... yes 2024-03-04T12:27:48.773 INFO:tasks.workunit.client.0.smithi060.stdout:checking gdbm-ndbm.h usability... no 2024-03-04T12:27:48.800 INFO:tasks.workunit.client.0.smithi060.stdout:checking gdbm-ndbm.h presence... no 2024-03-04T12:27:48.800 INFO:tasks.workunit.client.0.smithi060.stdout:checking for gdbm-ndbm.h... no 2024-03-04T12:27:48.842 INFO:tasks.workunit.client.0.smithi060.stdout:checking gdbm/ndbm.h usability... no 2024-03-04T12:27:48.870 INFO:tasks.workunit.client.0.smithi060.stdout:checking gdbm/ndbm.h presence... no 2024-03-04T12:27:48.870 INFO:tasks.workunit.client.0.smithi060.stdout:checking for gdbm/ndbm.h... no 2024-03-04T12:27:48.911 INFO:tasks.workunit.client.0.smithi060.stdout:checking ndbm.h usability... no 2024-03-04T12:27:48.943 INFO:tasks.workunit.client.0.smithi060.stdout:checking ndbm.h presence... no 2024-03-04T12:27:48.943 INFO:tasks.workunit.client.0.smithi060.stdout:checking for ndbm.h... no 2024-03-04T12:27:48.985 INFO:tasks.workunit.client.0.smithi060.stdout:checking libaio.h usability... yes 2024-03-04T12:27:49.002 INFO:tasks.workunit.client.0.smithi060.stdout:checking libaio.h presence... yes 2024-03-04T12:27:49.003 INFO:tasks.workunit.client.0.smithi060.stdout:checking for libaio.h... yes 2024-03-04T12:27:49.063 INFO:tasks.workunit.client.0.smithi060.stdout:checking for dm_make_handle in -ldm... no 2024-03-04T12:27:49.063 INFO:tasks.workunit.client.0.smithi060.stdout: 2024-03-04T12:27:49.063 INFO:tasks.workunit.client.0.smithi060.stdout:WARNING: could not find a valid DMAPI base library. 2024-03-04T12:27:49.063 INFO:tasks.workunit.client.0.smithi060.stdout:If you want DMAPI support please install the data migration 2024-03-04T12:27:49.063 INFO:tasks.workunit.client.0.smithi060.stdout:API (dmapi) library package. Alternatively, run "make install" 2024-03-04T12:27:49.063 INFO:tasks.workunit.client.0.smithi060.stdout:from the dmapi source. 2024-03-04T12:27:49.063 INFO:tasks.workunit.client.0.smithi060.stdout: 2024-03-04T12:27:49.104 INFO:tasks.workunit.client.0.smithi060.stdout:checking linux/fiemap.h usability... yes 2024-03-04T12:27:49.119 INFO:tasks.workunit.client.0.smithi060.stdout:checking linux/fiemap.h presence... yes 2024-03-04T12:27:49.120 INFO:tasks.workunit.client.0.smithi060.stdout:checking for linux/fiemap.h... yes 2024-03-04T12:27:49.172 INFO:tasks.workunit.client.0.smithi060.stdout:checking for fallocate... yes 2024-03-04T12:27:49.227 INFO:tasks.workunit.client.0.smithi060.stdout:checking for open_by_handle_at... yes 2024-03-04T12:27:49.277 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/prctl.h usability... yes 2024-03-04T12:27:49.293 INFO:tasks.workunit.client.0.smithi060.stdout:checking sys/prctl.h presence... yes 2024-03-04T12:27:49.293 INFO:tasks.workunit.client.0.smithi060.stdout:checking for sys/prctl.h... yes 2024-03-04T12:27:49.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:49 smithi086 ceph-mon[36851]: pgmap v61: 129 pgs: 129 active+clean; 66 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 288 op/s 2024-03-04T12:27:49.349 INFO:tasks.workunit.client.0.smithi060.stdout:checking linux/fs.h usability... yes 2024-03-04T12:27:49.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:49 smithi063 ceph-mon[36717]: pgmap v61: 129 pgs: 129 active+clean; 66 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 288 op/s 2024-03-04T12:27:49.364 INFO:tasks.workunit.client.0.smithi060.stdout:checking linux/fs.h presence... yes 2024-03-04T12:27:49.364 INFO:tasks.workunit.client.0.smithi060.stdout:checking for linux/fs.h... yes 2024-03-04T12:27:49.413 INFO:tasks.workunit.client.0.smithi060.stdout:checking btrfsutil.h usability... no 2024-03-04T12:27:49.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:49 smithi060 ceph-mon[32745]: pgmap v61: 129 pgs: 129 active+clean; 66 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 288 op/s 2024-03-04T12:27:49.439 INFO:tasks.workunit.client.0.smithi060.stdout:checking btrfsutil.h presence... no 2024-03-04T12:27:49.439 INFO:tasks.workunit.client.0.smithi060.stdout:checking for btrfsutil.h... no 2024-03-04T12:27:49.595 INFO:tasks.workunit.client.0.smithi060.stdout:checking for copy_file_range... yes 2024-03-04T12:27:49.653 INFO:tasks.workunit.client.0.smithi060.stdout:checking for renameat2... yes 2024-03-04T12:27:49.677 INFO:tasks.workunit.client.0.smithi060.stdout:configure: creating ./config.status 2024-03-04T12:27:50.029 INFO:tasks.workunit.client.0.smithi060.stdout:config.status: creating include/builddefs 2024-03-04T12:27:50.040 INFO:tasks.workunit.client.0.smithi060.stdout:config.status: creating include/config.h 2024-03-04T12:27:50.049 INFO:tasks.workunit.client.0.smithi060.stdout:config.status: executing libtool commands 2024-03-04T12:27:50.141 INFO:tasks.workunit.client.0.smithi060.stdout:Building include 2024-03-04T12:27:50.141 INFO:tasks.workunit.client.0.smithi060.stdout:Building lib 2024-03-04T12:27:50.141 INFO:tasks.workunit.client.0.smithi060.stdout:Building tests 2024-03-04T12:27:50.149 INFO:tasks.workunit.client.0.smithi060.stdout: [LTDEP] 2024-03-04T12:27:50.331 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] dataascii.lo 2024-03-04T12:27:50.332 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] databin.lo 2024-03-04T12:27:50.332 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] datapid.lo 2024-03-04T12:27:50.332 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] file_lock.lo 2024-03-04T12:27:50.437 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] forker.lo 2024-03-04T12:27:50.454 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] pattern.lo 2024-03-04T12:27:50.482 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] open_flags.lo 2024-03-04T12:27:50.487 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] random_range.lo 2024-03-04T12:27:50.575 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] string_to_tokens.lo 2024-03-04T12:27:50.593 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] str_to_bytes.lo 2024-03-04T12:27:50.631 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] tlibio.lo 2024-03-04T12:27:50.684 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] write_log.lo 2024-03-04T12:27:50.737 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] random.lo 2024-03-04T12:27:50.926 INFO:tasks.workunit.client.0.smithi060.stdout: [LD] 2024-03-04T12:27:51.038 INFO:tasks.workunit.client.0.smithi060.stdout:Building ltp 2024-03-04T12:27:51.038 INFO:tasks.workunit.client.0.smithi060.stdout:Building src 2024-03-04T12:27:51.038 INFO:tasks.workunit.client.0.smithi060.stdout:Building m4 2024-03-04T12:27:51.039 INFO:tasks.workunit.client.0.smithi060.stdout:Building common 2024-03-04T12:27:51.046 INFO:tasks.workunit.client.0.smithi060.stdout: [DEP] 2024-03-04T12:27:51.060 INFO:tasks.workunit.client.0.smithi060.stdout: [DEP] 2024-03-04T12:27:51.255 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] doio 2024-03-04T12:27:51.255 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] fsstress 2024-03-04T12:27:51.255 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] fsx 2024-03-04T12:27:51.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:51 smithi086 ceph-mon[36851]: pgmap v62: 129 pgs: 129 active+clean; 68 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 323 op/s 2024-03-04T12:27:51.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:51 smithi063 ceph-mon[36717]: pgmap v62: 129 pgs: 129 active+clean; 68 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 323 op/s 2024-03-04T12:27:51.402 INFO:tasks.workunit.client.0.smithi060.stderr:fsstress.c: In function 'attr_list_path': 2024-03-04T12:27:51.402 INFO:tasks.workunit.client.0.smithi060.stderr:fsstress.c:859:9: warning: 'attr_list' is deprecated: Use listxattr or llistxattr instead [-Wdeprecated-declarations] 2024-03-04T12:27:51.403 INFO:tasks.workunit.client.0.smithi060.stderr: 859 | rval = attr_list(name->path, buffer, buffersize, flags, cursor); 2024-03-04T12:27:51.403 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~ 2024-03-04T12:27:51.403 INFO:tasks.workunit.client.0.smithi060.stderr:In file included from ../src/global.h:25, 2024-03-04T12:27:51.403 INFO:tasks.workunit.client.0.smithi060.stderr: from fsstress.c:12: 2024-03-04T12:27:51.403 INFO:tasks.workunit.client.0.smithi060.stderr:/usr/include/attr/attributes.h:165:12: note: declared here 2024-03-04T12:27:51.403 INFO:tasks.workunit.client.0.smithi060.stderr: 165 | extern int attr_list(const char *__path, char *__buffer, const int __buffersize, 2024-03-04T12:27:51.403 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~~~~~ 2024-03-04T12:27:51.403 INFO:tasks.workunit.client.0.smithi060.stderr:fsstress.c: In function 'attr_remove_path': 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr:fsstress.c:878:9: warning: 'attr_remove' is deprecated: Use removexattr or lremovexattr instead [-Wdeprecated-declarations] 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr: 878 | rval = attr_remove(name->path, attrname, flags); 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~ 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr:In file included from ../src/global.h:25, 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr: from fsstress.c:12: 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr:/usr/include/attr/attributes.h:152:12: note: declared here 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr: 152 | extern int attr_remove (const char *__path, const char *__attrname, 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~~~~~~~ 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr:fsstress.c: In function 'attr_set_path': 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr:fsstress.c:898:9: warning: 'attr_set' is deprecated: Use setxattr or lsetxattr instead [-Wdeprecated-declarations] 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr: 898 | rval = attr_set(name->path, attrname, attrvalue, valuelength, flags); 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~ 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr:In file included from ../src/global.h:25, 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr: from fsstress.c:12: 2024-03-04T12:27:51.404 INFO:tasks.workunit.client.0.smithi060.stderr:/usr/include/attr/attributes.h:139:12: note: declared here 2024-03-04T12:27:51.405 INFO:tasks.workunit.client.0.smithi060.stderr: 139 | extern int attr_set (const char *__path, const char *__attrname, 2024-03-04T12:27:51.405 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~~~~ 2024-03-04T12:27:51.414 INFO:tasks.workunit.client.0.smithi060.stderr:fsstress.c: In function 'allocsp_f': 2024-03-04T12:27:51.414 INFO:tasks.workunit.client.0.smithi060.stderr:fsstress.c:2047:32: error: 'XFS_IOC_ALLOCSP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-03-04T12:27:51.414 INFO:tasks.workunit.client.0.smithi060.stderr: 2047 | e = xfsctl(f.path, fd, XFS_IOC_ALLOCSP64, &fl) < 0 ? errno : 0; 2024-03-04T12:27:51.414 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~~~~~~~~~~~~~ 2024-03-04T12:27:51.414 INFO:tasks.workunit.client.0.smithi060.stderr: | XFS_IOC_RESVSP64 2024-03-04T12:27:51.414 INFO:tasks.workunit.client.0.smithi060.stderr:fsstress.c:2047:32: note: each undeclared identifier is reported only once for each function it appears in 2024-03-04T12:27:51.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:51 smithi060 ceph-mon[32745]: pgmap v62: 129 pgs: 129 active+clean; 68 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 323 op/s 2024-03-04T12:27:51.431 INFO:tasks.workunit.client.0.smithi060.stderr:fsstress.c: In function 'freesp_f': 2024-03-04T12:27:51.432 INFO:tasks.workunit.client.0.smithi060.stderr:fsstress.c:3640:32: error: 'XFS_IOC_FREESP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-03-04T12:27:51.432 INFO:tasks.workunit.client.0.smithi060.stderr: 3640 | e = xfsctl(f.path, fd, XFS_IOC_FREESP64, &fl) < 0 ? errno : 0; 2024-03-04T12:27:51.432 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~~~~~~~~~~~~ 2024-03-04T12:27:51.432 INFO:tasks.workunit.client.0.smithi060.stderr: | XFS_IOC_RESVSP64 2024-03-04T12:27:51.465 INFO:tasks.workunit.client.0.smithi060.stderr:gmake[3]: *** [Makefile:47: fsstress] Error 1 2024-03-04T12:27:51.465 INFO:tasks.workunit.client.0.smithi060.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-03-04T12:27:51.835 INFO:tasks.workunit.client.0.smithi060.stderr:doio.c: In function 'parse_cmdline': 2024-03-04T12:27:51.835 INFO:tasks.workunit.client.0.smithi060.stderr:doio.c:3113:33: warning: '%s' directive argument is null [-Wformat-overflow=] 2024-03-04T12:27:51.835 INFO:tasks.workunit.client.0.smithi060.stderr: 3113 | fprintf(stderr, 2024-03-04T12:27:51.835 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~~~~~~~~~~~ 2024-03-04T12:27:51.835 INFO:tasks.workunit.client.0.smithi060.stderr: 3114 | "%s%s: Illegal -C arg (%s). Must be one of: ", 2024-03-04T12:27:51.835 INFO:tasks.workunit.client.0.smithi060.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-04T12:27:51.835 INFO:tasks.workunit.client.0.smithi060.stderr: 3115 | Prog, TagName, tok); 2024-03-04T12:27:51.835 INFO:tasks.workunit.client.0.smithi060.stderr: | ~~~~~~~~~~~~~~~~~~~ 2024-03-04T12:27:51.850 INFO:tasks.workunit.client.0.smithi060.stderr:fsx.c: In function 'main': 2024-03-04T12:27:51.850 INFO:tasks.workunit.client.0.smithi060.stderr:fsx.c:2662:33: warning: 'strncpy' specified bound 4096 equals destination size [-Wstringop-truncation] 2024-03-04T12:27:51.850 INFO:tasks.workunit.client.0.smithi060.stderr: 2662 | strncpy(opsfile, optarg, sizeof(opsfile)); 2024-03-04T12:27:51.850 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-04T12:27:51.850 INFO:tasks.workunit.client.0.smithi060.stderr:fsx.c:2632:25: warning: 'strncpy' specified bound 1024 equals destination size [-Wstringop-truncation] 2024-03-04T12:27:51.850 INFO:tasks.workunit.client.0.smithi060.stderr: 2632 | strncpy(dname, optarg, sizeof(dname)); 2024-03-04T12:27:51.850 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-04T12:27:51.950 INFO:tasks.workunit.client.0.smithi060.stderr:gmake[2]: *** [include/buildrules:30: ltp] Error 2 2024-03-04T12:27:51.950 INFO:tasks.workunit.client.0.smithi060.stderr:gmake[2]: *** Waiting for unfinished jobs.... 2024-03-04T12:27:52.540 INFO:tasks.workunit.client.0.smithi060.stdout: [DEP] 2024-03-04T12:27:52.541 INFO:tasks.workunit.client.0.smithi060.stdout: [DEP] 2024-03-04T12:27:52.543 INFO:tasks.workunit.client.0.smithi060.stdout: [DEP] 2024-03-04T12:27:52.759 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] dirstress 2024-03-04T12:27:52.761 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] fill 2024-03-04T12:27:52.761 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] fill2 2024-03-04T12:27:52.762 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] getpagesize 2024-03-04T12:27:52.902 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] holes 2024-03-04T12:27:52.907 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] lstat64 2024-03-04T12:27:53.054 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] nametest 2024-03-04T12:27:53.173 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] permname 2024-03-04T12:27:53.306 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] randholes 2024-03-04T12:27:53.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:53 smithi086 ceph-mon[36851]: pgmap v63: 129 pgs: 129 active+clean; 68 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 543 KiB/s wr, 111 op/s 2024-03-04T12:27:53.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:53 smithi063 ceph-mon[36717]: pgmap v63: 129 pgs: 129 active+clean; 68 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 543 KiB/s wr, 111 op/s 2024-03-04T12:27:53.372 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] runas 2024-03-04T12:27:53.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:53 smithi060 ceph-mon[32745]: pgmap v63: 129 pgs: 129 active+clean; 68 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 543 KiB/s wr, 111 op/s 2024-03-04T12:27:53.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-04T12:27:53.430 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-04T12:27:53.487 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] truncfile 2024-03-04T12:27:53.496 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] usemem 2024-03-04T12:27:53.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.562+0000 7f126cfe1640 1 -- 172.21.15.60:0/3424768476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268100e00 msgr2=0x7f1268104e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:53.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.562+0000 7f126cfe1640 1 --2- 172.21.15.60:0/3424768476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268100e00 0x7f1268104e40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1250009900 tx=0x7f125002bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:53.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.562+0000 7f126cfe1640 1 -- 172.21.15.60:0/3424768476 shutdown_connections 2024-03-04T12:27:53.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.562+0000 7f126cfe1640 1 --2- 172.21.15.60:0/3424768476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1268105520 0x7f126810d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.562+0000 7f126cfe1640 1 --2- 172.21.15.60:0/3424768476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268100e00 0x7f1268104e40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.562+0000 7f126cfe1640 1 --2- 172.21.15.60:0/3424768476 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1268100450 0x7f1268100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.562+0000 7f126cfe1640 1 -- 172.21.15.60:0/3424768476 >> 172.21.15.60:0/3424768476 conn(0x7f1268077380 msgr2=0x7f1268077790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:53.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.562+0000 7f126cfe1640 1 -- 172.21.15.60:0/3424768476 shutdown_connections 2024-03-04T12:27:53.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.562+0000 7f126cfe1640 1 -- 172.21.15.60:0/3424768476 wait complete. 2024-03-04T12:27:53.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.563+0000 7f126cfe1640 1 Processor -- start 2024-03-04T12:27:53.572 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.563+0000 7f126cfe1640 1 -- start start 2024-03-04T12:27:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.563+0000 7f126cfe1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1268100450 0x7f126819d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.563+0000 7f126cfe1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268105520 0x7f126819dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f126cfe1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1268197910 0x7f1268197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f126cfe1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12681116f0 con 0x7f1268197910 2024-03-04T12:27:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f126cfe1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1268111570 con 0x7f1268100450 2024-03-04T12:27:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f126cfe1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1268111870 con 0x7f1268105520 2024-03-04T12:27:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f125ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268105520 0x7f126819dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f125ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268105520 0x7f126819dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:59658/0 (socket says 172.21.15.60:59658) 2024-03-04T12:27:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f125ffff640 1 -- 172.21.15.60:0/1015044808 learned_addr learned my addr 172.21.15.60:0/1015044808 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f1266d9d640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1268197910 0x7f1268197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f126659c640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1268100450 0x7f126819d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.573 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f125ffff640 1 -- 172.21.15.60:0/1015044808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1268100450 msgr2=0x7f126819d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:53.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f125ffff640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1268100450 0x7f126819d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f125ffff640 1 -- 172.21.15.60:0/1015044808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1268197910 msgr2=0x7f1268197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:53.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f125ffff640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1268197910 0x7f1268197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f125ffff640 1 -- 172.21.15.60:0/1015044808 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1268198430 con 0x7f1268105520 2024-03-04T12:27:53.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.564+0000 7f125ffff640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268105520 0x7f126819dca0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1250036040 tx=0x7f125002cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:53.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.565+0000 7f125f7fe640 1 -- 172.21.15.60:0/1015044808 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250041070 con 0x7f1268105520 2024-03-04T12:27:53.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.565+0000 7f125f7fe640 1 -- 172.21.15.60:0/1015044808 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12500054e0 con 0x7f1268105520 2024-03-04T12:27:53.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.565+0000 7f125f7fe640 1 -- 172.21.15.60:0/1015044808 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1250034440 con 0x7f1268105520 2024-03-04T12:27:53.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.565+0000 7f126cfe1640 1 -- 172.21.15.60:0/1015044808 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12681986c0 con 0x7f1268105520 2024-03-04T12:27:53.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.565+0000 7f126cfe1640 1 -- 172.21.15.60:0/1015044808 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12681a4910 con 0x7f1268105520 2024-03-04T12:27:53.575 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] mmapcat 2024-03-04T12:27:53.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.568+0000 7f125f7fe640 1 -- 172.21.15.60:0/1015044808 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f1250044070 con 0x7f1268105520 2024-03-04T12:27:53.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.568+0000 7f126fbb7740 1 Processor -- start 2024-03-04T12:27:53.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.568+0000 7f126fbb7740 1 -- start start 2024-03-04T12:27:53.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f125f7fe640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f12540a1f70 0x7f12540a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126fbb7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e62790 0x55fe53e62b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126fbb7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126fbb7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e6bc90 0x55fe53e6e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126fbb7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe53dc5e00 con 0x55fe53e62790 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126fbb7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe53d6eb10 con 0x55fe53e630b0 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126fbb7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe53dc1ea0 con 0x55fe53e6bc90 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f1266d9d640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f12540a1f70 0x7f12540a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e62790 0x55fe53e62b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126659c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e62790 0x55fe53e62b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52722/0 (socket says 172.21.15.60:52722) 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126659c640 1 -- 172.21.15.60:0/310712353 learned_addr learned my addr 172.21.15.60:0/310712353 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f125ffff640 1 --2- 172.21.15.60:0/310712353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f1266d9d640 1 --2- 172.21.15.60:0/310712353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e6bc90 0x55fe53e6e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f125f7fe640 1 -- 172.21.15.60:0/1015044808 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f12500f0320 con 0x7f1268105520 2024-03-04T12:27:53.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126659c640 1 -- 172.21.15.60:0/310712353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e6bc90 msgr2=0x55fe53e6e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:53.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126659c640 1 --2- 172.21.15.60:0/310712353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e6bc90 0x55fe53e6e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126659c640 1 -- 172.21.15.60:0/310712353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 msgr2=0x55fe53e6b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:53.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126659c640 1 --2- 172.21.15.60:0/310712353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.569+0000 7f126659c640 1 -- 172.21.15.60:0/310712353 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe53ba47d0 con 0x55fe53e62790 2024-03-04T12:27:53.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.570+0000 7f1266d9d640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f12540a1f70 0x7f12540a4430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1260009fd0 tx=0x7f1260009d80 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:53.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.570+0000 7f126659c640 1 --2- 172.21.15.60:0/310712353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e62790 0x55fe53e62b70 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1258010fb0 tx=0x7f125800a740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0cfbce1e9cdd4a server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:53.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.570+0000 7f125cff9640 1 -- 172.21.15.60:0/310712353 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258010260 con 0x55fe53e62790 2024-03-04T12:27:53.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.570+0000 7f125cff9640 1 -- 172.21.15.60:0/310712353 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1258005500 con 0x55fe53e62790 2024-03-04T12:27:53.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.570+0000 7f125cff9640 1 -- 172.21.15.60:0/310712353 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125800f6d0 con 0x55fe53e62790 2024-03-04T12:27:53.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.571+0000 7f126fbb7740 1 -- 172.21.15.60:0/310712353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e62790 msgr2=0x55fe53e62b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:53.582 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.571+0000 7f126fbb7740 1 --2- 172.21.15.60:0/310712353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e62790 0x55fe53e62b70 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f1258010fb0 tx=0x7f125800a740 comp rx=0 tx=0).stop 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.571+0000 7f126fbb7740 1 -- 172.21.15.60:0/310712353 shutdown_connections 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.571+0000 7f126fbb7740 1 --2- 172.21.15.60:0/310712353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e6bc90 0x55fe53e6e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.571+0000 7f126fbb7740 1 --2- 172.21.15.60:0/310712353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.571+0000 7f126fbb7740 1 --2- 172.21.15.60:0/310712353 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e62790 0x55fe53e62b70 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.572+0000 7f126fbb7740 1 -- 172.21.15.60:0/310712353 >> 172.21.15.60:0/310712353 conn(0x55fe53e5cec0 msgr2=0x55fe53e5d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.572+0000 7f126fbb7740 1 -- 172.21.15.60:0/310712353 shutdown_connections 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.572+0000 7f126fbb7740 1 -- 172.21.15.60:0/310712353 wait complete. 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.572+0000 7f126fbb7740 1 -- start start 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.573+0000 7f126fbb7740 1 Processor -- start 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.573+0000 7f126fbb7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.573+0000 7f126fbb7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e630b0 0x55fe53e68730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.573+0000 7f126fbb7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e6bc90 0x55fe53e68c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.583 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:59682/0 (socket says 172.21.15.60:59682) 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126659c640 1 -- 172.21.15.60:0/139393103 learned_addr learned my addr 172.21.15.60:0/139393103 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126fbb7740 1 -- 172.21.15.60:0/139393103 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe53e6ea80 con 0x55fe53e630b0 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126fbb7740 1 -- 172.21.15.60:0/139393103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe53d820a0 con 0x55fe53e6bc90 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126fbb7740 1 -- 172.21.15.60:0/139393103 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe53e6ec00 con 0x55fe53e62790 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f125ffff640 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e630b0 0x55fe53e68730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126fbb7740 10 client.0 _ll_register_callbacks cb 0x55fe53d81a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f1266d9d640 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e6bc90 0x55fe53e68c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126659c640 1 -- 172.21.15.60:0/139393103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e6bc90 msgr2=0x55fe53e68c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126659c640 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e6bc90 0x55fe53e68c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126659c640 1 -- 172.21.15.60:0/139393103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e630b0 msgr2=0x55fe53e68730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126659c640 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e630b0 0x55fe53e68730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126659c640 1 -- 172.21.15.60:0/139393103 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe53ada3f0 con 0x55fe53e62790 2024-03-04T12:27:53.584 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f1266d9d640 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e6bc90 0x55fe53e68c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f126659c640 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6b3a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1258004b30 tx=0x7f12580048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a473609893dde72 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.574+0000 7f124dffb640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.575+0000 7f124dffb640 1 -- 172.21.15.60:0/139393103 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258005230 con 0x55fe53e62790 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.575+0000 7f124dffb640 1 -- 172.21.15.60:0/139393103 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f1258016050 con 0x55fe53e62790 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.575+0000 7f126fbb7740 1 -- 172.21.15.60:0/139393103 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1258005500 con 0x55fe53e62790 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.575+0000 7f124dffb640 1 -- 172.21.15.60:0/139393103 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1258010260 con 0x55fe53e62790 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.575+0000 7f126fbb7740 10 client.25939 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.575+0000 7f126fbb7740 1 -- 172.21.15.60:0/139393103 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55fe53a9a0f0 con 0x55fe53e62790 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.575+0000 7f124dffb640 1 -- 172.21.15.60:0/139393103 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1258005500 con 0x55fe53e62790 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.575+0000 7f124dffb640 1 -- 172.21.15.60:0/139393103 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f125800fae0 con 0x55fe53e62790 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.576+0000 7f126fbb7740 10 client.25939 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.576+0000 7f126fbb7740 10 client.25939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.576+0000 7f126fbb7740 20 client.25939 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:53.585 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.576+0000 7f126fbb7740 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fe53e7b5b0 0x55fe53e7d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.576+0000 7f126fbb7740 4 client.25939 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:53.586 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.576+0000 7f126fbb7740 1 -- 172.21.15.60:0/139393103 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe53dd3340 con 0x55fe53e7b5b0 2024-03-04T12:27:53.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.576+0000 7f125ffff640 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fe53e7b5b0 0x55fe53e7d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f124dffb640 10 client.25939 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:53.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f125ffff640 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fe53e7b5b0 0x55fe53e7d9a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee65f9539813934f server_cookie=35f7c448e310d2c2 in_seq=0 out_seq=0 2024-03-04T12:27:53.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f124dffb640 1 -- 172.21.15.60:0/139393103 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fe53dd3340 con 0x55fe53e7b5b0 2024-03-04T12:27:53.587 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f124dffb640 10 client.25939 handle_command_reply: tid=0 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f126fbb7740 1 client.25939 shutdown 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f126fbb7740 1 -- 172.21.15.60:0/139393103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 msgr2=0x55fe53e6b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f126fbb7740 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6b3a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1258004b30 tx=0x7f12580048f0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f126fbb7740 1 -- 172.21.15.60:0/139393103 shutdown_connections 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f126fbb7740 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fe53e7b5b0 0x55fe53e7d9a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f126fbb7740 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e6bc90 0x55fe53e68c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f126fbb7740 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e630b0 0x55fe53e68730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f126fbb7740 1 --2- 172.21.15.60:0/139393103 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6b3a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f126fbb7740 1 -- 172.21.15.60:0/139393103 >> 172.21.15.60:0/139393103 conn(0x55fe53e5cec0 msgr2=0x55fe53e5ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.577+0000 7f124dffb640 0 client.25939 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.578+0000 7f126fbb7740 1 -- 172.21.15.60:0/139393103 shutdown_connections 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.578+0000 7f126fbb7740 1 -- 172.21.15.60:0/139393103 wait complete. 2024-03-04T12:27:53.588 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.578+0000 7f126fbb7740 20 client.25939 trim_cache size 0 max 16384 2024-03-04T12:27:53.605 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.597+0000 7f126fbb7740 1 Processor -- start 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.597+0000 7f126fbb7740 1 -- start start 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126fbb7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126fbb7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126fbb7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e6bc90 0x55fe53e72e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126fbb7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12681ac340 con 0x55fe53e6bc90 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126fbb7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12681add20 con 0x55fe53e630b0 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126fbb7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12681ac9a0 con 0x55fe53e62790 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f1266d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e6bc90 0x55fe53e72e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:59690/0 (socket says 172.21.15.60:59690) 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126659c640 1 -- 172.21.15.60:0/1462890064 learned_addr learned my addr 172.21.15.60:0/1462890064 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f1266d9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e6bc90 0x55fe53e72e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52738/0 (socket says 172.21.15.60:52738) 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f125ffff640 1 --2- 172.21.15.60:0/1462890064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126659c640 1 -- 172.21.15.60:0/1462890064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 msgr2=0x55fe53e6b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:53.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126659c640 1 --2- 172.21.15.60:0/1462890064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126659c640 1 -- 172.21.15.60:0/1462890064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e6bc90 msgr2=0x55fe53e72e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:53.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126659c640 1 --2- 172.21.15.60:0/1462890064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e6bc90 0x55fe53e72e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.598+0000 7f126659c640 1 -- 172.21.15.60:0/1462890064 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe53e62ea0 con 0x55fe53e62790 2024-03-04T12:27:53.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.599+0000 7f126659c640 1 --2- 172.21.15.60:0/1462890064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6ae60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f12580107b0 tx=0x7f1258005fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45b16d70f04bc3c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:53.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.599+0000 7f124ffff640 1 -- 172.21.15.60:0/1462890064 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1258003690 con 0x55fe53e62790 2024-03-04T12:27:53.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.599+0000 7f124ffff640 1 -- 172.21.15.60:0/1462890064 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f125803a070 con 0x55fe53e62790 2024-03-04T12:27:53.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.599+0000 7f124ffff640 1 -- 172.21.15.60:0/1462890064 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f125800f960 con 0x55fe53e62790 2024-03-04T12:27:53.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 -- 172.21.15.60:0/1462890064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 msgr2=0x55fe53e6ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:53.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 --2- 172.21.15.60:0/1462890064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6ae60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f12580107b0 tx=0x7f1258005fa0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 -- 172.21.15.60:0/1462890064 shutdown_connections 2024-03-04T12:27:53.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 --2- 172.21.15.60:0/1462890064 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e6bc90 0x55fe53e72e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 --2- 172.21.15.60:0/1462890064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 --2- 172.21.15.60:0/1462890064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6ae60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 -- 172.21.15.60:0/1462890064 >> 172.21.15.60:0/1462890064 conn(0x55fe53e5cec0 msgr2=0x55fe53e821c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:53.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 -- 172.21.15.60:0/1462890064 shutdown_connections 2024-03-04T12:27:53.611 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 -- 172.21.15.60:0/1462890064 wait complete. 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 -- start start 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 Processor -- start 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e6bc90 0x55fe53eca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12681ad6c0 con 0x55fe53e6bc90 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12681ad000 con 0x55fe53e630b0 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12681aa5e0 con 0x55fe53e62790 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.600+0000 7f126fbb7740 10 client.0 _ll_register_callbacks cb 0x7f1258010e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f125ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f125ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:44198/0 (socket says 172.21.15.60:44198) 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f125ffff640 1 -- 172.21.15.60:0/1215815631 learned_addr learned my addr 172.21.15.60:0/1215815631 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:53.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f126659c640 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f1266d9d640 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e6bc90 0x55fe53eca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f125ffff640 1 -- 172.21.15.60:0/1215815631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 msgr2=0x55fe53e6f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f125ffff640 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f125ffff640 1 -- 172.21.15.60:0/1215815631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e6bc90 msgr2=0x55fe53eca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f125ffff640 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e6bc90 0x55fe53eca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f125ffff640 1 -- 172.21.15.60:0/1215815631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe53ecaef0 con 0x55fe53e630b0 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f1266d9d640 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e6bc90 0x55fe53eca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f125ffff640 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6f5a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f12500b93f0 tx=0x7f1250004dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aac04065bc92c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f124dffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f124dffb640 1 -- 172.21.15.60:0/1215815631 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12500d4070 con 0x55fe53e630b0 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f124dffb640 1 -- 172.21.15.60:0/1215815631 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f12500b9090 con 0x55fe53e630b0 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f126fbb7740 1 -- 172.21.15.60:0/1215815631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f125803a070 con 0x55fe53e630b0 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f124dffb640 1 -- 172.21.15.60:0/1215815631 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f125004b040 con 0x55fe53e630b0 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f126fbb7740 10 client.25847 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:53.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.601+0000 7f126fbb7740 1 -- 172.21.15.60:0/1215815631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55fe53ed22e0 con 0x55fe53e630b0 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.602+0000 7f124dffb640 1 -- 172.21.15.60:0/1215815631 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f125803a070 con 0x55fe53e630b0 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.602+0000 7f124dffb640 1 -- 172.21.15.60:0/1215815631 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f12500035f0 con 0x55fe53e630b0 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.602+0000 7f126fbb7740 10 client.25847 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.602+0000 7f126fbb7740 10 client.25847 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.602+0000 7f126fbb7740 20 client.25847 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.602+0000 7f126fbb7740 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fe53ed2a20 0x55fe53ed4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.602+0000 7f126fbb7740 4 client.25847 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.602+0000 7f126fbb7740 1 -- 172.21.15.60:0/1215815631 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fe53dcd690 con 0x55fe53ed2a20 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f126659c640 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fe53ed2a20 0x55fe53ed4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f126659c640 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fe53ed2a20 0x55fe53ed4e40 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1acd1e57b5d22a95 server_cookie=f0945f0577871100 in_seq=0 out_seq=0 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f124dffb640 10 client.25847 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f124dffb640 1 -- 172.21.15.60:0/1215815631 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fe53dcd690 con 0x55fe53ed2a20 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f124dffb640 10 client.25847 handle_command_reply: tid=0 2024-03-04T12:27:53.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f126fbb7740 1 client.25847 shutdown 2024-03-04T12:27:53.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f126fbb7740 1 -- 172.21.15.60:0/1215815631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 msgr2=0x55fe53e6f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:53.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f126fbb7740 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6f5a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f12500b93f0 tx=0x7f1250004dc0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f126fbb7740 1 -- 172.21.15.60:0/1215815631 shutdown_connections 2024-03-04T12:27:53.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f126fbb7740 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fe53ed2a20 0x55fe53ed4e40 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f126fbb7740 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fe53e6bc90 0x55fe53eca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f126fbb7740 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fe53e630b0 0x55fe53e6f5a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f126fbb7740 1 --2- 172.21.15.60:0/1215815631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fe53e62790 0x55fe53e6f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.603+0000 7f126fbb7740 1 -- 172.21.15.60:0/1215815631 >> 172.21.15.60:0/1215815631 conn(0x55fe53e5cec0 msgr2=0x55fe53e7d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:53.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f124dffb640 0 client.25847 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:53.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126fbb7740 1 -- 172.21.15.60:0/1215815631 shutdown_connections 2024-03-04T12:27:53.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126fbb7740 1 -- 172.21.15.60:0/1215815631 wait complete. 2024-03-04T12:27:53.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126fbb7740 20 client.25847 trim_cache size 0 max 16384 2024-03-04T12:27:53.616 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:27:53.616 INFO:teuthology.orchestra.run.smithi060.stdout: "status": "no active scrubs running", 2024-03-04T12:27:53.616 INFO:teuthology.orchestra.run.smithi060.stdout: "scrubs": {} 2024-03-04T12:27:53.616 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:27:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126cfe1640 1 -- 172.21.15.60:0/1015044808 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f12540a1f70 msgr2=0x7f12540a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126cfe1640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f12540a1f70 0x7f12540a4430 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1260009fd0 tx=0x7f1260009d80 comp rx=0 tx=0).stop 2024-03-04T12:27:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126cfe1640 1 -- 172.21.15.60:0/1015044808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268105520 msgr2=0x7f126819dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126cfe1640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268105520 0x7f126819dca0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1250036040 tx=0x7f125002cbf0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126cfe1640 1 -- 172.21.15.60:0/1015044808 shutdown_connections 2024-03-04T12:27:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126cfe1640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f12540a1f70 0x7f12540a4430 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126cfe1640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1268197910 0x7f1268197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126cfe1640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268105520 0x7f126819dca0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126cfe1640 1 --2- 172.21.15.60:0/1015044808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1268100450 0x7f126819d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.604+0000 7f126cfe1640 1 -- 172.21.15.60:0/1015044808 >> 172.21.15.60:0/1015044808 conn(0x7f1268077380 msgr2=0x7f12680fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.605+0000 7f126cfe1640 1 -- 172.21.15.60:0/1015044808 shutdown_connections 2024-03-04T12:27:53.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.605+0000 7f126cfe1640 1 -- 172.21.15.60:0/1015044808 wait complete. 2024-03-04T12:27:53.618 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] append_reader 2024-03-04T12:27:53.627 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-04T12:27:53.627 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:27:53.666 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] append_writer 2024-03-04T12:27:53.711 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] dirperf 2024-03-04T12:27:53.751 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] metaperf 2024-03-04T12:27:53.761 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.754+0000 7fe665cea640 1 -- 172.21.15.60:0/787510684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe660102ec0 msgr2=0x7fe66010b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:53.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.754+0000 7fe665cea640 1 --2- 172.21.15.60:0/787510684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe660102ec0 0x7fe66010b3a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe654009900 tx=0x7fe65402bd60 comp rx=0 tx=0).stop 2024-03-04T12:27:53.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.755+0000 7fe665cea640 1 -- 172.21.15.60:0/787510684 shutdown_connections 2024-03-04T12:27:53.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.755+0000 7fe665cea640 1 --2- 172.21.15.60:0/787510684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe66010b8e0 0x7fe66010dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.755+0000 7fe665cea640 1 --2- 172.21.15.60:0/787510684 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe660102ec0 0x7fe66010b3a0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.755+0000 7fe665cea640 1 --2- 172.21.15.60:0/787510684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6601025a0 0x7fe660102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.755+0000 7fe665cea640 1 -- 172.21.15.60:0/787510684 >> 172.21.15.60:0/787510684 conn(0x7fe660077350 msgr2=0x7fe660077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:53.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.755+0000 7fe665cea640 1 -- 172.21.15.60:0/787510684 shutdown_connections 2024-03-04T12:27:53.762 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.755+0000 7fe665cea640 1 -- 172.21.15.60:0/787510684 wait complete. 2024-03-04T12:27:53.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.755+0000 7fe665cea640 1 Processor -- start 2024-03-04T12:27:53.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.755+0000 7fe665cea640 1 -- start start 2024-03-04T12:27:53.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe665cea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6601025a0 0x7fe66019d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe665cea640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe660102ec0 0x7fe66019dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe665cea640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe66010b8e0 0x7fe6601979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe665cea640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6601116f0 con 0x7fe6601025a0 2024-03-04T12:27:53.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe665cea640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe660111570 con 0x7fe660102ec0 2024-03-04T12:27:53.764 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe665cea640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe660111870 con 0x7fe66010b8e0 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe65f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6601025a0 0x7fe66019d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe65f7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6601025a0 0x7fe66019d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52764/0 (socket says 172.21.15.60:52764) 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe65f7fe640 1 -- 172.21.15.60:0/3301490872 learned_addr learned my addr 172.21.15.60:0/3301490872 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe65effd640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe660102ec0 0x7fe66019dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe65ffff640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe66010b8e0 0x7fe6601979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe65effd640 1 -- 172.21.15.60:0/3301490872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe66010b8e0 msgr2=0x7fe6601979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe65effd640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe66010b8e0 0x7fe6601979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe65effd640 1 -- 172.21.15.60:0/3301490872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6601025a0 msgr2=0x7fe66019d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe65effd640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6601025a0 0x7fe66019d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.756+0000 7fe65effd640 1 -- 172.21.15.60:0/3301490872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe660198170 con 0x7fe660102ec0 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.757+0000 7fe65effd640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe660102ec0 0x7fe66019dd30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe654009230 tx=0x7fe65402dc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.757+0000 7fe65cff9640 1 -- 172.21.15.60:0/3301490872 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe654043020 con 0x7fe660102ec0 2024-03-04T12:27:53.765 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.757+0000 7fe65cff9640 1 -- 172.21.15.60:0/3301490872 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6540054e0 con 0x7fe660102ec0 2024-03-04T12:27:53.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.757+0000 7fe65cff9640 1 -- 172.21.15.60:0/3301490872 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe654003c20 con 0x7fe660102ec0 2024-03-04T12:27:53.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.757+0000 7fe665cea640 1 -- 172.21.15.60:0/3301490872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe660198400 con 0x7fe660102ec0 2024-03-04T12:27:53.766 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.757+0000 7fe665cea640 1 -- 172.21.15.60:0/3301490872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe660100120 con 0x7fe660102ec0 2024-03-04T12:27:53.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.758+0000 7fe665cea640 1 -- 172.21.15.60:0/3301490872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe660069d90 con 0x7fe660102ec0 2024-03-04T12:27:53.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.759+0000 7fe65cff9640 1 -- 172.21.15.60:0/3301490872 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7fe654046020 con 0x7fe660102ec0 2024-03-04T12:27:53.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.760+0000 7fe65cff9640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fe6480a1f70 0x7fe6480a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:53.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.760+0000 7fe65f7fe640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fe6480a1f70 0x7fe6480a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:53.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.760+0000 7fe65cff9640 1 -- 172.21.15.60:0/3301490872 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fe6540f02e0 con 0x7fe660102ec0 2024-03-04T12:27:53.767 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.760+0000 7fe65f7fe640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fe6480a1f70 0x7fe6480a4430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe6600ff6a0 tx=0x7fe650005f90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:53.771 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:53.764+0000 7fe65cff9640 1 -- 172.21.15.60:0/3301490872 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe65404d020 con 0x7fe660102ec0 2024-03-04T12:27:53.791 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] devzero 2024-03-04T12:27:53.834 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] feature 2024-03-04T12:27:53.982 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] alloc 2024-03-04T12:27:53.990 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] fault 2024-03-04T12:27:54.064 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] fstest 2024-03-04T12:27:54.069 INFO:tasks.workunit.client.0.smithi060.stdout: [CC] t_access_root 2024-03-04T12:27:54.088 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.082+0000 7fe665cea640 1 -- 172.21.15.60:0/3301490872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe660067920 con 0x7fe660102ec0 2024-03-04T12:27:54.089 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.082+0000 7fe65cff9640 1 -- 172.21.15.60:0/3301490872 <== mon.2 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+11382 (secure 0 0 0) 0x7fe6540dc030 con 0x7fe660102ec0 2024-03-04T12:27:54.089 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:27:54.090 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:27:30.073987+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":69,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_25741":{"gid":25741,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/2924944116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2924944116},{"type":"v1","addr":"172.21.15.86:6837","nonce":2924944116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:27:54.090 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 36 2024-03-04T12:27:54.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.085+0000 7fe665cea640 1 -- 172.21.15.60:0/3301490872 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fe6480a1f70 msgr2=0x7fe6480a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.085+0000 7fe665cea640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fe6480a1f70 0x7fe6480a4430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe6600ff6a0 tx=0x7fe650005f90 comp rx=0 tx=0).stop 2024-03-04T12:27:54.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.085+0000 7fe665cea640 1 -- 172.21.15.60:0/3301490872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe660102ec0 msgr2=0x7fe66019dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.085+0000 7fe665cea640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe660102ec0 0x7fe66019dd30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe654009230 tx=0x7fe65402dc70 comp rx=0 tx=0).stop 2024-03-04T12:27:54.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.085+0000 7fe665cea640 1 -- 172.21.15.60:0/3301490872 shutdown_connections 2024-03-04T12:27:54.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.085+0000 7fe665cea640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe66010b8e0 0x7fe6601979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.085+0000 7fe665cea640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe660102ec0 0x7fe66019dd30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.085+0000 7fe665cea640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fe6480a1f70 0x7fe6480a4430 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.085+0000 7fe665cea640 1 --2- 172.21.15.60:0/3301490872 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fe6601025a0 0x7fe66019d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.085+0000 7fe665cea640 1 -- 172.21.15.60:0/3301490872 >> 172.21.15.60:0/3301490872 conn(0x7fe660077350 msgr2=0x7fe6600fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.086+0000 7fe665cea640 1 -- 172.21.15.60:0/3301490872 shutdown_connections 2024-03-04T12:27:54.093 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.086+0000 7fe665cea640 1 -- 172.21.15.60:0/3301490872 wait complete. 2024-03-04T12:27:54.106 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:27:54.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-04T12:27:54.128 INFO:tasks.workunit.client.0.smithi060.stderr:alloc.c: In function 'main': 2024-03-04T12:27:54.129 INFO:tasks.workunit.client.0.smithi060.stderr:alloc.c:154:1: error: 'bozo' undeclared (first use in this function) 2024-03-04T12:27:54.129 INFO:tasks.workunit.client.0.smithi060.stderr: 154 | bozo! 2024-03-04T12:27:54.129 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~ 2024-03-04T12:27:54.129 INFO:tasks.workunit.client.0.smithi060.stderr:alloc.c:154:1: note: each undeclared identifier is reported only once for each function it appears in 2024-03-04T12:27:54.129 INFO:tasks.workunit.client.0.smithi060.stderr:alloc.c:154:5: error: expected ';' before '!' token 2024-03-04T12:27:54.129 INFO:tasks.workunit.client.0.smithi060.stderr: 154 | bozo! 2024-03-04T12:27:54.129 INFO:tasks.workunit.client.0.smithi060.stderr: | ^ 2024-03-04T12:27:54.129 INFO:tasks.workunit.client.0.smithi060.stderr: | ; 2024-03-04T12:27:54.130 INFO:tasks.workunit.client.0.smithi060.stderr:alloc.c:184:25: error: 'rflag' undeclared (first use in this function); did you mean 'nflag'? 2024-03-04T12:27:54.130 INFO:tasks.workunit.client.0.smithi060.stderr: 184 | rflag = 1; 2024-03-04T12:27:54.130 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~ 2024-03-04T12:27:54.130 INFO:tasks.workunit.client.0.smithi060.stderr: | nflag 2024-03-04T12:27:54.130 INFO:tasks.workunit.client.0.smithi060.stderr:alloc.c:320:5: error: expected ';' before '!' token 2024-03-04T12:27:54.130 INFO:tasks.workunit.client.0.smithi060.stderr: 320 | bozo! 2024-03-04T12:27:54.130 INFO:tasks.workunit.client.0.smithi060.stderr: | ^ 2024-03-04T12:27:54.130 INFO:tasks.workunit.client.0.smithi060.stderr: | ; 2024-03-04T12:27:54.131 INFO:tasks.workunit.client.0.smithi060.stderr:alloc.c:128:25: warning: variable 'f' set but not used [-Wunused-but-set-variable] 2024-03-04T12:27:54.131 INFO:tasks.workunit.client.0.smithi060.stderr: 128 | struct flock64 f; 2024-03-04T12:27:54.131 INFO:tasks.workunit.client.0.smithi060.stderr: | ^ 2024-03-04T12:27:54.134 INFO:tasks.workunit.client.0.smithi060.stderr:gmake[3]: *** [Makefile:83: alloc] Error 1 2024-03-04T12:27:54.134 INFO:tasks.workunit.client.0.smithi060.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-03-04T12:27:54.254 INFO:tasks.workunit.client.0.smithi060.stderr:fstest.c: In function 'run_child': 2024-03-04T12:27:54.254 INFO:tasks.workunit.client.0.smithi060.stderr:fstest.c:95:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-04T12:27:54.254 INFO:tasks.workunit.client.0.smithi060.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-04T12:27:54.254 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~ 2024-03-04T12:27:54.254 INFO:tasks.workunit.client.0.smithi060.stderr:fstest.c:95:24: note: directive argument in the range [0, 2147483647] 2024-03-04T12:27:54.254 INFO:tasks.workunit.client.0.smithi060.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-04T12:27:54.254 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~~~~~~~ 2024-03-04T12:27:54.254 INFO:tasks.workunit.client.0.smithi060.stderr:fstest.c:95:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-04T12:27:54.254 INFO:tasks.workunit.client.0.smithi060.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-04T12:27:54.254 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-04T12:27:54.255 INFO:tasks.workunit.client.0.smithi060.stderr:fstest.c:166:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-04T12:27:54.255 INFO:tasks.workunit.client.0.smithi060.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-04T12:27:54.255 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~ 2024-03-04T12:27:54.255 INFO:tasks.workunit.client.0.smithi060.stderr:fstest.c:166:24: note: directive argument in the range [0, 2147483647] 2024-03-04T12:27:54.255 INFO:tasks.workunit.client.0.smithi060.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-04T12:27:54.255 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~~~~~~~ 2024-03-04T12:27:54.255 INFO:tasks.workunit.client.0.smithi060.stderr:fstest.c:166:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-04T12:27:54.255 INFO:tasks.workunit.client.0.smithi060.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-04T12:27:54.255 INFO:tasks.workunit.client.0.smithi060.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-04T12:27:54.268 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.262+0000 7fbb2bf48640 1 -- 172.21.15.60:0/548685938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb24100440 msgr2=0x7fbb24100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.262+0000 7fbb2bf48640 1 --2- 172.21.15.60:0/548685938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb24100440 0x7fbb24100820 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c009900 tx=0x7fbb1c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:54.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.262+0000 7fbb2bf48640 1 -- 172.21.15.60:0/548685938 shutdown_connections 2024-03-04T12:27:54.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.262+0000 7fbb2bf48640 1 --2- 172.21.15.60:0/548685938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbb24105510 0x7fbb2410d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.262+0000 7fbb2bf48640 1 --2- 172.21.15.60:0/548685938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb24100df0 0x7fbb24104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.262+0000 7fbb2bf48640 1 --2- 172.21.15.60:0/548685938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb24100440 0x7fbb24100820 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.262+0000 7fbb2bf48640 1 -- 172.21.15.60:0/548685938 >> 172.21.15.60:0/548685938 conn(0x7fbb240773e0 msgr2=0x7fbb240777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.262+0000 7fbb2bf48640 1 -- 172.21.15.60:0/548685938 shutdown_connections 2024-03-04T12:27:54.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.262+0000 7fbb2bf48640 1 -- 172.21.15.60:0/548685938 wait complete. 2024-03-04T12:27:54.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.262+0000 7fbb2bf48640 1 Processor -- start 2024-03-04T12:27:54.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb2bf48640 1 -- start start 2024-03-04T12:27:54.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb2bf48640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb24100440 0x7fbb2419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb2bf48640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbb24105510 0x7fbb2419ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb2bf48640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb24196940 0x7fbb24196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb2bf48640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb241105c0 con 0x7fbb24100440 2024-03-04T12:27:54.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb2bf48640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb24110440 con 0x7fbb24105510 2024-03-04T12:27:54.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb2bf48640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb24110740 con 0x7fbb24196940 2024-03-04T12:27:54.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb29ce4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb24100440 0x7fbb2419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb29ce4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb24100440 0x7fbb2419c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52778/0 (socket says 172.21.15.60:52778) 2024-03-04T12:27:54.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb29ce4640 1 -- 172.21.15.60:0/1570516157 learned_addr learned my addr 172.21.15.60:0/1570516157 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb294e3640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbb24105510 0x7fbb2419ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb24196940 0x7fbb24196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.272 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb29ce4640 1 -- 172.21.15.60:0/1570516157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb24196940 msgr2=0x7fbb24196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb29ce4640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb24196940 0x7fbb24196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb29ce4640 1 -- 172.21.15.60:0/1570516157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbb24105510 msgr2=0x7fbb2419ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb29ce4640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbb24105510 0x7fbb2419ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb29ce4640 1 -- 172.21.15.60:0/1570516157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb24197460 con 0x7fbb24100440 2024-03-04T12:27:54.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.263+0000 7fbb29ce4640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb24100440 0x7fbb2419c790 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c006010 tx=0x7fbb1c02cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.264+0000 7fbb22ffd640 1 -- 172.21.15.60:0/1570516157 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c041070 con 0x7fbb24100440 2024-03-04T12:27:54.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.264+0000 7fbb2bf48640 1 -- 172.21.15.60:0/1570516157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb241976f0 con 0x7fbb24100440 2024-03-04T12:27:54.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.264+0000 7fbb22ffd640 1 -- 172.21.15.60:0/1570516157 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb1c0054e0 con 0x7fbb24100440 2024-03-04T12:27:54.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.264+0000 7fbb22ffd640 1 -- 172.21.15.60:0/1570516157 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c034440 con 0x7fbb24100440 2024-03-04T12:27:54.273 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.264+0000 7fbb2bf48640 1 -- 172.21.15.60:0/1570516157 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb241a38e0 con 0x7fbb24100440 2024-03-04T12:27:54.280 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.266+0000 7fbb22ffd640 1 -- 172.21.15.60:0/1570516157 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7fbb1c044070 con 0x7fbb24100440 2024-03-04T12:27:54.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.266+0000 7fbb2ebfd740 1 Processor -- start 2024-03-04T12:27:54.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.266+0000 7fbb2ebfd740 1 -- start start 2024-03-04T12:27:54.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.266+0000 7fbb2ebfd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b15323be0 0x564b15323fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.266+0000 7fbb22ffd640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbb0c0a1ef0 0x7fbb0c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.266+0000 7fbb2ebfd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15324500 0x564b1532cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.266+0000 7fbb2ebfd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 0x564b1532f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.266+0000 7fbb2ebfd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b152e8dc0 con 0x564b1532d0e0 2024-03-04T12:27:54.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.266+0000 7fbb2ebfd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b15284ec0 con 0x564b15324500 2024-03-04T12:27:54.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.266+0000 7fbb2ebfd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b1522dd30 con 0x564b15323be0 2024-03-04T12:27:54.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb2a4e5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b15323be0 0x564b15323fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb2a4e5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b15323be0 0x564b15323fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:59734/0 (socket says 172.21.15.60:59734) 2024-03-04T12:27:54.281 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb2a4e5640 1 -- 172.21.15.60:0/1246487388 learned_addr learned my addr 172.21.15.60:0/1246487388 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb29ce4640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbb0c0a1ef0 0x7fbb0c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/1246487388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 0x564b1532f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb22ffd640 1 -- 172.21.15.60:0/1570516157 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fbb1c0f0100 con 0x7fbb24100440 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb294e3640 1 --2- 172.21.15.60:0/1246487388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15324500 0x564b1532cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb2a4e5640 1 -- 172.21.15.60:0/1246487388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b15323be0 msgr2=0x564b15323fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/1246487388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b15323be0 0x564b15323fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb2a4e5640 1 -- 172.21.15.60:0/1246487388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15324500 msgr2=0x564b1532cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/1246487388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15324500 0x564b1532cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb2a4e5640 1 -- 172.21.15.60:0/1246487388 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b14f993f0 con 0x564b1532d0e0 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb294e3640 1 --2- 172.21.15.60:0/1246487388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15324500 0x564b1532cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb29ce4640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbb0c0a1ef0 0x7fbb0c0a43b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0b9230 tx=0x7fbb1c0b1cf0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/1246487388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 0x564b1532f4d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fbb14013a70 tx=0x7fbb14005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9dc20b33e97f13 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb0bfff640 1 -- 172.21.15.60:0/1246487388 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb14004b20 con 0x564b1532d0e0 2024-03-04T12:27:54.282 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb0bfff640 1 -- 172.21.15.60:0/1246487388 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb14004cc0 con 0x564b1532d0e0 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.267+0000 7fbb0bfff640 1 -- 172.21.15.60:0/1246487388 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb14003e60 con 0x564b1532d0e0 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.268+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/1246487388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 msgr2=0x564b1532f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.268+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/1246487388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 0x564b1532f4d0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fbb14013a70 tx=0x7fbb14005bd0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.268+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/1246487388 shutdown_connections 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.268+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/1246487388 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 0x564b1532f4d0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.268+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/1246487388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15324500 0x564b1532cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.268+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/1246487388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b15323be0 0x564b15323fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.268+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/1246487388 >> 172.21.15.60:0/1246487388 conn(0x564b1531e310 msgr2=0x564b1531e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.268+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/1246487388 shutdown_connections 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.268+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/1246487388 wait complete. 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb2ebfd740 1 -- start start 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb2ebfd740 1 Processor -- start 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb2ebfd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15323be0 0x564b1532a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.283 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb2ebfd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15324500 0x564b1532a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb2ebfd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b1532d0e0 0x564b1532abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb2ebfd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b152234a0 con 0x564b15324500 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb2ebfd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b15280f60 con 0x564b15323be0 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb2ebfd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b1532fe70 con 0x564b1532d0e0 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb2ebfd740 10 client.0 _ll_register_callbacks cb 0x564b1527f1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb294e3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15324500 0x564b1532a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb294e3640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15324500 0x564b1532a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52804/0 (socket says 172.21.15.60:52804) 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb294e3640 1 -- 172.21.15.60:0/4102928046 learned_addr learned my addr 172.21.15.60:0/4102928046 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.269+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b1532d0e0 0x564b1532abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15323be0 0x564b1532a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb294e3640 1 -- 172.21.15.60:0/4102928046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b1532d0e0 msgr2=0x564b1532abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb294e3640 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b1532d0e0 0x564b1532abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb294e3640 1 -- 172.21.15.60:0/4102928046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15323be0 msgr2=0x564b1532a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.284 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb294e3640 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15323be0 0x564b1532a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb294e3640 1 -- 172.21.15.60:0/4102928046 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b150637d0 con 0x564b15324500 2024-03-04T12:27:54.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b1532d0e0 0x564b1532abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:54.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb294e3640 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15324500 0x564b1532a6a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fbb18011750 tx=0x7fbb18011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c991b55553ec48d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb097fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:54.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb097fa640 1 -- 172.21.15.60:0/4102928046 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb18011e60 con 0x564b15324500 2024-03-04T12:27:54.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/4102928046 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb14004cc0 con 0x564b15324500 2024-03-04T12:27:54.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb097fa640 1 -- 172.21.15.60:0/4102928046 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fbb1800fdb0 con 0x564b15324500 2024-03-04T12:27:54.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb097fa640 1 -- 172.21.15.60:0/4102928046 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb18024050 con 0x564b15324500 2024-03-04T12:27:54.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb2ebfd740 10 client.16452 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:54.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/4102928046 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x564b1533c090 con 0x564b15324500 2024-03-04T12:27:54.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.270+0000 7fbb097fa640 1 -- 172.21.15.60:0/4102928046 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb14004cc0 con 0x564b15324500 2024-03-04T12:27:54.285 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.271+0000 7fbb097fa640 1 -- 172.21.15.60:0/4102928046 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fbb18018030 con 0x564b15324500 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.271+0000 7fbb2ebfd740 10 client.16452 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.271+0000 7fbb2ebfd740 10 client.16452 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.271+0000 7fbb2ebfd740 20 client.16452 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.271+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x564b1533d4a0 0x564b1533f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.271+0000 7fbb2ebfd740 4 client.16452 mds_command: new command op to 24394 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.271+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/4102928046 --> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564b15292400 con 0x564b1533d4a0 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.271+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x564b1533d4a0 0x564b1533f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.271+0000 7fbb097fa640 10 client.16452 ms_handle_connect on v2:172.21.15.86:6832/777854669 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.271+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x564b1533d4a0 0x564b1533f890 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2034cf4d5558f797 server_cookie=24535167a2be26fa in_seq=0 out_seq=0 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb097fa640 1 -- 172.21.15.60:0/4102928046 <== mds.4 v2:172.21.15.86:6832/777854669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564b15292400 con 0x564b1533d4a0 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb097fa640 10 client.16452 handle_command_reply: tid=0 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb2ebfd740 1 client.16452 shutdown 2024-03-04T12:27:54.286 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/4102928046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15324500 msgr2=0x564b1532a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15324500 0x564b1532a6a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fbb18011750 tx=0x7fbb18011c10 comp rx=0 tx=0).stop 2024-03-04T12:27:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/4102928046 shutdown_connections 2024-03-04T12:27:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b1532d0e0 0x564b1532abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15324500 0x564b1532a6a0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x564b1533d4a0 0x564b1533f890 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/4102928046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15323be0 0x564b1532a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb097fa640 0 client.16452 ms_handle_reset on v2:172.21.15.86:6832/777854669 2024-03-04T12:27:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/4102928046 >> 172.21.15.60:0/4102928046 conn(0x564b1531e310 msgr2=0x564b15320060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/4102928046 shutdown_connections 2024-03-04T12:27:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/4102928046 wait complete. 2024-03-04T12:27:54.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.272+0000 7fbb2ebfd740 20 client.16452 trim_cache size 0 max 16384 2024-03-04T12:27:54.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb2ebfd740 1 Processor -- start 2024-03-04T12:27:54.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb2ebfd740 1 -- start start 2024-03-04T12:27:54.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb2ebfd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15323be0 0x564b1532c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb2ebfd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b15324500 0x564b153273c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb2ebfd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 0x564b15327900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb2ebfd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb241ac2d0 con 0x564b1532d0e0 2024-03-04T12:27:54.292 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb2ebfd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb241adcb0 con 0x564b15323be0 2024-03-04T12:27:54.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb2ebfd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb241ac930 con 0x564b15324500 2024-03-04T12:27:54.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb2a4e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 0x564b15327900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb2a4e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 0x564b15327900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52816/0 (socket says 172.21.15.60:52816) 2024-03-04T12:27:54.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb2a4e5640 1 -- 172.21.15.60:0/1732564638 learned_addr learned my addr 172.21.15.60:0/1732564638 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb294e3640 1 --2- 172.21.15.60:0/1732564638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b15324500 0x564b153273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.284+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/1732564638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15323be0 0x564b1532c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.285+0000 7fbb2a4e5640 1 -- 172.21.15.60:0/1732564638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b15324500 msgr2=0x564b153273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.285+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/1732564638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b15324500 0x564b153273c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.285+0000 7fbb2a4e5640 1 -- 172.21.15.60:0/1732564638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15323be0 msgr2=0x564b1532c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.285+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/1732564638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15323be0 0x564b1532c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.293 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.285+0000 7fbb2a4e5640 1 -- 172.21.15.60:0/1732564638 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b1532f4d0 con 0x564b1532d0e0 2024-03-04T12:27:54.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.285+0000 7fbb294e3640 1 --2- 172.21.15.60:0/1732564638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b15324500 0x564b153273c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.285+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/1732564638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 0x564b15327900 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbb14003c70 tx=0x7fbb14003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57f4d1e3b45ffbe server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.285+0000 7fbb0b7fe640 1 -- 172.21.15.60:0/1732564638 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb14013680 con 0x564b1532d0e0 2024-03-04T12:27:54.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.285+0000 7fbb0b7fe640 1 -- 172.21.15.60:0/1732564638 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb140130d0 con 0x564b1532d0e0 2024-03-04T12:27:54.294 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.285+0000 7fbb0b7fe640 1 -- 172.21.15.60:0/1732564638 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb14029df0 con 0x564b1532d0e0 2024-03-04T12:27:54.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.286+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/1732564638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 msgr2=0x564b15327900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.286+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/1732564638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 0x564b15327900 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbb14003c70 tx=0x7fbb14003c00 comp rx=0 tx=0).stop 2024-03-04T12:27:54.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.286+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/1732564638 shutdown_connections 2024-03-04T12:27:54.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.286+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/1732564638 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b1532d0e0 0x564b15327900 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.298 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.286+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/1732564638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b15324500 0x564b153273c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.286+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/1732564638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15323be0 0x564b1532c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.286+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/1732564638 >> 172.21.15.60:0/1732564638 conn(0x564b1531e310 msgr2=0x564b15328a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.286+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/1732564638 shutdown_connections 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.286+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/1732564638 wait complete. 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.286+0000 7fbb2ebfd740 1 -- start start 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.286+0000 7fbb2ebfd740 1 Processor -- start 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2ebfd740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15323be0 0x564b15332c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2ebfd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15324500 0x564b15333170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2ebfd740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b1532d0e0 0x564b153336b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2ebfd740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb241ad650 con 0x564b15323be0 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2ebfd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb241acf90 con 0x564b15324500 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2ebfd740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb241aa570 con 0x564b1532d0e0 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2ebfd740 10 client.0 _ll_register_callbacks cb 0x7fbb2406abc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2a4e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15323be0 0x564b15332c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.299 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2a4e5640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15323be0 0x564b15332c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52822/0 (socket says 172.21.15.60:52822) 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2a4e5640 1 -- 172.21.15.60:0/2642422493 learned_addr learned my addr 172.21.15.60:0/2642422493 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb294e3640 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15324500 0x564b15333170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b1532d0e0 0x564b153336b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2a4e5640 1 -- 172.21.15.60:0/2642422493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b1532d0e0 msgr2=0x564b153336b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b1532d0e0 0x564b153336b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2a4e5640 1 -- 172.21.15.60:0/2642422493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15324500 msgr2=0x564b15333170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15324500 0x564b15333170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2a4e5640 1 -- 172.21.15.60:0/2642422493 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b15393b80 con 0x564b15323be0 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb294e3640 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15324500 0x564b15333170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2a4e5640 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15323be0 0x564b15332c30 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x564b15328900 tx=0x7fbb14029c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39222da1813b9316 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb097fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb097fa640 1 -- 172.21.15.60:0/2642422493 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb14004a00 con 0x564b15323be0 2024-03-04T12:27:54.300 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/2642422493 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb140130d0 con 0x564b15323be0 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.287+0000 7fbb097fa640 1 -- 172.21.15.60:0/2642422493 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fbb14004ba0 con 0x564b15323be0 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.288+0000 7fbb097fa640 1 -- 172.21.15.60:0/2642422493 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb14010aa0 con 0x564b15323be0 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.288+0000 7fbb2ebfd740 10 client.16458 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.288+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/2642422493 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x564b15394e60 con 0x564b15323be0 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.288+0000 7fbb097fa640 1 -- 172.21.15.60:0/2642422493 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb140130d0 con 0x564b15323be0 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.288+0000 7fbb097fa640 1 -- 172.21.15.60:0/2642422493 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fbb14011540 con 0x564b15323be0 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.288+0000 7fbb2ebfd740 10 client.16458 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.288+0000 7fbb2ebfd740 10 client.16458 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.l 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.288+0000 7fbb2ebfd740 20 client.16458 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.288+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x564b153952e0 0x564b153976d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.288+0000 7fbb2ebfd740 4 client.16458 mds_command: new command op to 24394 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.288+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/2642422493 --> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564b1528c750 con 0x564b153952e0 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.288+0000 7fbb294e3640 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x564b153952e0 0x564b153976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.301 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb097fa640 10 client.16458 ms_handle_connect on v2:172.21.15.86:6832/777854669 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb294e3640 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x564b153952e0 0x564b153976d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=aac8d76da44e0a86 server_cookie=c27854c8ca2e736c in_seq=0 out_seq=0 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb097fa640 1 -- 172.21.15.60:0/2642422493 <== mds.4 v2:172.21.15.86:6832/777854669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564b1528c750 con 0x564b153952e0 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb097fa640 10 client.16458 handle_command_reply: tid=0 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb2ebfd740 1 client.16458 shutdown 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/2642422493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15323be0 msgr2=0x564b15332c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15323be0 0x564b15332c30 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x564b15328900 tx=0x7fbb14029c30 comp rx=0 tx=0).stop 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/2642422493 shutdown_connections 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564b1532d0e0 0x564b153336b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.86:6832/777854669,v1:172.21.15.86:6833/777854669] conn(0x564b153952e0 0x564b153976d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564b15324500 0x564b15333170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb097fa640 0 client.16458 ms_handle_reset on v2:172.21.15.86:6832/777854669 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb2ebfd740 1 --2- 172.21.15.60:0/2642422493 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x564b15323be0 0x564b15332c30 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/2642422493 >> 172.21.15.60:0/2642422493 conn(0x564b1531e310 msgr2=0x564b1531a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/2642422493 shutdown_connections 2024-03-04T12:27:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb2ebfd740 1 -- 172.21.15.60:0/2642422493 wait complete. 2024-03-04T12:27:54.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.289+0000 7fbb2ebfd740 20 client.16458 trim_cache size 0 max 16384 2024-03-04T12:27:54.303 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:27:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.290+0000 7fbb2bf48640 1 -- 172.21.15.60:0/1570516157 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbb0c0a1ef0 msgr2=0x7fbb0c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.290+0000 7fbb2bf48640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbb0c0a1ef0 0x7fbb0c0a43b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0b9230 tx=0x7fbb1c0b1cf0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.290+0000 7fbb2bf48640 1 -- 172.21.15.60:0/1570516157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb24100440 msgr2=0x7fbb2419c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.290+0000 7fbb2bf48640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb24100440 0x7fbb2419c790 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c006010 tx=0x7fbb1c02cd70 comp rx=0 tx=0).stop 2024-03-04T12:27:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.290+0000 7fbb2bf48640 1 -- 172.21.15.60:0/1570516157 shutdown_connections 2024-03-04T12:27:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.290+0000 7fbb2bf48640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb24196940 0x7fbb24196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.290+0000 7fbb2bf48640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbb24105510 0x7fbb2419ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.290+0000 7fbb2bf48640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbb0c0a1ef0 0x7fbb0c0a43b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.290+0000 7fbb2bf48640 1 --2- 172.21.15.60:0/1570516157 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbb24100440 0x7fbb2419c790 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.290+0000 7fbb2bf48640 1 -- 172.21.15.60:0/1570516157 >> 172.21.15.60:0/1570516157 conn(0x7fbb240773e0 msgr2=0x7fbb240fe4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.290+0000 7fbb2bf48640 1 -- 172.21.15.60:0/1570516157 shutdown_connections 2024-03-04T12:27:54.304 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.290+0000 7fbb2bf48640 1 -- 172.21.15.60:0/1570516157 wait complete. 2024-03-04T12:27:54.308 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:27:54.308 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-04T12:27:54.327 INFO:tasks.workunit.client.0.smithi060.stderr:gmake[2]: *** [include/buildrules:30: src] Error 2 2024-03-04T12:27:54.327 INFO:tasks.workunit.client.0.smithi060.stderr:make[1]: *** [Makefile:53: default] Error 2 2024-03-04T12:27:54.328 INFO:tasks.workunit.client.0.smithi060.stderr:make: *** [Makefile:51: default] Error 2 2024-03-04T12:27:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-04T12:27:54.329 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-03-04T12:27:54.329 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-04T12:27:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.399+0000 7f52e0e20640 1 -- 172.21.15.60:0/479428464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc068e40 msgr2=0x7f52dc069220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.399+0000 7f52e0e20640 1 --2- 172.21.15.60:0/479428464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc068e40 0x7f52dc069220 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009900 tx=0x7f52cc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.399+0000 7f52e0e20640 1 -- 172.21.15.60:0/479428464 shutdown_connections 2024-03-04T12:27:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.399+0000 7f52e0e20640 1 --2- 172.21.15.60:0/479428464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52dc106610 0x7f52dc10e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.399+0000 7f52e0e20640 1 --2- 172.21.15.60:0/479428464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc0697f0 0x7f52dc105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.399+0000 7f52e0e20640 1 --2- 172.21.15.60:0/479428464 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc068e40 0x7f52dc069220 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.399+0000 7f52e0e20640 1 -- 172.21.15.60:0/479428464 >> 172.21.15.60:0/479428464 conn(0x7f52dc077320 msgr2=0x7f52dc077730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.399+0000 7f52e0e20640 1 -- 172.21.15.60:0/479428464 shutdown_connections 2024-03-04T12:27:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.399+0000 7f52e0e20640 1 -- 172.21.15.60:0/479428464 wait complete. 2024-03-04T12:27:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52e0e20640 1 Processor -- start 2024-03-04T12:27:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52e0e20640 1 -- start start 2024-03-04T12:27:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52e0e20640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc068e40 0x7f52dc19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52e0e20640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52dc106610 0x7f52dc19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52e0e20640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc1979d0 0x7f52dc197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52e0e20640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc1116f0 con 0x7f52dc1979d0 2024-03-04T12:27:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52e0e20640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc111570 con 0x7f52dc106610 2024-03-04T12:27:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52e0e20640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc111870 con 0x7f52dc068e40 2024-03-04T12:27:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52dad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc1979d0 0x7f52dc197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52da59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc068e40 0x7f52dc19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52da59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc068e40 0x7f52dc19d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:59782/0 (socket says 172.21.15.60:59782) 2024-03-04T12:27:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52dad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc1979d0 0x7f52dc197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52830/0 (socket says 172.21.15.60:52830) 2024-03-04T12:27:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.400+0000 7f52da59c640 1 -- 172.21.15.60:0/975251074 learned_addr learned my addr 172.21.15.60:0/975251074 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.401+0000 7f52d9d9b640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52dc106610 0x7f52dc19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.401+0000 7f52da59c640 1 -- 172.21.15.60:0/975251074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52dc106610 msgr2=0x7f52dc19dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.401+0000 7f52da59c640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52dc106610 0x7f52dc19dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.401+0000 7f52da59c640 1 -- 172.21.15.60:0/975251074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc1979d0 msgr2=0x7f52dc197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.401+0000 7f52da59c640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc1979d0 0x7f52dc197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.401+0000 7f52da59c640 1 -- 172.21.15.60:0/975251074 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc1984f0 con 0x7f52dc068e40 2024-03-04T12:27:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.401+0000 7f52da59c640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc068e40 0x7f52dc19d820 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f52cc02c240 tx=0x7f52cc0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.401+0000 7f52cb7fe640 1 -- 172.21.15.60:0/975251074 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc043020 con 0x7f52dc068e40 2024-03-04T12:27:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.401+0000 7f52e0e20640 1 -- 172.21.15.60:0/975251074 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52dc198780 con 0x7f52dc068e40 2024-03-04T12:27:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.401+0000 7f52cb7fe640 1 -- 172.21.15.60:0/975251074 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52cc0054e0 con 0x7f52dc068e40 2024-03-04T12:27:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.401+0000 7f52cb7fe640 1 -- 172.21.15.60:0/975251074 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc003ab0 con 0x7f52dc068e40 2024-03-04T12:27:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.401+0000 7f52e0e20640 1 -- 172.21.15.60:0/975251074 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52dc1a4a20 con 0x7f52dc068e40 2024-03-04T12:27:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.403+0000 7f52cb7fe640 1 -- 172.21.15.60:0/975251074 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f52cc046020 con 0x7f52dc068e40 2024-03-04T12:27:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52e3a02740 1 Processor -- start 2024-03-04T12:27:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52e3a02740 1 -- start start 2024-03-04T12:27:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52e3a02740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac10d0 0x55fc9dabfe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.418 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52e3a02740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dac07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52e3a02740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 0x55fc9dacbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52e3a02740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9da245f0 con 0x55fc9dac98e0 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52e3a02740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9d9cd930 con 0x55fc9dac10d0 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52e3a02740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9da20690 con 0x55fc9dac0360 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52cb7fe640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f52c40a1ee0 0x7f52c40a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52dad9d640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f52c40a1ee0 0x7f52c40a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52d9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 0x55fc9dacbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52cb7fe640 1 -- 172.21.15.60:0/975251074 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f52cc0f0400 con 0x7f52dc068e40 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52d9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 0x55fc9dacbcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52840/0 (socket says 172.21.15.60:52840) 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52d9d9b640 1 -- 172.21.15.60:0/2787683857 learned_addr learned my addr 172.21.15.60:0/2787683857 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.404+0000 7f52da59c640 1 --2- 172.21.15.60:0/2787683857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dac07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.405+0000 7f52dad9d640 1 --2- 172.21.15.60:0/2787683857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac10d0 0x55fc9dabfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.419 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.405+0000 7f52da59c640 1 -- 172.21.15.60:0/2787683857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac10d0 msgr2=0x55fc9dabfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.405+0000 7f52da59c640 1 --2- 172.21.15.60:0/2787683857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac10d0 0x55fc9dabfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.405+0000 7f52da59c640 1 -- 172.21.15.60:0/2787683857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 msgr2=0x55fc9dacbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.405+0000 7f52da59c640 1 --2- 172.21.15.60:0/2787683857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 0x55fc9dacbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.405+0000 7f52da59c640 1 -- 172.21.15.60:0/2787683857 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc9d739210 con 0x55fc9dac0360 2024-03-04T12:27:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.405+0000 7f52d9d9b640 1 --2- 172.21.15.60:0/2787683857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 0x55fc9dacbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.405+0000 7f52dad9d640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f52c40a1ee0 0x7f52c40a43a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52d4005fd0 tx=0x7f52d4009ea0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.405+0000 7f52da59c640 1 --2- 172.21.15.60:0/2787683857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dac07e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0ed860 tx=0x7f52cc02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24dcedd2b5901f75 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.405+0000 7f52c8ff9640 1 -- 172.21.15.60:0/2787683857 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc0bd200 con 0x55fc9dac0360 2024-03-04T12:27:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.405+0000 7f52c8ff9640 1 -- 172.21.15.60:0/2787683857 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52cc0bcb00 con 0x55fc9dac0360 2024-03-04T12:27:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.405+0000 7f52c8ff9640 1 -- 172.21.15.60:0/2787683857 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc02ca90 con 0x55fc9dac0360 2024-03-04T12:27:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.406+0000 7f52e3a02740 1 -- 172.21.15.60:0/2787683857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 msgr2=0x55fc9dac07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.406+0000 7f52e3a02740 1 --2- 172.21.15.60:0/2787683857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dac07e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0ed860 tx=0x7f52cc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.406+0000 7f52e3a02740 1 -- 172.21.15.60:0/2787683857 shutdown_connections 2024-03-04T12:27:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.406+0000 7f52e3a02740 1 --2- 172.21.15.60:0/2787683857 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 0x55fc9dacbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.406+0000 7f52e3a02740 1 --2- 172.21.15.60:0/2787683857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dac07e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.406+0000 7f52e3a02740 1 --2- 172.21.15.60:0/2787683857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac10d0 0x55fc9dabfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.406+0000 7f52e3a02740 1 -- 172.21.15.60:0/2787683857 >> 172.21.15.60:0/2787683857 conn(0x55fc9da87b90 msgr2=0x55fc9dabb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.406+0000 7f52e3a02740 1 -- 172.21.15.60:0/2787683857 shutdown_connections 2024-03-04T12:27:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.406+0000 7f52e3a02740 1 -- 172.21.15.60:0/2787683857 wait complete. 2024-03-04T12:27:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.406+0000 7f52e3a02740 1 -- start start 2024-03-04T12:27:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52e3a02740 1 Processor -- start 2024-03-04T12:27:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52e3a02740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac0360 0x55fc9dac6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52e3a02740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 0x55fc9dac6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52e3a02740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac98e0 0x55fc9dac73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52e3a02740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9da20690 con 0x55fc9dac10d0 2024-03-04T12:27:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52e3a02740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9d9e0d90 con 0x55fc9dac0360 2024-03-04T12:27:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52e3a02740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9dacc3d0 con 0x55fc9dac98e0 2024-03-04T12:27:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52e3a02740 10 client.0 _ll_register_callbacks cb 0x55fc9d738a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52d9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 0x55fc9dac6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52d9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 0x55fc9dac6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52848/0 (socket says 172.21.15.60:52848) 2024-03-04T12:27:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52d9d9b640 1 -- 172.21.15.60:0/3462383818 learned_addr learned my addr 172.21.15.60:0/3462383818 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52da59c640 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac0360 0x55fc9dac6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52dad9d640 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac98e0 0x55fc9dac73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52da59c640 1 -- 172.21.15.60:0/3462383818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac98e0 msgr2=0x55fc9dac73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.422 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52da59c640 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac98e0 0x55fc9dac73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52da59c640 1 -- 172.21.15.60:0/3462383818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 msgr2=0x55fc9dac6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52da59c640 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 0x55fc9dac6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52da59c640 1 -- 172.21.15.60:0/3462383818 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc9d805620 con 0x55fc9dac0360 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52dad9d640 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac98e0 0x55fc9dac73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.407+0000 7f52d9d9b640 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 0x55fc9dac6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52da59c640 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac0360 0x55fc9dac6970 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55fc9dacc2e0 tx=0x7f52cc03f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec273597d3c60a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52c1ffb640 1 -- 172.21.15.60:0/3462383818 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc0da060 con 0x55fc9dac0360 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52c1ffb640 1 -- 172.21.15.60:0/3462383818 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f52cc003c50 con 0x55fc9dac0360 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52e3a02740 1 -- 172.21.15.60:0/3462383818 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52cc0bcb00 con 0x55fc9dac0360 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52c1ffb640 1 -- 172.21.15.60:0/3462383818 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52cc0c0c50 con 0x55fc9dac0360 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52e3a02740 10 client.25880 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:54.423 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52e3a02740 1 -- 172.21.15.60:0/3462383818 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55fc9dad8580 con 0x55fc9dac0360 2024-03-04T12:27:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52c1ffb640 1 -- 172.21.15.60:0/3462383818 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52cc0bcb00 con 0x55fc9dac0360 2024-03-04T12:27:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52c1ffb640 1 -- 172.21.15.60:0/3462383818 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f52cc0c4e10 con 0x55fc9dac0360 2024-03-04T12:27:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52e3a02740 10 client.25880 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52e3a02740 10 client.25880 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52e3a02740 20 client.25880 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.408+0000 7f52e3a02740 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fc9dad9700 0x55fc9dadbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.409+0000 7f52e3a02740 4 client.25880 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.409+0000 7f52e3a02740 1 -- 172.21.15.60:0/3462383818 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fc9da31b30 con 0x55fc9dad9700 2024-03-04T12:27:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.409+0000 7f52d9d9b640 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fc9dad9700 0x55fc9dadbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.409+0000 7f52c1ffb640 10 client.25880 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.409+0000 7f52d9d9b640 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fc9dad9700 0x55fc9dadbaf0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ed058070547b71e server_cookie=a445749686bf6cb2 in_seq=0 out_seq=0 2024-03-04T12:27:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52c1ffb640 1 -- 172.21.15.60:0/3462383818 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fc9da31b30 con 0x55fc9dad9700 2024-03-04T12:27:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52c1ffb640 10 client.25880 handle_command_reply: tid=0 2024-03-04T12:27:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52e3a02740 1 client.25880 shutdown 2024-03-04T12:27:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52e3a02740 1 -- 172.21.15.60:0/3462383818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac0360 msgr2=0x55fc9dac6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52e3a02740 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac0360 0x55fc9dac6970 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55fc9dacc2e0 tx=0x7f52cc03f000 comp rx=0 tx=0).stop 2024-03-04T12:27:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52e3a02740 1 -- 172.21.15.60:0/3462383818 shutdown_connections 2024-03-04T12:27:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52e3a02740 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fc9dad9700 0x55fc9dadbaf0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52e3a02740 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac98e0 0x55fc9dac73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52e3a02740 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 0x55fc9dac6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52e3a02740 1 --2- 172.21.15.60:0/3462383818 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac0360 0x55fc9dac6970 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52c1ffb640 0 client.25880 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:54.426 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52e3a02740 1 -- 172.21.15.60:0/3462383818 >> 172.21.15.60:0/3462383818 conn(0x55fc9da87b90 msgr2=0x55fc9dabcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.426 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52e3a02740 1 -- 172.21.15.60:0/3462383818 shutdown_connections 2024-03-04T12:27:54.426 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52e3a02740 1 -- 172.21.15.60:0/3462383818 wait complete. 2024-03-04T12:27:54.426 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.410+0000 7f52e3a02740 20 client.25880 trim_cache size 0 max 16384 2024-03-04T12:27:54.435 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.423+0000 7f52e3a02740 1 Processor -- start 2024-03-04T12:27:54.435 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.423+0000 7f52e3a02740 1 -- start start 2024-03-04T12:27:54.435 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.423+0000 7f52e3a02740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dabab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.435 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.423+0000 7f52e3a02740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 0x55fc9dabb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.435 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.423+0000 7f52e3a02740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac98e0 0x55fc9dac5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.435 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.423+0000 7f52e3a02740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc1ae310 con 0x55fc9dac10d0 2024-03-04T12:27:54.435 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.423+0000 7f52e3a02740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc1afcf0 con 0x55fc9dac98e0 2024-03-04T12:27:54.435 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.423+0000 7f52e3a02740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc1ae970 con 0x55fc9dac0360 2024-03-04T12:27:54.435 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52d9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 0x55fc9dabb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.436 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52d9d9b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 0x55fc9dabb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52860/0 (socket says 172.21.15.60:52860) 2024-03-04T12:27:54.436 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52d9d9b640 1 -- 172.21.15.60:0/1684238739 learned_addr learned my addr 172.21.15.60:0/1684238739 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.436 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52dad9d640 1 --2- 172.21.15.60:0/1684238739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac98e0 0x55fc9dac5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.436 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52da59c640 1 --2- 172.21.15.60:0/1684238739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dabab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.436 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52d9d9b640 1 -- 172.21.15.60:0/1684238739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 msgr2=0x55fc9dabab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.436 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52d9d9b640 1 --2- 172.21.15.60:0/1684238739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dabab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.436 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52d9d9b640 1 -- 172.21.15.60:0/1684238739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac98e0 msgr2=0x55fc9dac5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.436 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52d9d9b640 1 --2- 172.21.15.60:0/1684238739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac98e0 0x55fc9dac5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.436 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52d9d9b640 1 -- 172.21.15.60:0/1684238739 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc9dabf630 con 0x55fc9dac10d0 2024-03-04T12:27:54.436 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52d9d9b640 1 --2- 172.21.15.60:0/1684238739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 0x55fc9dabb0c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x55fc9dadc2a0 tx=0x7f52d0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c63b0a91872d56 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.436 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52c3fff640 1 -- 172.21.15.60:0/1684238739 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d0023030 con 0x55fc9dac10d0 2024-03-04T12:27:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52c3fff640 1 -- 172.21.15.60:0/1684238739 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52d0005020 con 0x55fc9dac10d0 2024-03-04T12:27:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.424+0000 7f52c3fff640 1 -- 172.21.15.60:0/1684238739 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d0004ab0 con 0x55fc9dac10d0 2024-03-04T12:27:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.425+0000 7f52e3a02740 1 -- 172.21.15.60:0/1684238739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 msgr2=0x55fc9dabb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.425+0000 7f52e3a02740 1 --2- 172.21.15.60:0/1684238739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 0x55fc9dabb0c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x55fc9dadc2a0 tx=0x7f52d0005f90 comp rx=0 tx=0).stop 2024-03-04T12:27:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.425+0000 7f52e3a02740 1 -- 172.21.15.60:0/1684238739 shutdown_connections 2024-03-04T12:27:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.425+0000 7f52e3a02740 1 --2- 172.21.15.60:0/1684238739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac98e0 0x55fc9dac5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.425+0000 7f52e3a02740 1 --2- 172.21.15.60:0/1684238739 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac10d0 0x55fc9dabb0c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.425+0000 7f52e3a02740 1 --2- 172.21.15.60:0/1684238739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dabab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.425+0000 7f52e3a02740 1 -- 172.21.15.60:0/1684238739 >> 172.21.15.60:0/1684238739 conn(0x55fc9da87b90 msgr2=0x55fc9dac1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.425+0000 7f52e3a02740 1 -- 172.21.15.60:0/1684238739 shutdown_connections 2024-03-04T12:27:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.425+0000 7f52e3a02740 1 -- 172.21.15.60:0/1684238739 wait complete. 2024-03-04T12:27:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52e3a02740 1 -- start start 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52e3a02740 1 Processor -- start 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52e3a02740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dad0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52e3a02740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac10d0 0x55fc9dad08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52e3a02740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 0x55fc9dacea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52e3a02740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc1af690 con 0x55fc9dac98e0 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52e3a02740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc1aefd0 con 0x55fc9dac10d0 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52e3a02740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc1ac5b0 con 0x55fc9dac0360 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52dad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 0x55fc9dacea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52e3a02740 10 client.0 _ll_register_callbacks cb 0x7f52cc0bc8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52dad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 0x55fc9dacea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52866/0 (socket says 172.21.15.60:52866) 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52dad9d640 1 -- 172.21.15.60:0/1029499917 learned_addr learned my addr 172.21.15.60:0/1029499917 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52da59c640 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dad0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52d9d9b640 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac10d0 0x55fc9dad08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52da59c640 1 -- 172.21.15.60:0/1029499917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac10d0 msgr2=0x55fc9dad08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52da59c640 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac10d0 0x55fc9dad08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52da59c640 1 -- 172.21.15.60:0/1029499917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 msgr2=0x55fc9dacea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52da59c640 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 0x55fc9dacea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52da59c640 1 -- 172.21.15.60:0/1029499917 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc9dacf180 con 0x55fc9dac0360 2024-03-04T12:27:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.426+0000 7f52dad9d640 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 0x55fc9dacea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52da59c640 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dad0360 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0badb0 tx=0x7f52cc000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ab70c4c64361743 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-03-04T12:27:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52c1ffb640 1 -- 172.21.15.60:0/1029499917 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52cc0da070 con 0x55fc9dac0360 2024-03-04T12:27:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52c1ffb640 1 -- 172.21.15.60:0/1029499917 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f52cc0d0b50 con 0x55fc9dac0360 2024-03-04T12:27:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52e3a02740 1 -- 172.21.15.60:0/1029499917 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52d0005020 con 0x55fc9dac0360 2024-03-04T12:27:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52c1ffb640 1 -- 172.21.15.60:0/1029499917 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52cc0d4040 con 0x55fc9dac0360 2024-03-04T12:27:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52e3a02740 10 client.25984 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52e3a02740 1 -- 172.21.15.60:0/1029499917 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55fc9dacf180 con 0x55fc9dac0360 2024-03-04T12:27:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52c1ffb640 1 -- 172.21.15.60:0/1029499917 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52d0005020 con 0x55fc9dac0360 2024-03-04T12:27:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52c1ffb640 1 -- 172.21.15.60:0/1029499917 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f52cc0bbf50 con 0x55fc9dac0360 2024-03-04T12:27:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52e3a02740 10 client.25984 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52e3a02740 10 client.25984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52e3a02740 20 client.25984 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52e3a02740 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fc9db30820 0x55fc9db32c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52e3a02740 4 client.25984 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-04T12:27:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.427+0000 7f52e3a02740 1 -- 172.21.15.60:0/1029499917 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fc9da2be80 con 0x55fc9db30820 2024-03-04T12:27:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.428+0000 7f52d9d9b640 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fc9db30820 0x55fc9db32c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.428+0000 7f52d9d9b640 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fc9db30820 0x55fc9db32c10 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dda161b6a5aa57ba server_cookie=6fc8f588a5bd8ddd in_seq=0 out_seq=0 2024-03-04T12:27:54.441 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.428+0000 7f52c1ffb640 10 client.25984 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:54.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.430+0000 7fc9813c4640 1 -- 172.21.15.60:0/1678440630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c101530 msgr2=0x7fc97c101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.430+0000 7fc9813c4640 1 --2- 172.21.15.60:0/1678440630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c101530 0x7fc97c101910 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009860 tx=0x7fc96c02bb70 comp rx=0 tx=0).stop 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.430+0000 7fc9813c4640 1 -- 172.21.15.60:0/1678440630 shutdown_connections 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.430+0000 7fc9813c4640 1 --2- 172.21.15.60:0/1678440630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc97c106600 0x7fc97c10e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.430+0000 7fc9813c4640 1 --2- 172.21.15.60:0/1678440630 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc97c101ee0 0x7fc97c105f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.430+0000 7fc9813c4640 1 --2- 172.21.15.60:0/1678440630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c101530 0x7fc97c101910 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.430+0000 7fc9813c4640 1 -- 172.21.15.60:0/1678440630 >> 172.21.15.60:0/1678440630 conn(0x7fc97c0fcdd0 msgr2=0x7fc97c0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.430+0000 7fc9813c4640 1 -- 172.21.15.60:0/1678440630 shutdown_connections 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.430+0000 7fc9813c4640 1 -- 172.21.15.60:0/1678440630 wait complete. 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc9813c4640 1 Processor -- start 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc9813c4640 1 -- start start 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc9813c4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c101530 0x7fc97c073630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc9813c4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc97c106600 0x7fc97c073b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc9813c4640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc97c0740b0 0x7fc97c1a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc9813c4640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c111690 con 0x7fc97c0740b0 2024-03-04T12:27:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc9813c4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c111510 con 0x7fc97c106600 2024-03-04T12:27:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc9813c4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c111810 con 0x7fc97c101530 2024-03-04T12:27:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc97affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c101530 0x7fc97c073630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc97affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c101530 0x7fc97c073630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:59834/0 (socket says 172.21.15.60:59834) 2024-03-04T12:27:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc97affd640 1 -- 172.21.15.60:0/2707502794 learned_addr learned my addr 172.21.15.60:0/2707502794 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc97b7fe640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc97c0740b0 0x7fc97c1a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.431+0000 7fc97a7fc640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc97c106600 0x7fc97c073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.432+0000 7fc97affd640 1 -- 172.21.15.60:0/2707502794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc97c106600 msgr2=0x7fc97c073b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.432+0000 7fc97affd640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc97c106600 0x7fc97c073b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.432+0000 7fc97affd640 1 -- 172.21.15.60:0/2707502794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc97c0740b0 msgr2=0x7fc97c1a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.432+0000 7fc97affd640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc97c0740b0 0x7fc97c1a5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.432+0000 7fc97affd640 1 -- 172.21.15.60:0/2707502794 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c1a6460 con 0x7fc97c101530 2024-03-04T12:27:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.432+0000 7fc97affd640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c101530 0x7fc97c073630 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc96c005e00 tx=0x7fc96c0045d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.433+0000 7fc963fff640 1 -- 172.21.15.60:0/2707502794 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c041020 con 0x7fc97c101530 2024-03-04T12:27:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.433+0000 7fc963fff640 1 -- 172.21.15.60:0/2707502794 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc96c02c9f0 con 0x7fc97c101530 2024-03-04T12:27:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.433+0000 7fc963fff640 1 -- 172.21.15.60:0/2707502794 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c039040 con 0x7fc97c101530 2024-03-04T12:27:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.433+0000 7fc9813c4640 1 -- 172.21.15.60:0/2707502794 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c1a66f0 con 0x7fc97c101530 2024-03-04T12:27:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.433+0000 7fc9813c4640 1 -- 172.21.15.60:0/2707502794 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c1a6ad0 con 0x7fc97c101530 2024-03-04T12:27:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.436+0000 7fc963fff640 1 -- 172.21.15.60:0/2707502794 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7fc96c044020 con 0x7fc97c101530 2024-03-04T12:27:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.436+0000 7fc963fff640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc9640a2070 0x7fc9640a4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.436+0000 7fc963fff640 1 -- 172.21.15.60:0/2707502794 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fc96c0efae0 con 0x7fc97c101530 2024-03-04T12:27:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.436+0000 7fc97a7fc640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc9640a2070 0x7fc9640a4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc97a7fc640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc9640a2070 0x7fc9640a4530 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0750c0 tx=0x7fc970009260 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc983fa2740 1 Processor -- start 2024-03-04T12:27:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc983fa2740 1 -- start start 2024-03-04T12:27:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc983fa2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 0x55668f845db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc983fa2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f8462f0 0x55668f84e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc983fa2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f84eed0 0x55668f8512c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc983fa2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55668f80cdc0 con 0x55668f8459d0 2024-03-04T12:27:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc983fa2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55668f7a8ec0 con 0x55668f84eed0 2024-03-04T12:27:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc983fa2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55668f751d30 con 0x55668f8462f0 2024-03-04T12:27:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc97affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 0x55668f845db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc97affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 0x55668f845db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52900/0 (socket says 172.21.15.60:52900) 2024-03-04T12:27:54.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc97affd640 1 -- 172.21.15.60:0/2521789874 learned_addr learned my addr 172.21.15.60:0/2521789874 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc97a7fc640 1 --2- 172.21.15.60:0/2521789874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f8462f0 0x55668f84e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc97b7fe640 1 --2- 172.21.15.60:0/2521789874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f84eed0 0x55668f8512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc97affd640 1 -- 172.21.15.60:0/2521789874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f8462f0 msgr2=0x55668f84e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc97affd640 1 --2- 172.21.15.60:0/2521789874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f8462f0 0x55668f84e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc97affd640 1 -- 172.21.15.60:0/2521789874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f84eed0 msgr2=0x55668f8512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.437+0000 7fc97affd640 1 --2- 172.21.15.60:0/2521789874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f84eed0 0x55668f8512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.438+0000 7fc97affd640 1 -- 172.21.15.60:0/2521789874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55668f4bd3f0 con 0x55668f8459d0 2024-03-04T12:27:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.438+0000 7fc97affd640 1 --2- 172.21.15.60:0/2521789874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 0x55668f845db0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009830 tx=0x7fc96c0bbe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d838176d343b153 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.438+0000 7fc9617fa640 1 -- 172.21.15.60:0/2521789874 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0bc940 con 0x55668f8459d0 2024-03-04T12:27:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.438+0000 7fc9617fa640 1 -- 172.21.15.60:0/2521789874 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc96c0bc240 con 0x55668f8459d0 2024-03-04T12:27:54.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.438+0000 7fc9617fa640 1 -- 172.21.15.60:0/2521789874 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c032d50 con 0x55668f8459d0 2024-03-04T12:27:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.439+0000 7fc983fa2740 1 -- 172.21.15.60:0/2521789874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 msgr2=0x55668f845db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.439+0000 7fc983fa2740 1 --2- 172.21.15.60:0/2521789874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 0x55668f845db0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009830 tx=0x7fc96c0bbe40 comp rx=0 tx=0).stop 2024-03-04T12:27:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.439+0000 7fc983fa2740 1 -- 172.21.15.60:0/2521789874 shutdown_connections 2024-03-04T12:27:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.439+0000 7fc983fa2740 1 --2- 172.21.15.60:0/2521789874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f84eed0 0x55668f8512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.439+0000 7fc983fa2740 1 --2- 172.21.15.60:0/2521789874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f8462f0 0x55668f84e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.439+0000 7fc983fa2740 1 --2- 172.21.15.60:0/2521789874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 0x55668f845db0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.439+0000 7fc983fa2740 1 -- 172.21.15.60:0/2521789874 >> 172.21.15.60:0/2521789874 conn(0x55668f840100 msgr2=0x55668f8404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.439+0000 7fc983fa2740 1 -- 172.21.15.60:0/2521789874 shutdown_connections 2024-03-04T12:27:54.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.439+0000 7fc983fa2740 1 -- 172.21.15.60:0/2521789874 wait complete. 2024-03-04T12:27:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52c1ffb640 1 -- 172.21.15.60:0/1029499917 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fc9da2be80 con 0x55fc9db30820 2024-03-04T12:27:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52c1ffb640 10 client.25984 handle_command_reply: tid=0 2024-03-04T12:27:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52e3a02740 1 client.25984 shutdown 2024-03-04T12:27:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52e3a02740 1 -- 172.21.15.60:0/1029499917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 msgr2=0x55fc9dad0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52e3a02740 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dad0360 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0badb0 tx=0x7f52cc000bc0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52e3a02740 1 -- 172.21.15.60:0/1029499917 shutdown_connections 2024-03-04T12:27:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52e3a02740 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x55fc9db30820 0x55fc9db32c10 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52e3a02740 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55fc9dac98e0 0x55fc9dacea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52e3a02740 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc9dac10d0 0x55fc9dad08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52c1ffb640 0 client.25984 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52e3a02740 1 --2- 172.21.15.60:0/1029499917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fc9dac0360 0x55fc9dad0360 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52e3a02740 1 -- 172.21.15.60:0/1029499917 >> 172.21.15.60:0/1029499917 conn(0x55fc9da87b90 msgr2=0x55fc9dac1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.439+0000 7fc983fa2740 1 -- start start 2024-03-04T12:27:54.455 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52e3a02740 1 -- 172.21.15.60:0/1029499917 shutdown_connections 2024-03-04T12:27:54.455 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52e3a02740 1 -- 172.21.15.60:0/1029499917 wait complete. 2024-03-04T12:27:54.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.439+0000 7fc983fa2740 1 Processor -- start 2024-03-04T12:27:54.455 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.439+0000 7f52e3a02740 20 client.25984 trim_cache size 0 max 16384 2024-03-04T12:27:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:{ 2024-03-04T12:27:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "message": "", 2024-03-04T12:27:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout: "return_code": 0 2024-03-04T12:27:54.456 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stdout:} 2024-03-04T12:27:54.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc983fa2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f8459d0 0x55668f84bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc983fa2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8462f0 0x55668f84c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc983fa2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f84eed0 0x55668f84c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc983fa2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55668f7474a0 con 0x55668f84eed0 2024-03-04T12:27:54.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc97affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f8459d0 0x55668f84bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc97affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f8459d0 0x55668f84bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:59864/0 (socket says 172.21.15.60:59864) 2024-03-04T12:27:54.457 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc97affd640 1 -- 172.21.15.60:0/698922539 learned_addr learned my addr 172.21.15.60:0/698922539 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.458 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.440+0000 7f52e0e20640 1 -- 172.21.15.60:0/975251074 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f52c40a1ee0 msgr2=0x7f52c40a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.458 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.440+0000 7f52e0e20640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f52c40a1ee0 0x7f52c40a43a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f52d4005fd0 tx=0x7f52d4009ea0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.458 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.440+0000 7f52e0e20640 1 -- 172.21.15.60:0/975251074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc068e40 msgr2=0x7f52dc19d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc97a7fc640 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8462f0 0x55668f84c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc983fa2740 1 -- 172.21.15.60:0/698922539 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55668f7a4f60 con 0x55668f8462f0 2024-03-04T12:27:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc983fa2740 1 -- 172.21.15.60:0/698922539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55668f851a80 con 0x55668f8459d0 2024-03-04T12:27:54.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc97b7fe640 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f84eed0 0x55668f84c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.459 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.440+0000 7f52e0e20640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc068e40 0x7f52dc19d820 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f52cc02c240 tx=0x7f52cc0042f0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.459 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.440+0000 7f52e0e20640 1 -- 172.21.15.60:0/975251074 shutdown_connections 2024-03-04T12:27:54.459 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.440+0000 7f52e0e20640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f52c40a1ee0 0x7f52c40a43a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.459 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.440+0000 7f52e0e20640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f52dc1979d0 0x7f52dc197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.459 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.440+0000 7f52e0e20640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52dc106610 0x7f52dc19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.459 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.440+0000 7f52e0e20640 1 --2- 172.21.15.60:0/975251074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52dc068e40 0x7f52dc19d820 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.440+0000 7f52e0e20640 1 -- 172.21.15.60:0/975251074 >> 172.21.15.60:0/975251074 conn(0x7f52dc077320 msgr2=0x7f52dc0fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.440+0000 7f52e0e20640 1 -- 172.21.15.60:0/975251074 shutdown_connections 2024-03-04T12:27:54.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc983fa2740 10 client.0 _ll_register_callbacks cb 0x55668f7a31c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:54.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc97affd640 1 -- 172.21.15.60:0/698922539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8462f0 msgr2=0x55668f84c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc97affd640 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8462f0 0x55668f84c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.440+0000 7fc97affd640 1 -- 172.21.15.60:0/698922539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f84eed0 msgr2=0x55668f84c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi060.front.sepia.ceph.com.smithi060.stderr:2024-03-04T12:27:54.441+0000 7f52e0e20640 1 -- 172.21.15.60:0/975251074 wait complete. 2024-03-04T12:27:54.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.441+0000 7fc97affd640 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f84eed0 0x55668f84c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.441+0000 7fc97affd640 1 -- 172.21.15.60:0/698922539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55668f5877d0 con 0x55668f8459d0 2024-03-04T12:27:54.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.441+0000 7fc97b7fe640 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f84eed0 0x55668f84c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.441+0000 7fc983fa2740 1 -- 172.21.15.60:0/698922539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc96c0bc240 con 0x55668f8459d0 2024-03-04T12:27:54.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.441+0000 7fc97affd640 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f8459d0 0x55668f84bf40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009980 tx=0x7fc96c0ed340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c4041472bfbeace server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.441+0000 7fc95eff5640 10 client.25996 ms_handle_connect on v2:172.21.15.86:3300/0 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.441+0000 7fc95eff5640 1 -- 172.21.15.60:0/698922539 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0cc050 con 0x55668f8459d0 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.441+0000 7fc95eff5640 1 -- 172.21.15.60:0/698922539 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fc96c034640 con 0x55668f8459d0 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.441+0000 7fc95eff5640 1 -- 172.21.15.60:0/698922539 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc96c0d8070 con 0x55668f8459d0 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.441+0000 7fc983fa2740 10 client.25996 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.441+0000 7fc983fa2740 1 -- 172.21.15.60:0/698922539 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({fsmap=36}) v3 -- 0x55668f85da30 con 0x55668f8459d0 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.441+0000 7fc95eff5640 1 -- 172.21.15.60:0/698922539 <== mon.1 v2:172.21.15.86:3300/0 4 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fc96c0cf020 con 0x55668f8459d0 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.442+0000 7fc983fa2740 10 client.25996 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.442+0000 7fc983fa2740 10 client.25996 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.442+0000 7fc983fa2740 20 client.25996 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.442+0000 7fc983fa2740 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55668f85ed80 0x55668f861170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.442+0000 7fc983fa2740 4 client.25996 mds_command: new command op to 24407 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:54.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.442+0000 7fc983fa2740 1 -- 172.21.15.60:0/698922539 --> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55668f7b6400 con 0x55668f85ed80 2024-03-04T12:27:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.442+0000 7fc97a7fc640 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55668f85ed80 0x55668f861170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.442+0000 7fc95eff5640 10 client.25996 ms_handle_connect on v2:172.21.15.63:6832/2270964219 2024-03-04T12:27:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.442+0000 7fc97a7fc640 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55668f85ed80 0x55668f861170 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=862b5143f31d9ed3 server_cookie=2ab3517a9a87ec02 in_seq=0 out_seq=0 2024-03-04T12:27:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc95eff5640 1 -- 172.21.15.60:0/698922539 <== mds.3 v2:172.21.15.63:6832/2270964219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55668f7b6400 con 0x55668f85ed80 2024-03-04T12:27:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc95eff5640 10 client.25996 handle_command_reply: tid=0 2024-03-04T12:27:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc983fa2740 1 client.25996 shutdown 2024-03-04T12:27:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc983fa2740 1 -- 172.21.15.60:0/698922539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f8459d0 msgr2=0x55668f84bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc983fa2740 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f8459d0 0x55668f84bf40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009980 tx=0x7fc96c0ed340 comp rx=0 tx=0).stop 2024-03-04T12:27:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc983fa2740 1 -- 172.21.15.60:0/698922539 shutdown_connections 2024-03-04T12:27:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc983fa2740 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55668f85ed80 0x55668f861170 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc983fa2740 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f84eed0 0x55668f84c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc95eff5640 0 client.25996 ms_handle_reset on v2:172.21.15.63:6832/2270964219 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc983fa2740 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8462f0 0x55668f84c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc983fa2740 1 --2- 172.21.15.60:0/698922539 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f8459d0 0x55668f84bf40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc983fa2740 1 -- 172.21.15.60:0/698922539 >> 172.21.15.60:0/698922539 conn(0x55668f840100 msgr2=0x55668f841e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc983fa2740 1 -- 172.21.15.60:0/698922539 shutdown_connections 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc983fa2740 1 -- 172.21.15.60:0/698922539 wait complete. 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.443+0000 7fc983fa2740 20 client.25996 trim_cache size 0 max 16384 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.455+0000 7fc983fa2740 1 Processor -- start 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.455+0000 7fc983fa2740 1 -- start start 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.455+0000 7fc983fa2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 0x55668f84e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.455+0000 7fc983fa2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8462f0 0x55668f84a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.455+0000 7fc983fa2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f84eed0 0x55668f84ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.455+0000 7fc983fa2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c1b0610 con 0x55668f8459d0 2024-03-04T12:27:54.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.455+0000 7fc983fa2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c1b1ff0 con 0x55668f8462f0 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.455+0000 7fc983fa2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c1b0c70 con 0x55668f84eed0 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc97affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 0x55668f84e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc97b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f84eed0 0x55668f84ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc97affd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 0x55668f84e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52914/0 (socket says 172.21.15.60:52914) 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc97affd640 1 -- 172.21.15.60:0/2010873293 learned_addr learned my addr 172.21.15.60:0/2010873293 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc97a7fc640 1 --2- 172.21.15.60:0/2010873293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8462f0 0x55668f84a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc97affd640 1 -- 172.21.15.60:0/2010873293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f84eed0 msgr2=0x55668f84ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc97affd640 1 --2- 172.21.15.60:0/2010873293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f84eed0 0x55668f84ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc97affd640 1 -- 172.21.15.60:0/2010873293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8462f0 msgr2=0x55668f84a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc97affd640 1 --2- 172.21.15.60:0/2010873293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8462f0 0x55668f84a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc97affd640 1 -- 172.21.15.60:0/2010873293 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55668f847b40 con 0x55668f8459d0 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc97b7fe640 1 --2- 172.21.15.60:0/2010873293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f84eed0 0x55668f84ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc97affd640 1 --2- 172.21.15.60:0/2010873293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 0x55668f84e670 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc96c005b60 tx=0x7fc96c0d5000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33872b8b10857db server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc960ff9640 1 -- 172.21.15.60:0/2010873293 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0cc070 con 0x55668f8459d0 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc960ff9640 1 -- 172.21.15.60:0/2010873293 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc96c0bc240 con 0x55668f8459d0 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.456+0000 7fc960ff9640 1 -- 172.21.15.60:0/2010873293 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0bc940 con 0x55668f8459d0 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.457+0000 7fc983fa2740 1 -- 172.21.15.60:0/2010873293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 msgr2=0x55668f84e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.457+0000 7fc983fa2740 1 --2- 172.21.15.60:0/2010873293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 0x55668f84e670 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fc96c005b60 tx=0x7fc96c0d5000 comp rx=0 tx=0).stop 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.457+0000 7fc983fa2740 1 -- 172.21.15.60:0/2010873293 shutdown_connections 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.457+0000 7fc983fa2740 1 --2- 172.21.15.60:0/2010873293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f84eed0 0x55668f84ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.457+0000 7fc983fa2740 1 --2- 172.21.15.60:0/2010873293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8462f0 0x55668f84a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.457+0000 7fc983fa2740 1 --2- 172.21.15.60:0/2010873293 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8459d0 0x55668f84e670 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.457+0000 7fc983fa2740 1 -- 172.21.15.60:0/2010873293 >> 172.21.15.60:0/2010873293 conn(0x55668f840100 msgr2=0x55668f865990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.457+0000 7fc983fa2740 1 -- 172.21.15.60:0/2010873293 shutdown_connections 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.457+0000 7fc983fa2740 1 -- 172.21.15.60:0/2010873293 wait complete. 2024-03-04T12:27:54.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc983fa2740 1 -- start start 2024-03-04T12:27:54.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc983fa2740 1 Processor -- start 2024-03-04T12:27:54.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc983fa2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8459d0 0x55668f855910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc983fa2740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8462f0 0x55668f855e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc983fa2740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f84eed0 0x55668f853d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc983fa2740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c1b1990 con 0x55668f8462f0 2024-03-04T12:27:54.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc983fa2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c1b12d0 con 0x55668f8459d0 2024-03-04T12:27:54.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc983fa2740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c1ae8b0 con 0x55668f84eed0 2024-03-04T12:27:54.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc983fa2740 10 client.0 _ll_register_callbacks cb 0x7fc96c005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:54.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc97affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8459d0 0x55668f855910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc97a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8462f0 0x55668f855e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc97affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8459d0 0x55668f855910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:44376/0 (socket says 172.21.15.60:44376) 2024-03-04T12:27:54.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc97a7fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8462f0 0x55668f855e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52930/0 (socket says 172.21.15.60:52930) 2024-03-04T12:27:54.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc97affd640 1 -- 172.21.15.60:0/346636501 learned_addr learned my addr 172.21.15.60:0/346636501 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc97b7fe640 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f84eed0 0x55668f853d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc97affd640 1 -- 172.21.15.60:0/346636501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f84eed0 msgr2=0x55668f853d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc97affd640 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f84eed0 0x55668f853d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc97affd640 1 -- 172.21.15.60:0/346636501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8462f0 msgr2=0x55668f855e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc97affd640 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8462f0 0x55668f855e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc97affd640 1 -- 172.21.15.60:0/346636501 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55668f854540 con 0x55668f8459d0 2024-03-04T12:27:54.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.458+0000 7fc97a7fc640 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8462f0 0x55668f855e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc97affd640 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8459d0 0x55668f855910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009830 tx=0x7fc96c0c4780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b771a56c2e82dea server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc95eff5640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-03-04T12:27:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc95eff5640 1 -- 172.21.15.60:0/346636501 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc96c09d020 con 0x55668f8459d0 2024-03-04T12:27:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc983fa2740 1 -- 172.21.15.60:0/346636501 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc96c0bc240 con 0x55668f8459d0 2024-03-04T12:27:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc95eff5640 1 -- 172.21.15.60:0/346636501 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fc96c034640 con 0x55668f8459d0 2024-03-04T12:27:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc95eff5640 1 -- 172.21.15.60:0/346636501 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc96c0cc070 con 0x55668f8459d0 2024-03-04T12:27:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc983fa2740 10 client.25904 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc983fa2740 1 -- 172.21.15.60:0/346636501 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55668f854540 con 0x55668f8459d0 2024-03-04T12:27:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc95eff5640 1 -- 172.21.15.60:0/346636501 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc96c0bc240 con 0x55668f8459d0 2024-03-04T12:27:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc95eff5640 1 -- 172.21.15.60:0/346636501 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fc96c0cf020 con 0x55668f8459d0 2024-03-04T12:27:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc983fa2740 10 client.25904 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc983fa2740 10 client.25904 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.k 2024-03-04T12:27:54.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc983fa2740 20 client.25904 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc983fa2740 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55668f8b5e60 0x55668f8b8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc983fa2740 4 client.25904 mds_command: new command op to 24407 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.459+0000 7fc983fa2740 1 -- 172.21.15.60:0/346636501 --> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55668f7b0750 con 0x55668f8b5e60 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc97a7fc640 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55668f8b5e60 0x55668f8b8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc97a7fc640 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55668f8b5e60 0x55668f8b8250 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4113faf40932afe2 server_cookie=dc4b4ecd371b625c in_seq=0 out_seq=0 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc95eff5640 10 client.25904 ms_handle_connect on v2:172.21.15.63:6832/2270964219 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc95eff5640 1 -- 172.21.15.60:0/346636501 <== mds.3 v2:172.21.15.63:6832/2270964219 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55668f7b0750 con 0x55668f8b5e60 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc95eff5640 10 client.25904 handle_command_reply: tid=0 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc983fa2740 1 client.25904 shutdown 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc983fa2740 1 -- 172.21.15.60:0/346636501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8459d0 msgr2=0x55668f855910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc983fa2740 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8459d0 0x55668f855910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc96c009830 tx=0x7fc96c0c4780 comp rx=0 tx=0).stop 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc983fa2740 1 -- 172.21.15.60:0/346636501 shutdown_connections 2024-03-04T12:27:54.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc983fa2740 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.63:6832/2270964219,v1:172.21.15.63:6833/2270964219] conn(0x55668f8b5e60 0x55668f8b8250 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc983fa2740 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55668f84eed0 0x55668f853d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc983fa2740 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55668f8462f0 0x55668f855e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc95eff5640 0 client.25904 ms_handle_reset on v2:172.21.15.63:6832/2270964219 2024-03-04T12:27:54.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc983fa2740 1 --2- 172.21.15.60:0/346636501 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55668f8459d0 0x55668f855910 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc983fa2740 1 -- 172.21.15.60:0/346636501 >> 172.21.15.60:0/346636501 conn(0x55668f840100 msgr2=0x55668f84a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc983fa2740 1 -- 172.21.15.60:0/346636501 shutdown_connections 2024-03-04T12:27:54.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc983fa2740 1 -- 172.21.15.60:0/346636501 wait complete. 2024-03-04T12:27:54.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.460+0000 7fc983fa2740 20 client.25904 trim_cache size 0 max 16384 2024-03-04T12:27:54.481 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:27:54.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.461+0000 7fc9813c4640 1 -- 172.21.15.60:0/2707502794 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc9640a2070 msgr2=0x7fc9640a4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.461+0000 7fc9813c4640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc9640a2070 0x7fc9640a4530 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0750c0 tx=0x7fc970009260 comp rx=0 tx=0).stop 2024-03-04T12:27:54.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.461+0000 7fc9813c4640 1 -- 172.21.15.60:0/2707502794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c101530 msgr2=0x7fc97c073630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.461+0000 7fc9813c4640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c101530 0x7fc97c073630 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc96c005e00 tx=0x7fc96c0045d0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.461+0000 7fc9813c4640 1 -- 172.21.15.60:0/2707502794 shutdown_connections 2024-03-04T12:27:54.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.461+0000 7fc9813c4640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fc9640a2070 0x7fc9640a4530 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.461+0000 7fc9813c4640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fc97c0740b0 0x7fc97c1a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.461+0000 7fc9813c4640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc97c106600 0x7fc97c073b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.461+0000 7fc9813c4640 1 --2- 172.21.15.60:0/2707502794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c101530 0x7fc97c073630 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.461+0000 7fc9813c4640 1 -- 172.21.15.60:0/2707502794 >> 172.21.15.60:0/2707502794 conn(0x7fc97c0fcdd0 msgr2=0x7fc97c0fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.461+0000 7fc9813c4640 1 -- 172.21.15.60:0/2707502794 shutdown_connections 2024-03-04T12:27:54.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.461+0000 7fc9813c4640 1 -- 172.21.15.60:0/2707502794 wait complete. 2024-03-04T12:27:54.483 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:27:54.483 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-04T12:27:54.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.622+0000 7fa6bbc92640 1 -- 172.21.15.60:0/3702922972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b4100440 msgr2=0x7fa6b4100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.622+0000 7fa6bbc92640 1 --2- 172.21.15.60:0/3702922972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b4100440 0x7fa6b4100820 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009900 tx=0x7fa6b002bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:54.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.622+0000 7fa6bbc92640 1 -- 172.21.15.60:0/3702922972 shutdown_connections 2024-03-04T12:27:54.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.622+0000 7fa6bbc92640 1 --2- 172.21.15.60:0/3702922972 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6b4105510 0x7fa6b410d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.622+0000 7fa6bbc92640 1 --2- 172.21.15.60:0/3702922972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6b4100df0 0x7fa6b4104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.622+0000 7fa6bbc92640 1 --2- 172.21.15.60:0/3702922972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b4100440 0x7fa6b4100820 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.622+0000 7fa6bbc92640 1 -- 172.21.15.60:0/3702922972 >> 172.21.15.60:0/3702922972 conn(0x7fa6b40773e0 msgr2=0x7fa6b40777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.622+0000 7fa6bbc92640 1 -- 172.21.15.60:0/3702922972 shutdown_connections 2024-03-04T12:27:54.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.622+0000 7fa6bbc92640 1 -- 172.21.15.60:0/3702922972 wait complete. 2024-03-04T12:27:54.631 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.622+0000 7fa6bbc92640 1 Processor -- start 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.622+0000 7fa6bbc92640 1 -- start start 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6bbc92640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6b4100440 0x7fa6b419d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6bbc92640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b4105510 0x7fa6b419dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6bbc92640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6b4197a30 0x7fa6b4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6bbc92640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b41105a0 con 0x7fa6b4100440 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6bbc92640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b4110420 con 0x7fa6b4197a30 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6bbc92640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b4110720 con 0x7fa6b4105510 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6b9a2e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6b4100440 0x7fa6b419d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6b9a2e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6b4100440 0x7fa6b419d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52956/0 (socket says 172.21.15.60:52956) 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6b9a2e640 1 -- 172.21.15.60:0/605718443 learned_addr learned my addr 172.21.15.60:0/605718443 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6b922d640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b4105510 0x7fa6b419dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6ba22f640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6b4197a30 0x7fa6b4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6b9a2e640 1 -- 172.21.15.60:0/605718443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b4105510 msgr2=0x7fa6b419dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.632 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6b9a2e640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b4105510 0x7fa6b419dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6b9a2e640 1 -- 172.21.15.60:0/605718443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6b4197a30 msgr2=0x7fa6b4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6b9a2e640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6b4197a30 0x7fa6b4197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6b9a2e640 1 -- 172.21.15.60:0/605718443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b4198550 con 0x7fa6b4100440 2024-03-04T12:27:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6b922d640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b4105510 0x7fa6b419dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6ba22f640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6b4197a30 0x7fa6b4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.623+0000 7fa6b9a2e640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6b4100440 0x7fa6b419d830 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0006010 tx=0x7fa6b002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.624+0000 7fa6aaffd640 1 -- 172.21.15.60:0/605718443 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0041070 con 0x7fa6b4100440 2024-03-04T12:27:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.624+0000 7fa6aaffd640 1 -- 172.21.15.60:0/605718443 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6b0004450 con 0x7fa6b4100440 2024-03-04T12:27:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.624+0000 7fa6bbc92640 1 -- 172.21.15.60:0/605718443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b41987e0 con 0x7fa6b4100440 2024-03-04T12:27:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.624+0000 7fa6aaffd640 1 -- 172.21.15.60:0/605718443 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0003e00 con 0x7fa6b4100440 2024-03-04T12:27:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.624+0000 7fa6bbc92640 1 -- 172.21.15.60:0/605718443 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b41a49e0 con 0x7fa6b4100440 2024-03-04T12:27:54.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.626+0000 7fa6be870740 1 Processor -- start 2024-03-04T12:27:54.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6be870740 1 -- start start 2024-03-04T12:27:54.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6be870740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 0x557b46ee7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6be870740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ee82f0 0x557b46ef0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6be870740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ef0ed0 0x557b46ef32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6be870740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b46eaedc0 con 0x557b46ee79d0 2024-03-04T12:27:54.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6be870740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b46e4aec0 con 0x557b46ef0ed0 2024-03-04T12:27:54.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6be870740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b46df3d30 con 0x557b46ee82f0 2024-03-04T12:27:54.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6ba22f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 0x557b46ee7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6b922d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ef0ed0 0x557b46ef32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6ba22f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 0x557b46ee7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52968/0 (socket says 172.21.15.60:52968) 2024-03-04T12:27:54.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6ba22f640 1 -- 172.21.15.60:0/569031523 learned_addr learned my addr 172.21.15.60:0/569031523 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6aaffd640 1 -- 172.21.15.60:0/605718443 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7fa6b0044070 con 0x7fa6b4100440 2024-03-04T12:27:54.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6b922d640 1 -- 172.21.15.60:0/569031523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ee82f0 msgr2=0x557b46ef0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6b922d640 1 --2- 172.21.15.60:0/569031523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ee82f0 0x557b46ef0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6b922d640 1 -- 172.21.15.60:0/569031523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 msgr2=0x557b46ee7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6b922d640 1 --2- 172.21.15.60:0/569031523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 0x557b46ee7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6b922d640 1 -- 172.21.15.60:0/569031523 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b46b5f3f0 con 0x557b46ef0ed0 2024-03-04T12:27:54.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.627+0000 7fa6ba22f640 1 --2- 172.21.15.60:0/569031523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 0x557b46ee7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.628+0000 7fa6b922d640 1 --2- 172.21.15.60:0/569031523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ef0ed0 0x557b46ef32c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa6b41991d0 tx=0x7fa6a4010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f51aea659fc1168 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.628+0000 7fa6aaffd640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa6a00a1f70 0x7fa6a00a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.628+0000 7fa69ffff640 1 -- 172.21.15.60:0/569031523 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a40117c0 con 0x557b46ef0ed0 2024-03-04T12:27:54.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.628+0000 7fa69ffff640 1 -- 172.21.15.60:0/569031523 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6a4011e40 con 0x557b46ef0ed0 2024-03-04T12:27:54.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.628+0000 7fa69ffff640 1 -- 172.21.15.60:0/569031523 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a401ad80 con 0x557b46ef0ed0 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.628+0000 7fa6ba22f640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa6a00a1f70 0x7fa6a00a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.628+0000 7fa6aaffd640 1 -- 172.21.15.60:0/605718443 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fa6b00effb0 con 0x7fa6b4100440 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.629+0000 7fa6be870740 1 -- 172.21.15.60:0/569031523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ef0ed0 msgr2=0x557b46ef32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.629+0000 7fa6ba22f640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa6a00a1f70 0x7fa6a00a4430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009fd0 tx=0x7fa6ac009500 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.629+0000 7fa6be870740 1 --2- 172.21.15.60:0/569031523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ef0ed0 0x557b46ef32c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa6b41991d0 tx=0x7fa6a4010860 comp rx=0 tx=0).stop 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.629+0000 7fa6be870740 1 -- 172.21.15.60:0/569031523 shutdown_connections 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.629+0000 7fa6be870740 1 --2- 172.21.15.60:0/569031523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ef0ed0 0x557b46ef32c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.629+0000 7fa6be870740 1 --2- 172.21.15.60:0/569031523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ee82f0 0x557b46ef0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.629+0000 7fa6be870740 1 --2- 172.21.15.60:0/569031523 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 0x557b46ee7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.629+0000 7fa6be870740 1 -- 172.21.15.60:0/569031523 >> 172.21.15.60:0/569031523 conn(0x557b46ee2100 msgr2=0x557b46ee24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.629+0000 7fa6be870740 1 -- 172.21.15.60:0/569031523 shutdown_connections 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.629+0000 7fa6be870740 1 -- 172.21.15.60:0/569031523 wait complete. 2024-03-04T12:27:54.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.629+0000 7fa6be870740 1 -- start start 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.629+0000 7fa6be870740 1 Processor -- start 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6be870740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ee79d0 0x557b46eedf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6be870740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee82f0 0x557b46eee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6be870740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ef0ed0 0x557b46eeea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6be870740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b46de94a0 con 0x557b46ef0ed0 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6be870740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b46e46f60 con 0x557b46ee82f0 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6be870740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b46ef3cc0 con 0x557b46ee79d0 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6be870740 10 client.0 _ll_register_callbacks cb 0x557b46e451c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6b922d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee82f0 0x557b46eee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6ba22f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ef0ed0 0x557b46eeea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6b9a2e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ee79d0 0x557b46eedf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6b922d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee82f0 0x557b46eee4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.60:44398/0 (socket says 172.21.15.60:44398) 2024-03-04T12:27:54.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6b922d640 1 -- 172.21.15.60:0/2255733813 learned_addr learned my addr 172.21.15.60:0/2255733813 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6b9a2e640 1 -- 172.21.15.60:0/2255733813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee82f0 msgr2=0x557b46eee4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6b9a2e640 1 --2- 172.21.15.60:0/2255733813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee82f0 0x557b46eee4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6b9a2e640 1 -- 172.21.15.60:0/2255733813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ef0ed0 msgr2=0x557b46eeea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6b9a2e640 1 --2- 172.21.15.60:0/2255733813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ef0ed0 0x557b46eeea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6b9a2e640 1 -- 172.21.15.60:0/2255733813 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b46c297d0 con 0x557b46ee79d0 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.630+0000 7fa6b922d640 1 --2- 172.21.15.60:0/2255733813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee82f0 0x557b46eee4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa6b9a2e640 1 --2- 172.21.15.60:0/2255733813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ee79d0 0x557b46eedf90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa6b00ed470 tx=0x7fa6b00bc4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85c7120b09c92cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa69d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa69d7fa640 1 -- 172.21.15.60:0/2255733813 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b004b070 con 0x557b46ee79d0 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa69d7fa640 1 -- 172.21.15.60:0/2255733813 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fa6b00b8d80 con 0x557b46ee79d0 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa6be870740 1 -- 172.21.15.60:0/2255733813 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557b46e58400 con 0x557b46ee79d0 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa69d7fa640 1 -- 172.21.15.60:0/2255733813 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6b002cd20 con 0x557b46ee79d0 2024-03-04T12:27:54.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa6be870740 10 client.26011 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa6be870740 1 -- 172.21.15.60:0/2255733813 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x557b46effea0 con 0x557b46ee79d0 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa69d7fa640 1 -- 172.21.15.60:0/2255733813 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557b46e58400 con 0x557b46ee79d0 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa69d7fa640 1 -- 172.21.15.60:0/2255733813 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa6b00f2110 con 0x557b46ee79d0 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa6be870740 10 client.26011 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa6be870740 10 client.26011 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa6be870740 20 client.26011 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa6be870740 1 --2- 172.21.15.60:0/2255733813 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x557b46f011f0 0x557b46f03610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa6be870740 4 client.26011 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.631+0000 7fa6be870740 1 -- 172.21.15.60:0/2255733813 --> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b46e52750 con 0x557b46f011f0 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.632+0000 7fa6b922d640 1 --2- 172.21.15.60:0/2255733813 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x557b46f011f0 0x557b46f03610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.632+0000 7fa6b922d640 1 --2- 172.21.15.60:0/2255733813 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x557b46f011f0 0x557b46f03610 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a95034037bcf8e60 server_cookie=42cc53855dce7e8f in_seq=0 out_seq=0 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.632+0000 7fa69d7fa640 10 client.26011 ms_handle_connect on v2:172.21.15.63:6836/3974574804 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa69d7fa640 1 -- 172.21.15.60:0/2255733813 <== mds.2 v2:172.21.15.63:6836/3974574804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557b46e52750 con 0x557b46f011f0 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa69d7fa640 10 client.26011 handle_command_reply: tid=0 2024-03-04T12:27:54.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa6be870740 1 client.26011 shutdown 2024-03-04T12:27:54.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa6be870740 1 -- 172.21.15.60:0/2255733813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ee79d0 msgr2=0x557b46eedf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa6be870740 1 --2- 172.21.15.60:0/2255733813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ee79d0 0x557b46eedf90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa6b00ed470 tx=0x7fa6b00bc4e0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa6be870740 1 -- 172.21.15.60:0/2255733813 shutdown_connections 2024-03-04T12:27:54.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa6be870740 1 --2- 172.21.15.60:0/2255733813 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x557b46f011f0 0x557b46f03610 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa6be870740 1 --2- 172.21.15.60:0/2255733813 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ef0ed0 0x557b46eeea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa6be870740 1 --2- 172.21.15.60:0/2255733813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee82f0 0x557b46eee4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa69d7fa640 0 client.26011 ms_handle_reset on v2:172.21.15.63:6836/3974574804 2024-03-04T12:27:54.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa6be870740 1 --2- 172.21.15.60:0/2255733813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ee79d0 0x557b46eedf90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa6be870740 1 -- 172.21.15.60:0/2255733813 >> 172.21.15.60:0/2255733813 conn(0x557b46ee2100 msgr2=0x557b46ee3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa6be870740 1 -- 172.21.15.60:0/2255733813 shutdown_connections 2024-03-04T12:27:54.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa6be870740 1 -- 172.21.15.60:0/2255733813 wait complete. 2024-03-04T12:27:54.649 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.633+0000 7fa6be870740 20 client.26011 trim_cache size 0 max 16384 2024-03-04T12:27:54.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.645+0000 7fa6be870740 1 Processor -- start 2024-03-04T12:27:54.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.645+0000 7fa6be870740 1 -- start start 2024-03-04T12:27:54.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.645+0000 7fa6be870740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 0x557b46ef06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.645+0000 7fa6be870740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee82f0 0x557b46eec880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.645+0000 7fa6be870740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ef0ed0 0x557b46eecdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.645+0000 7fa6be870740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b41ae420 con 0x557b46ee79d0 2024-03-04T12:27:54.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.645+0000 7fa6be870740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b41afe00 con 0x557b46ee82f0 2024-03-04T12:27:54.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.645+0000 7fa6be870740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b41aea80 con 0x557b46ef0ed0 2024-03-04T12:27:54.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa6b9a2e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 0x557b46ef06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.654 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa6b9a2e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 0x557b46ef06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52998/0 (socket says 172.21.15.60:52998) 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa6b9a2e640 1 -- 172.21.15.60:0/2676703701 learned_addr learned my addr 172.21.15.60:0/2676703701 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa6ba22f640 1 --2- 172.21.15.60:0/2676703701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ef0ed0 0x557b46eecdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa6b922d640 1 --2- 172.21.15.60:0/2676703701 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee82f0 0x557b46eec880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa6b9a2e640 1 -- 172.21.15.60:0/2676703701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ef0ed0 msgr2=0x557b46eecdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa6b9a2e640 1 --2- 172.21.15.60:0/2676703701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ef0ed0 0x557b46eecdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa6b9a2e640 1 -- 172.21.15.60:0/2676703701 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee82f0 msgr2=0x557b46eec880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa6b9a2e640 1 --2- 172.21.15.60:0/2676703701 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee82f0 0x557b46eec880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa6b9a2e640 1 -- 172.21.15.60:0/2676703701 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b46f08840 con 0x557b46ee79d0 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa6ba22f640 1 --2- 172.21.15.60:0/2676703701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ef0ed0 0x557b46eecdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa6b9a2e640 1 --2- 172.21.15.60:0/2676703701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 0x557b46ef06c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa6b00c4f30 tx=0x7fa6b002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68882a40e4fd2d42 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa69f7fe640 1 -- 172.21.15.60:0/2676703701 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b004b070 con 0x557b46ee79d0 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa69f7fe640 1 -- 172.21.15.60:0/2676703701 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6b00bce10 con 0x557b46ee79d0 2024-03-04T12:27:54.655 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.646+0000 7fa69f7fe640 1 -- 172.21.15.60:0/2676703701 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b00dc0b0 con 0x557b46ee79d0 2024-03-04T12:27:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.647+0000 7fa6be870740 1 -- 172.21.15.60:0/2676703701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 msgr2=0x557b46ef06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.647+0000 7fa6be870740 1 --2- 172.21.15.60:0/2676703701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 0x557b46ef06c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa6b00c4f30 tx=0x7fa6b002bc30 comp rx=0 tx=0).stop 2024-03-04T12:27:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.647+0000 7fa6be870740 1 -- 172.21.15.60:0/2676703701 shutdown_connections 2024-03-04T12:27:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.647+0000 7fa6be870740 1 --2- 172.21.15.60:0/2676703701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ef0ed0 0x557b46eecdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.647+0000 7fa6be870740 1 --2- 172.21.15.60:0/2676703701 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee82f0 0x557b46eec880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.647+0000 7fa6be870740 1 --2- 172.21.15.60:0/2676703701 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee79d0 0x557b46ef06c0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.647+0000 7fa6be870740 1 -- 172.21.15.60:0/2676703701 >> 172.21.15.60:0/2676703701 conn(0x557b46ee2100 msgr2=0x557b46ee92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.647+0000 7fa6be870740 1 -- 172.21.15.60:0/2676703701 shutdown_connections 2024-03-04T12:27:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.647+0000 7fa6be870740 1 -- 172.21.15.60:0/2676703701 wait complete. 2024-03-04T12:27:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.647+0000 7fa6be870740 1 -- start start 2024-03-04T12:27:54.656 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.647+0000 7fa6be870740 1 Processor -- start 2024-03-04T12:27:54.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6be870740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee79d0 0x557b46ef7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6be870740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee82f0 0x557b46ef81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6be870740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ef0ed0 0x557b46ef6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.660 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6be870740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b41af7a0 con 0x557b46ee82f0 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6be870740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b41af0e0 con 0x557b46ee79d0 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6be870740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b41ac6c0 con 0x557b46ef0ed0 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6be870740 10 client.0 _ll_register_callbacks cb 0x7fa6a400bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6b922d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee82f0 0x557b46ef81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6b9a2e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee79d0 0x557b46ef7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6b922d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee82f0 0x557b46ef81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53010/0 (socket says 172.21.15.60:53010) 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6b922d640 1 -- 172.21.15.60:0/1771731231 learned_addr learned my addr 172.21.15.60:0/1771731231 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6ba22f640 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ef0ed0 0x557b46ef6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6b922d640 1 -- 172.21.15.60:0/1771731231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ef0ed0 msgr2=0x557b46ef6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6b922d640 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ef0ed0 0x557b46ef6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6b922d640 1 -- 172.21.15.60:0/1771731231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee79d0 msgr2=0x557b46ef7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6b922d640 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee79d0 0x557b46ef7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6b922d640 1 -- 172.21.15.60:0/1771731231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b46ef6a30 con 0x557b46ee82f0 2024-03-04T12:27:54.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.648+0000 7fa6b9a2e640 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee79d0 0x557b46ef7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa6b922d640 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee82f0 0x557b46ef81e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4011010 tx=0x7fa6a401a990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ad0e30284a5963 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa69d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa69d7fa640 1 -- 172.21.15.60:0/1771731231 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a401ac50 con 0x557b46ee82f0 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa6be870740 1 -- 172.21.15.60:0/1771731231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6b00bce10 con 0x557b46ee82f0 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa69d7fa640 1 -- 172.21.15.60:0/1771731231 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fa6a401adf0 con 0x557b46ee82f0 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa69d7fa640 1 -- 172.21.15.60:0/1771731231 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6a401a6d0 con 0x557b46ee82f0 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa6be870740 10 client.16512 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa6be870740 1 -- 172.21.15.60:0/1771731231 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x557b46f58240 con 0x557b46ee82f0 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa69d7fa640 1 -- 172.21.15.60:0/1771731231 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6b00bce10 con 0x557b46ee82f0 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa69d7fa640 1 -- 172.21.15.60:0/1771731231 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa6a4011620 con 0x557b46ee82f0 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa6be870740 10 client.16512 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa6be870740 10 client.16512 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa6be870740 20 client.16512 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:54.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa6be870740 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x557b46f58760 0x557b46f5ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa6be870740 4 client.16512 mds_command: new command op to 24479 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.649+0000 7fa6be870740 1 -- 172.21.15.60:0/1771731231 --> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557b46dd0640 con 0x557b46f58760 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.650+0000 7fa6b9a2e640 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x557b46f58760 0x557b46f5ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.650+0000 7fa69d7fa640 10 client.16512 ms_handle_connect on v2:172.21.15.63:6836/3974574804 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.650+0000 7fa6b9a2e640 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x557b46f58760 0x557b46f5ab50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d4d6e2017bc783be server_cookie=37e569e88d835357 in_seq=0 out_seq=0 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa69d7fa640 1 -- 172.21.15.60:0/1771731231 <== mds.2 v2:172.21.15.63:6836/3974574804 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+84239 (crc 0 0 0) 0x557b46dd0640 con 0x557b46f58760 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa69d7fa640 10 client.16512 handle_command_reply: tid=0 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa6be870740 1 client.16512 shutdown 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa6be870740 1 -- 172.21.15.60:0/1771731231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee82f0 msgr2=0x557b46ef81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa6be870740 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee82f0 0x557b46ef81e0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4011010 tx=0x7fa6a401a990 comp rx=0 tx=0).stop 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa6be870740 1 -- 172.21.15.60:0/1771731231 shutdown_connections 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa6be870740 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.63:6836/3974574804,v1:172.21.15.63:6837/3974574804] conn(0x557b46f58760 0x557b46f5ab50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa6be870740 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557b46ef0ed0 0x557b46ef6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa6be870740 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x557b46ee82f0 0x557b46ef81e0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa6be870740 1 --2- 172.21.15.60:0/1771731231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557b46ee79d0 0x557b46ef7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa69d7fa640 0 client.16512 ms_handle_reset on v2:172.21.15.63:6836/3974574804 2024-03-04T12:27:54.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa6be870740 1 -- 172.21.15.60:0/1771731231 >> 172.21.15.60:0/1771731231 conn(0x557b46ee2100 msgr2=0x557b46ee34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa6be870740 1 -- 172.21.15.60:0/1771731231 shutdown_connections 2024-03-04T12:27:54.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.652+0000 7fa6be870740 1 -- 172.21.15.60:0/1771731231 wait complete. 2024-03-04T12:27:54.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6be870740 20 client.16512 trim_cache size 0 max 16384 2024-03-04T12:27:54.665 INFO:teuthology.orchestra.run.smithi060.stdout:[ 2024-03-04T12:27:54.665 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.665 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.665 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 15145518, 2024-03-04T12:27:54.665 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628114, 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/trunc.c" 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 36876980, 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627857, 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/inject" 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 50067782, 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628028, 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/looptest.c" 2024-03-04T12:27:54.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 60798369, 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628039, 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/nametest.c" 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 85690099, 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628304, 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.idx" 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.667 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 88145242, 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627908, 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/datapid.c" 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 88331178, 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627982, 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/devzero.c" 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.668 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 104036562, 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627820, 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/Makefile" 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 116796498, 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627925, 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/fsx.c" 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 135401196, 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628329, 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/example.config" 2024-03-04T12:27:54.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 142309834, 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628019, 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/iopat.c" 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 144526712, 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627891, 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildgrouplist" 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.670 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.671 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 145937807, 2024-03-04T12:27:54.671 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628102, 2024-03-04T12:27:54.671 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_open_tmpfiles.c" 2024-03-04T12:27:54.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.671 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.671 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 165925414, 2024-03-04T12:27:54.671 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627977, 2024-03-04T12:27:54.671 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/chprojid_fail.c" 2024-03-04T12:27:54.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.671 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.671 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 174463506, 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628032, 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/md5.h" 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 183839499, 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627988, 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirstress.c" 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 194602196, 2024-03-04T12:27:54.672 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627861, 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/module" 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 197431264, 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627898, 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/install-sh" 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 200511642, 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627859, 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/log" 2024-03-04T12:27:54.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 236941976, 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628052, 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/popdir.pl" 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 237638720, 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628333, 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/CHANGES" 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.674 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 245010395, 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628043, 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/parse-extent-tree.awk" 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 245626071, 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627950, 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-cow-race.c" 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 248848676, 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627942, 2024-03-04T12:27:54.675 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/randomize.awk" 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 262902837, 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628076, 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_attr_corruption.c" 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 268944219, 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627915, 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/str_to_bytes.c" 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.676 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 274913497, 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627910, 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/forker.c" 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 285157337, 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628002, 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2fs" 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 303494259, 2024-03-04T12:27:54.677 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627924, 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/fsstress.c" 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 309293263, 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627900, 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/pattern.h" 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 334196108, 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627937, 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_liburing.m4" 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.678 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 340412425, 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627786, 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/prepare-commit-msg.sample" 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 345740528, 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628055, 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/punch-alternating.c" 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.679 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 371976726, 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628064, 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/scaleread.c" 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 376981377, 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628326, 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common" 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 378520951, 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627948, 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-append-write-fallocate-race.c" 2024-03-04T12:27:54.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 382904155, 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627837, 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/cgroup2" 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 392669768, 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627829, 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/acinclude.m4" 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.681 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 403369894, 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628108, 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_snapshot_deleted_subvolume.c" 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 430760593, 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627850, 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/f2fs" 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.682 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 446850512, 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627886, 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/group-names.txt" 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 451935590, 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628113, 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/testx.c" 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 457769834, 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628297, 2024-03-04T12:27:54.683 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/heads" 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 464604364, 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627998, 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fiexchange.h" 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 480470474, 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627813, 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/HEAD" 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.684 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 507370495, 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628047, 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/fio-insert-and-compare.py" 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 511646524, 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627932, 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_gdbmdev.m4" 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.685 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 517341081, 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627894, 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/dataascii.h" 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 523849527, 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627792, 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/post-update.sample" 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 526939251, 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628345, 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/Makefile" 2024-03-04T12:27:54.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 536947007, 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628004, 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fs_perms.c" 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 539966682, 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627979, 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dbtest.c" 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.687 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 558632771, 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628298, 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/tags" 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 569002490, 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628035, 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmap-rw-fault.c" 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 587353601, 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627875, 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/test_names" 2024-03-04T12:27:54.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 595273088, 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628021, 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/listxattr.c" 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 597139248, 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627992, 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/enospc_unlink.c" 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.689 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 649633086, 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627928, 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/rwtest.sh" 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 664703078, 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628003, 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2fs_check" 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 676740681, 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628057, 2024-03-04T12:27:54.690 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/randholes.c" 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 683113306, 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627822, 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README" 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 685400260, 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627916, 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/string_to_tokens.c" 2024-03-04T12:27:54.691 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 697263990, 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627907, 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/databin.c" 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 701451165, 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627963, 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aiocp.c" 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 703550772, 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628096, 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_stale_pmd.c" 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 735430208, 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628083, 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_enospc.c" 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.693 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 737167532, 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628309, 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs" 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 737580633, 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628038, 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/multi_open_unlink.c" 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 740376772, 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628034, 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mkswap.c" 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.694 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 741420734, 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627949, 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-append-write-read-race.c" 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 750881278, 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627956, 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-invalidate-failure.c" 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 760639474, 2024-03-04T12:27:54.695 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627784, 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git" 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 783648266, 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628027, 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/loggen.c" 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 788307367, 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628106, 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_reflink_read_race.c" 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.696 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 791556209, 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628339, 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/Makefile" 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 792021749, 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627901, 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/random_range.h" 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 812478707, 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628042, 2024-03-04T12:27:54.697 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/parse-dev-tree.awk" 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 812875497, 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628089, 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_holes.c" 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 814617634, 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628024, 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/log-writes.c" 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 822566053, 2024-03-04T12:27:54.698 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627941, 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/new" 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 831096825, 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627921, 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/aio-stress.c" 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 834510631, 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628010, 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ftrunc.c" 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 857032667, 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627897, 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/forker.h" 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 858959478, 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627868, 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/quota" 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 885708074, 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627835, 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/casefold" 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.700 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 905911049, 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627849, 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ext4" 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 915298806, 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627851, 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/fail_make_request" 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 919406646, 2024-03-04T12:27:54.701 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628323, 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/Makefile" 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 928752688, 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627911, 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/open_flags.c" 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 946178551, 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628318, 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/LICENSES" 2024-03-04T12:27:54.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 956583036, 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627962, 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-last-ref-held-by-io.c" 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 962108217, 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628344, 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress" 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.703 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 967747680, 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628327, 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/Makefile" 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 974836631, 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627919, 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/local.config.example" 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.704 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 992217432, 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627867, 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/punch" 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 996357490, 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627874, 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/scsi_debug" 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1004504255, 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628071, 2024-03-04T12:27:54.705 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/stale_handle.c" 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1010996799, 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628059, 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/rename.c" 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1015005010, 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627873, 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/report" 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.706 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1024148256, 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627975, 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_unlink_test_modified.c" 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1030921239, 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628331, 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/README" 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.707 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1052722801, 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628294, 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-commit.sample" 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1065166274, 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627938, 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_utilies.m4" 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1072318883, 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628291, 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/info" 2024-03-04T12:27:54.708 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1084052773, 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628088, 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_getcwd.c" 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1090389746, 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628017, 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/holes.c" 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.709 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1101001460, 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627863, 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/overlay" 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1109726417, 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628020, 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/itrash.c" 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1124813141, 2024-03-04T12:27:54.710 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628072, 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/stat_test.c" 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1131369212, 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627976, 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/checkpoint_journal.c" 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1142603624, 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628075, 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_access_root.c" 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.711 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1146241474, 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628116, 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/unwritten_mmap.c" 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1164357481, 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627877, 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ubifs" 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.712 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1189687453, 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628041, 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/open_by_handle.c" 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1201006250, 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627860, 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/metadump" 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1205126378, 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627834, 2024-03-04T12:27:54.713 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/btrfs" 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1211042958, 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628110, 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_truncate_cmtime.c" 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1229411314, 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627793, 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/update.sample" 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.714 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1234305986, 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627852, 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filestreams" 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1243055710, 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627917, 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/tlibio.c" 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1275224862, 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628007, 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fstest.c" 2024-03-04T12:27:54.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1308907745, 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628099, 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_writev_overlap.c" 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1311439776, 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628301, 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack" 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.716 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1374468917, 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627961, 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-io-setup-with-nonwritable-context-pointer.c" 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1384768012, 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628009, 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fsync-tester.c" 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1389521992, 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627833, 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/attr" 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.717 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1400065255, 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627819, 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/MAINTAINERS" 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1409895252, 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628040, 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/nsexec.c" 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1424720120, 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627880, 2024-03-04T12:27:54.718 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/zoned" 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1448578678, 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627923, 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/doio.h" 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1466189143, 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627973, 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_null_ocount.c" 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.719 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1481907459, 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627856, 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/gcov" 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1487432253, 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628070, 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/splice2pipe.c" 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.720 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1490127066, 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627960, 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-free-ring-with-bogus-nr-pages.c" 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1506481499, 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627968, 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/append_writer.c" 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1511751689, 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627974, 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_unlink_test.c" 2024-03-04T12:27:54.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1516416134, 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627926, 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/growfiles.c" 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1521069307, 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628343, 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/Makefile" 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.722 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1535957113, 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627966, 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/allocstale.c" 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1553501209, 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627972, 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/btrfs_crc32c_forged_name.py" 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1556666008, 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628346, 2024-03-04T12:27:54.723 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes" 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1558305500, 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628302, 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/info" 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1569185443, 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627797, 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-merge-commit.sample" 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.724 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1593071906, 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628085, 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ext4_dax_journal_corruption.c" 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1597883187, 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628300, 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects" 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1605746668, 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627791, 2024-03-04T12:27:54.725 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-receive.sample" 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1619195040, 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627876, 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/tracing" 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1660101853, 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628018, 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/holetest.c" 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.726 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1666641881, 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627823, 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.config-sections" 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1676794923, 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627854, 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filter.btrfs" 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1682794586, 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628107, 2024-03-04T12:27:54.727 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_rename_overwrite.c" 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1691620670, 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627828, 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/VERSION" 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1695000015, 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627866, 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/preamble" 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.728 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1710848717, 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627818, 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/index.lock" 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1716180899, 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628347, 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/Makefile" 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1718693801, 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628093, 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_cow_race.c" 2024-03-04T12:27:54.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1730540472, 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628086, 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_futimens.c" 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1750583676, 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627895, 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/databin.h" 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1769005384, 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627964, 2024-03-04T12:27:54.730 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aiodio_sparse2.c" 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1792594302, 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627821, 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/Makepkgs" 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1800336582, 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627914, 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/random_range.c" 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.731 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1833401501, 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627853, 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filter" 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1861101553, 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627881, 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/localhost.config" 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1861891348, 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627832, 2024-03-04T12:27:54.732 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/check" 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1875339343, 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628119, 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/usemem.c" 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1880999404, 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628051, 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/popattr.py" 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1887092554, 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627922, 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/doio.c" 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1900411416, 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628340, 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4" 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1907186617, 2024-03-04T12:27:54.734 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628067, 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/seek_sanity_test.c" 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1942938966, 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628014, 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/global.h" 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1950789847, 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628335, 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/Makefile" 2024-03-04T12:27:54.735 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1953647681, 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628005, 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fscrypt-crypt-util.c" 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1954885257, 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628115, 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/truncfile.c" 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.736 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1959745075, 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628054, 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/preallo_rw_pattern_writer.c" 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1972643696, 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628317, 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.gitignore" 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1976873353, 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627788, 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/fsmonitor-watchman.sample" 2024-03-04T12:27:54.737 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1983957300, 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627896, 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/file_lock.h" 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1985275399, 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628600, 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/FioCompare.py" 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.738 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1987659131, 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628029, 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/lstat64.c" 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1988699713, 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628312, 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/origin/HEAD" 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 1996970934, 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628334, 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include" 2024-03-04T12:27:54.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2037207467, 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627838, 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/config" 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2047917912, 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627869, 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/rc" 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2072022007, 2024-03-04T12:27:54.740 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627991, 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/e4compact.c" 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2074161152, 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627824, 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.device-mapper" 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2083427417, 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628049, 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/generate-schema.py" 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.741 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2089464262, 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628082, 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_encrypted_d_revalidate.c" 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2092640605, 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628308, 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs" 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2100479378, 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628063, 2024-03-04T12:27:54.742 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/runas.c" 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2111082012, 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627918, 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/write_log.c" 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2150168531, 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628338, 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp" 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.743 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2169531563, 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627812, 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/packed-refs" 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2178915993, 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628013, 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/getpagesize.c" 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2183117465, 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627787, 2024-03-04T12:27:54.744 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/push-to-checkout.sample" 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2184397299, 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628306, 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/origin" 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2190136576, 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628050, 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/permname.c" 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.745 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2203566559, 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627994, 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fake-dump-rootino.c" 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2205318374, 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627985, 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-invalidate-cache.c" 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2212629300, 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628342, 2024-03-04T12:27:54.746 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src" 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2216314706, 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627887, 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/requirement-checking.txt" 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2248410613, 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627983, 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-buf-fault.c" 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.747 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2251882362, 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627888, 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/testing-afs.txt" 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2288193026, 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627789, 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-rebase.sample" 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2296820300, 2024-03-04T12:27:54.748 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628030, 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/makeextents.c" 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2301856156, 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627841, 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmdust" 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2326725402, 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627912, 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/pattern.c" 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.749 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2327762176, 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628100, 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mtab.c" 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2331817066, 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627965, 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/alloc.c" 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.750 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2337066432, 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628058, 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/readdir-while-renames.c" 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2347564201, 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628111, 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_truncate_self.c" 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2354561068, 2024-03-04T12:27:54.751 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628599, 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf" 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2390138741, 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627980, 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/deduperace.c" 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2398659611, 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628062, 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/rewinddir-test.c" 2024-03-04T12:27:54.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2402242469, 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628045, 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/Makefile" 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2413215785, 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627890, 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/builddefs.in" 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.753 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2416427284, 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627790, 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/applypatch-msg.sample" 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2427353895, 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628016, 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/helpers.h" 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2448401042, 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627892, 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildmacros" 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.754 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2449274002, 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628011, 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/genhashnames.c" 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2465814984, 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628046, 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/ResultData.py" 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2470274186, 2024-03-04T12:27:54.755 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627933, 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_globals.m4" 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2483091912, 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627899, 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/open_flags.h" 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2484676621, 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628307, 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/origin/HEAD" 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.756 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2500626330, 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628319, 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/LICENSES/GPL-2.0" 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2520274436, 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628037, 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmapcat.c" 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2539463223, 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628008, 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fsync-err.c" 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.757 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2545860023, 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628296, 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs" 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2554269433, 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627936, 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libcdev.m4" 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2567975590, 2024-03-04T12:27:54.758 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627978, 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/cloner.c" 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2572926129, 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627845, 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmlogwrites" 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2582181752, 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627993, 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ext4_resize.c" 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.759 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.760 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.760 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2588338894, 2024-03-04T12:27:54.760 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628022, 2024-03-04T12:27:54.760 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/locktest.c" 2024-03-04T12:27:54.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.760 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.760 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2597676126, 2024-03-04T12:27:54.760 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627931, 2024-03-04T12:27:54.760 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_aiodev.m4" 2024-03-04T12:27:54.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.760 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2605596910, 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628305, 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes" 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2619429103, 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628303, 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.pack" 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2622451403, 2024-03-04T12:27:54.761 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627955, 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-hole-filling-race.c" 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2630336533, 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627872, 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/repair" 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2638639856, 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628048, 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/fio-results.sql" 2024-03-04T12:27:54.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2646369343, 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628095, 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_fallocate.c" 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2686887760, 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628078, 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_create_short_dirs.c" 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.763 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2702380316, 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628092, 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_cow_memory_failure.c" 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2705806874, 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628332, 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc" 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2707270517, 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628293, 2024-03-04T12:27:54.764 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks" 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2716645146, 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628324, 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/tar" 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2717690756, 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627839, 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/defrag" 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.765 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2731991507, 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627794, 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-push.sample" 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2742071533, 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628316, 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/heads/master" 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2799709160, 2024-03-04T12:27:54.766 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627878, 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/verity" 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2808835004, 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628101, 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ofd_locks.c" 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2811251371, 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628012, 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/getdevicesize.c" 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.767 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2834186746, 2024-03-04T12:27:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627796, 2024-03-04T12:27:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-applypatch.sample" 2024-03-04T12:27:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2835385329, 2024-03-04T12:27:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628094, 2024-03-04T12:27:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_dio.c" 2024-03-04T12:27:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.768 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2845170803, 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627810, 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/shallow" 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2855053345, 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628053, 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/preallo_rw_pattern_reader.c" 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.769 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2857268722, 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627984, 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-interleaved.c" 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2857837464, 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627939, 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_uuiddev.m4" 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2859909662, 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628118, 2024-03-04T12:27:54.770 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/uring_read_fault.c" 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2859928072, 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627831, 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/xfstests.spec.in" 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2866811136, 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628074, 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/swapon.c" 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.771 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2870400145, 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627905, 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/write_log.h" 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2928213493, 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628314, 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/HEAD" 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2935356529, 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627817, 2024-03-04T12:27:54.772 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/config" 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2948753456, 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628001, 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2attr" 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2951101475, 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627825, 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.fuse" 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.773 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2951388549, 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627870, 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/reflink" 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2978079813, 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628097, 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_write_ro.c" 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2980343275, 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628065, 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/scaleread.sh" 2024-03-04T12:27:54.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2989763929, 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628036, 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmap-write-concurrent.c" 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2992788983, 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627848, 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/encrypt" 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 2995824591, 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628081, 2024-03-04T12:27:54.775 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_type.c" 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3091096923, 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627871, 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/renameat2" 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3100143408, 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628066, 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/seek_copy_test.c" 2024-03-04T12:27:54.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3101856120, 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627864, 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/perf" 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3106897311, 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627883, 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/rc.sysinit" 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.777 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3127721826, 2024-03-04T12:27:54.778 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627885, 2024-03-04T12:27:54.778 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/auxiliary-programs.txt" 2024-03-04T12:27:54.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.778 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.778 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3128406579, 2024-03-04T12:27:54.778 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628006, 2024-03-04T12:27:54.778 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fssum.c" 2024-03-04T12:27:54.778 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.778 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.778 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.778 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3130216736, 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627906, 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/dataascii.c" 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3176564404, 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627971, 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bstat.c" 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3180752784, 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627934, 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libbtrfs.m4" 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.779 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3186562381, 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627954, 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-fcntl-race.c" 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3201659955, 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628073, 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/statx.h" 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.780 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3208147820, 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627862, 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/nfs" 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3208857766, 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627913, 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/random.c" 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3218196769, 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627858, 2024-03-04T12:27:54.781 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/locktest" 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3224183881, 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628098, 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_writev.c" 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3251585662, 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627865, 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/populate" 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.782 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3253773870, 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627904, 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/tlibio.h" 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3262883244, 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628322, 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm" 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.783 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3273395424, 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628056, 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/pwrite_mmap_blocked.c" 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3299963160, 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628087, 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_get_file_time.c" 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3326793751, 2024-03-04T12:27:54.784 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627935, 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libcap.m4" 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3327733039, 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627785, 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/description" 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3339193630, 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628330, 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash" 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.785 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3340835770, 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627989, 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dmerror" 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3343339479, 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628061, 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/resvtest.c" 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3354200374, 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628031, 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/md5.c" 2024-03-04T12:27:54.786 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3356213728, 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627842, 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmerror" 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3357979348, 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627987, 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirperf.c" 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.787 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3361629239, 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627959, 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-write-verify.c" 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3377667437, 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628060, 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/renameat2.c" 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.788 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3389429376, 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628120, 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/uuid_ioctl.c" 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3391331412, 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628295, 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/branches" 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3421780079, 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628077, 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_create_long_dirs.c" 2024-03-04T12:27:54.789 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3429840995, 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628325, 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/tar/Makefile" 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3453264649, 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627795, 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/commit-msg.sample" 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.790 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3453817756, 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628313, 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/heads/master" 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3460907172, 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628336, 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib" 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3465260133, 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627879, 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/xfs" 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.791 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3485690069, 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627893, 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildrules" 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3490249718, 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627970, 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/attr_replace_test.c" 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.792 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3496363722, 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628079, 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_offset.c" 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3507448255, 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627945, 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/soak" 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3514950093, 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627827, 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.selftest" 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.793 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3520903312, 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628025, 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/log-writes.h" 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3528989467, 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627997, 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fiemap-tester.c" 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3529883614, 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628090, 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_immutable.c" 2024-03-04T12:27:54.794 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3530971332, 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628311, 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/origin" 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3536932502, 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628091, 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_collision.c" 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.795 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3547803984, 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628080, 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_offset2.c" 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3553356909, 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627920, 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lsqa.pl" 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3554358368, 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627844, 2024-03-04T12:27:54.796 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmhugedisk" 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3563556352, 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627927, 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/iogen.c" 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3585296020, 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628315, 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/heads" 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.797 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3603659650, 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627990, 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dumpfile" 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3620596871, 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628337, 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/Makefile" 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3641180848, 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628341, 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/Makefile" 2024-03-04T12:27:54.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3646294562, 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628328, 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs" 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3705172137, 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627843, 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmflakey" 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3729684653, 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627889, 2024-03-04T12:27:54.799 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/xunit.xsd" 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3732462890, 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627902, 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/str_to_bytes.h" 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3748510104, 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627882, 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configure.ac" 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.800 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3759180496, 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627830, 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/macros.template" 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3763215343, 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627826, 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.overlay" 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.801 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3781611481, 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627840, 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmdelay" 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3787454378, 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627909, 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/file_lock.c" 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3820179523, 2024-03-04T12:27:54.802 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628109, 2024-03-04T12:27:54.803 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_stripealign.c" 2024-03-04T12:27:54.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.803 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.803 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3826331620, 2024-03-04T12:27:54.803 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628292, 2024-03-04T12:27:54.803 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/info/exclude" 2024-03-04T12:27:54.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.803 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.803 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.803 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3827698450, 2024-03-04T12:27:54.804 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628104, 2024-03-04T12:27:54.804 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_2.c" 2024-03-04T12:27:54.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.804 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.804 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3848130917, 2024-03-04T12:27:54.804 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628026, 2024-03-04T12:27:54.804 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/replay-log.c" 2024-03-04T12:27:54.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.804 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3859052483, 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627999, 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill.c" 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3868895663, 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627940, 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_xfslibs.m4" 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.805 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.806 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3871175009, 2024-03-04T12:27:54.806 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627836, 2024-03-04T12:27:54.806 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ceph" 2024-03-04T12:27:54.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.806 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.806 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3880677228, 2024-03-04T12:27:54.806 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628015, 2024-03-04T12:27:54.806 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/godown.c" 2024-03-04T12:27:54.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.806 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.807 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.807 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3883403115, 2024-03-04T12:27:54.807 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627944, 2024-03-04T12:27:54.807 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/setup" 2024-03-04T12:27:54.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.807 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.807 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3904545215, 2024-03-04T12:27:54.807 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627930, 2024-03-04T12:27:54.807 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_acldev.m4" 2024-03-04T12:27:54.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.807 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.808 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.808 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3913745567, 2024-03-04T12:27:54.808 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627969, 2024-03-04T12:27:54.808 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/attr-list-by-handle-cursor-test.c" 2024-03-04T12:27:54.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.808 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.808 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.808 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3922500342, 2024-03-04T12:27:54.808 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627952, 2024-03-04T12:27:54.808 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-eof-race.c" 2024-03-04T12:27:54.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3938786484, 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627903, 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/string_to_tokens.h" 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3961789900, 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628000, 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2.c" 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.809 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3963874142, 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627996, 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/feature.c" 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3964209623, 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628320, 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build" 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3975625141, 2024-03-04T12:27:54.810 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627986, 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirhash_collide.c" 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 3999156906, 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627958, 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-subblock-eof-read.c" 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4003210665, 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628310, 2024-03-04T12:27:54.811 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes" 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4003322253, 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627946, 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/af_unix.c" 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4015915125, 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627957, 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-invalidate-readahead.c" 2024-03-04T12:27:54.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4017106498, 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627943, 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/release.sh" 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4024991526, 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628069, 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/splice-test.c" 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.813 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4027032892, 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628084, 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ext4_dax_inline_corruption.c" 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4034769142, 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627947, 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/SOURCE" 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4059425113, 2024-03-04T12:27:54.814 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628023, 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/SOURCE" 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4062650391, 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627995, 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fault.c" 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4096812389, 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627953, 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-extend-stat.c" 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.815 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4102732125, 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628117, 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/unwritten_sync.c" 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4103847433, 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628112, 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/test-nextquota.c" 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.816 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4115184060, 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628068, 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/soak_duration.awk" 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4120013376, 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627951, 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-cycle-write.c" 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4131724534, 2024-03-04T12:27:54.817 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628103, 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_1.c" 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4139792175, 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627967, 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/append_reader.c" 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4161360526, 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628321, 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/Makefile" 2024-03-04T12:27:54.818 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4186743557, 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627855, 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/fuzzy" 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4221619859, 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627847, 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dump" 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.819 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4225237416, 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627981, 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/detached_mounts_propagation.c" 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4232775347, 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628044, 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/FioResultDecoder.py" 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4235409953, 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627846, 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmthin" 2024-03-04T12:27:54.820 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4235944732, 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628105, 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_3.c" 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4248435851, 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627884, 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/xfscrash" 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.821 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.822 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.822 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4260419634, 2024-03-04T12:27:54.822 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511627929, 2024-03-04T12:27:54.822 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/multilib.m4" 2024-03-04T12:27:54.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:27:54.822 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:27:54.822 INFO:teuthology.orchestra.run.smithi060.stdout: "damage_type": "backtrace", 2024-03-04T12:27:54.822 INFO:teuthology.orchestra.run.smithi060.stdout: "id": 4291632764, 2024-03-04T12:27:54.822 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 1099511628033, 2024-03-04T12:27:54.822 INFO:teuthology.orchestra.run.smithi060.stdout: "path": "/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/metaperf.c" 2024-03-04T12:27:54.822 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:27:54.822 INFO:teuthology.orchestra.run.smithi060.stdout:] 2024-03-04T12:27:54.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6bbc92640 1 -- 172.21.15.60:0/605718443 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa6a00a1f70 msgr2=0x7fa6a00a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6bbc92640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa6a00a1f70 0x7fa6a00a4430 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009fd0 tx=0x7fa6ac009500 comp rx=0 tx=0).stop 2024-03-04T12:27:54.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6bbc92640 1 -- 172.21.15.60:0/605718443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6b4100440 msgr2=0x7fa6b419d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:54.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6bbc92640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6b4100440 0x7fa6b419d830 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0006010 tx=0x7fa6b002cb30 comp rx=0 tx=0).stop 2024-03-04T12:27:54.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6bbc92640 1 -- 172.21.15.60:0/605718443 shutdown_connections 2024-03-04T12:27:54.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6bbc92640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6b4197a30 0x7fa6b4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6bbc92640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b4105510 0x7fa6b419dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6bbc92640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa6a00a1f70 0x7fa6a00a4430 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6bbc92640 1 --2- 172.21.15.60:0/605718443 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa6b4100440 0x7fa6b419d830 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:54.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6bbc92640 1 -- 172.21.15.60:0/605718443 >> 172.21.15.60:0/605718443 conn(0x7fa6b40773e0 msgr2=0x7fa6b40689e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:54.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6bbc92640 1 -- 172.21.15.60:0/605718443 shutdown_connections 2024-03-04T12:27:54.824 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:54.653+0000 7fa6bbc92640 1 -- 172.21.15.60:0/605718443 wait complete. 2024-03-04T12:27:54.825 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi060 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-04T12:27:55.059 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=72dcf58a6f1547c4bb919cde95d5548b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi060 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-04T12:27:55.062 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-04T12:27:55.072 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi060.front.sepia.ceph.com 2024-03-04T12:27:55.075 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:27:55.075 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-04T12:27:55.087 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-04T12:27:55.097 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-04T12:27:55.097 INFO:teuthology.task.exec:Executing custom commands... 2024-03-04T12:27:55.097 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi060.front.sepia.ceph.com 2024-03-04T12:27:55.097 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-04T12:27:55.157 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-04T12:27:55.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.195+0000 7f9f47473640 1 -- 172.21.15.60:0/2220626027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f40101ef0 msgr2=0x7f9f40105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.195+0000 7f9f47473640 1 --2- 172.21.15.60:0/2220626027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f40101ef0 0x7f9f40105f30 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9f30009860 tx=0x7f9f3002bb70 comp rx=0 tx=0).stop 2024-03-04T12:27:55.202 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.195+0000 7f9f47473640 1 -- 172.21.15.60:0/2220626027 shutdown_connections 2024-03-04T12:27:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.195+0000 7f9f47473640 1 --2- 172.21.15.60:0/2220626027 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f40106610 0x7f9f4010e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.195+0000 7f9f47473640 1 --2- 172.21.15.60:0/2220626027 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f40101ef0 0x7f9f40105f30 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.195+0000 7f9f47473640 1 --2- 172.21.15.60:0/2220626027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f40101540 0x7f9f40101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.195+0000 7f9f47473640 1 -- 172.21.15.60:0/2220626027 >> 172.21.15.60:0/2220626027 conn(0x7f9f400fcdc0 msgr2=0x7f9f400ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.195+0000 7f9f47473640 1 -- 172.21.15.60:0/2220626027 shutdown_connections 2024-03-04T12:27:55.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.195+0000 7f9f47473640 1 -- 172.21.15.60:0/2220626027 wait complete. 2024-03-04T12:27:55.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.196+0000 7f9f47473640 1 Processor -- start 2024-03-04T12:27:55.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.196+0000 7f9f47473640 1 -- start start 2024-03-04T12:27:55.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.196+0000 7f9f47473640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f40101540 0x7f9f4019b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.196+0000 7f9f47473640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f40106610 0x7f9f4019bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.196+0000 7f9f47473640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f40195840 0x7f9f40195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.205 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.196+0000 7f9f47473640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4010f520 con 0x7f9f40101540 2024-03-04T12:27:55.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.196+0000 7f9f47473640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4010f3a0 con 0x7f9f40195840 2024-03-04T12:27:55.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.196+0000 7f9f47473640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4010f6a0 con 0x7f9f40106610 2024-03-04T12:27:55.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.197+0000 7f9f4520f640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f40101540 0x7f9f4019b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.197+0000 7f9f44a0e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f40106610 0x7f9f4019bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.197+0000 7f9f44a0e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f40106610 0x7f9f4019bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:59966/0 (socket says 172.21.15.60:59966) 2024-03-04T12:27:55.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.197+0000 7f9f44a0e640 1 -- 172.21.15.60:0/583968938 learned_addr learned my addr 172.21.15.60:0/583968938 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:55.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.197+0000 7f9f45a10640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f40195840 0x7f9f40195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.197+0000 7f9f4520f640 1 -- 172.21.15.60:0/583968938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f40106610 msgr2=0x7f9f4019bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.197+0000 7f9f4520f640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f40106610 0x7f9f4019bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.197+0000 7f9f4520f640 1 -- 172.21.15.60:0/583968938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f40195840 msgr2=0x7f9f40195cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.197+0000 7f9f4520f640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f40195840 0x7f9f40195cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.197+0000 7f9f4520f640 1 -- 172.21.15.60:0/583968938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f401963d0 con 0x7f9f40101540 2024-03-04T12:27:55.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.198+0000 7f9f4520f640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f40101540 0x7f9f4019b6d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c005fd0 tx=0x7f9f3c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:55.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.198+0000 7f9f367fc640 1 -- 172.21.15.60:0/583968938 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c00cd00 con 0x7f9f40101540 2024-03-04T12:27:55.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.198+0000 7f9f367fc640 1 -- 172.21.15.60:0/583968938 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f3c0035f0 con 0x7f9f40101540 2024-03-04T12:27:55.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.198+0000 7f9f367fc640 1 -- 172.21.15.60:0/583968938 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c010670 con 0x7f9f40101540 2024-03-04T12:27:55.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.198+0000 7f9f47473640 1 -- 172.21.15.60:0/583968938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f401966c0 con 0x7f9f40101540 2024-03-04T12:27:55.208 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.198+0000 7f9f47473640 1 -- 172.21.15.60:0/583968938 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f401a2870 con 0x7f9f40101540 2024-03-04T12:27:55.209 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f4a055740 1 Processor -- start 2024-03-04T12:27:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f367fc640 1 -- 172.21.15.60:0/583968938 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f9f3c0037a0 con 0x7f9f40101540 2024-03-04T12:27:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f4a055740 1 -- start start 2024-03-04T12:27:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f4a055740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 0x55b42c699db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f4a055740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c69a2f0 0x55b42c6a2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f4a055740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c6a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f4a055740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b42c660dc0 con 0x55b42c6999d0 2024-03-04T12:27:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f4a055740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b42c5fcec0 con 0x55b42c69a2f0 2024-03-04T12:27:55.210 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f4a055740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b42c5a5d30 con 0x55b42c6a2ed0 2024-03-04T12:27:55.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f45a10640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 0x55b42c699db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f4520f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c69a2f0 0x55b42c6a2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f45a10640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 0x55b42c699db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53034/0 (socket says 172.21.15.60:53034) 2024-03-04T12:27:55.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f45a10640 1 -- 172.21.15.60:0/1139171417 learned_addr learned my addr 172.21.15.60:0/1139171417 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:55.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.201+0000 7f9f44a0e640 1 --2- 172.21.15.60:0/1139171417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c6a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.202+0000 7f9f45a10640 1 -- 172.21.15.60:0/1139171417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 msgr2=0x55b42c6a52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.202+0000 7f9f45a10640 1 --2- 172.21.15.60:0/1139171417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c6a52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.202+0000 7f9f45a10640 1 -- 172.21.15.60:0/1139171417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c69a2f0 msgr2=0x55b42c6a2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.202+0000 7f9f45a10640 1 --2- 172.21.15.60:0/1139171417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c69a2f0 0x55b42c6a2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.202+0000 7f9f45a10640 1 -- 172.21.15.60:0/1139171417 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b42c3113f0 con 0x55b42c6999d0 2024-03-04T12:27:55.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.202+0000 7f9f367fc640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f9f2c0a1f30 0x7f9f2c0a43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.202+0000 7f9f45a10640 1 --2- 172.21.15.60:0/1139171417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 0x55b42c699db0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f9f38003f80 tx=0x7f9f38009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2ed34a05a9a9a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:55.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.202+0000 7f9f367fc640 1 -- 172.21.15.60:0/583968938 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f9f3c0ce720 con 0x7f9f40101540 2024-03-04T12:27:55.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.202+0000 7f9f4520f640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f9f2c0a1f30 0x7f9f2c0a43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.202+0000 7f9f2b7fe640 1 -- 172.21.15.60:0/1139171417 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3801e030 con 0x55b42c6999d0 2024-03-04T12:27:55.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.202+0000 7f9f2b7fe640 1 -- 172.21.15.60:0/1139171417 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f3800fd10 con 0x55b42c6999d0 2024-03-04T12:27:55.212 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.202+0000 7f9f2b7fe640 1 -- 172.21.15.60:0/1139171417 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f38005020 con 0x55b42c6999d0 2024-03-04T12:27:55.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.203+0000 7f9f4a055740 1 -- 172.21.15.60:0/1139171417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 msgr2=0x55b42c699db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.203+0000 7f9f4520f640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f9f2c0a1f30 0x7f9f2c0a43f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c0cbb80 tx=0x7f9f3c0d4550 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:55.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.203+0000 7f9f4a055740 1 --2- 172.21.15.60:0/1139171417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 0x55b42c699db0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f9f38003f80 tx=0x7f9f38009500 comp rx=0 tx=0).stop 2024-03-04T12:27:55.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.203+0000 7f9f4a055740 1 -- 172.21.15.60:0/1139171417 shutdown_connections 2024-03-04T12:27:55.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.203+0000 7f9f4a055740 1 --2- 172.21.15.60:0/1139171417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c6a52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.216 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.203+0000 7f9f4a055740 1 --2- 172.21.15.60:0/1139171417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c69a2f0 0x55b42c6a2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.203+0000 7f9f4a055740 1 --2- 172.21.15.60:0/1139171417 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 0x55b42c699db0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.203+0000 7f9f4a055740 1 -- 172.21.15.60:0/1139171417 >> 172.21.15.60:0/1139171417 conn(0x55b42c694100 msgr2=0x55b42c6944e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.204+0000 7f9f4a055740 1 -- 172.21.15.60:0/1139171417 shutdown_connections 2024-03-04T12:27:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.204+0000 7f9f4a055740 1 -- 172.21.15.60:0/1139171417 wait complete. 2024-03-04T12:27:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.204+0000 7f9f4a055740 1 -- start start 2024-03-04T12:27:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.204+0000 7f9f4a055740 1 Processor -- start 2024-03-04T12:27:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f4a055740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c6999d0 0x55b42c69ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f4a055740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 0x55b42c6a0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f4a055740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c6a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f4a055740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b42c59b4a0 con 0x55b42c69a2f0 2024-03-04T12:27:55.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f4a055740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b42c5f8f60 con 0x55b42c6999d0 2024-03-04T12:27:55.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f4a055740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b42c6a5cc0 con 0x55b42c6a2ed0 2024-03-04T12:27:55.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f44a0e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 0x55b42c6a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f4a055740 10 client.0 _ll_register_callbacks cb 0x55b42c5f71c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:55.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f44a0e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 0x55b42c6a0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53042/0 (socket says 172.21.15.60:53042) 2024-03-04T12:27:55.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f45a10640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c6999d0 0x55b42c69ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f44a0e640 1 -- 172.21.15.60:0/142693661 learned_addr learned my addr 172.21.15.60:0/142693661 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:55.218 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f45a10640 1 --2- 172.21.15.60:0/142693661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c6a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f44a0e640 1 -- 172.21.15.60:0/142693661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 msgr2=0x55b42c6a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f44a0e640 1 --2- 172.21.15.60:0/142693661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c6a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f44a0e640 1 -- 172.21.15.60:0/142693661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c6999d0 msgr2=0x55b42c69ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f44a0e640 1 --2- 172.21.15.60:0/142693661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c6999d0 0x55b42c69ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f44a0e640 1 -- 172.21.15.60:0/142693661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b42c3db7d0 con 0x55b42c69a2f0 2024-03-04T12:27:55.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f44a0e640 1 --2- 172.21.15.60:0/142693661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 0x55b42c6a0480 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f9f30037040 tx=0x7f9f30009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ca05d33b63080d server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:55.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.205+0000 7f9f28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:55.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f28ff9640 1 -- 172.21.15.60:0/142693661 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f30043070 con 0x55b42c69a2f0 2024-03-04T12:27:55.219 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f28ff9640 1 -- 172.21.15.60:0/142693661 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f9f30004450 con 0x55b42c69a2f0 2024-03-04T12:27:55.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f4a055740 1 -- 172.21.15.60:0/142693661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f3800fd10 con 0x55b42c69a2f0 2024-03-04T12:27:55.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f28ff9640 1 -- 172.21.15.60:0/142693661 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f3004f050 con 0x55b42c69a2f0 2024-03-04T12:27:55.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f4a055740 10 client.16524 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:55.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f4a055740 1 -- 172.21.15.60:0/142693661 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55b42c6b1e30 con 0x55b42c69a2f0 2024-03-04T12:27:55.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f28ff9640 1 -- 172.21.15.60:0/142693661 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f3800fd10 con 0x55b42c69a2f0 2024-03-04T12:27:55.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f28ff9640 1 -- 172.21.15.60:0/142693661 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f9f30032640 con 0x55b42c69a2f0 2024-03-04T12:27:55.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f4a055740 10 client.16524 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:55.220 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f4a055740 10 client.16524 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-04T12:27:55.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f4a055740 20 client.16524 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:55.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f4a055740 1 --2- 172.21.15.60:0/142693661 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55b42c6b3240 0x55b42c6b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f4a055740 4 client.16524 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:55.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.206+0000 7f9f4a055740 1 -- 172.21.15.60:0/142693661 --> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b42c60a400 con 0x55b42c6b3240 2024-03-04T12:27:55.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.207+0000 7f9f45a10640 1 --2- 172.21.15.60:0/142693661 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55b42c6b3240 0x55b42c6b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.207+0000 7f9f45a10640 1 --2- 172.21.15.60:0/142693661 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55b42c6b3240 0x55b42c6b5630 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d6374fc639d30f1a server_cookie=8e703226161a4ea2 in_seq=0 out_seq=0 2024-03-04T12:27:55.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.207+0000 7f9f28ff9640 10 client.16524 ms_handle_connect on v2:172.21.15.86:6838/3347800322 2024-03-04T12:27:55.221 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f28ff9640 1 -- 172.21.15.60:0/142693661 <== mds.1 v2:172.21.15.86:6838/3347800322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b42c60a400 con 0x55b42c6b3240 2024-03-04T12:27:55.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f28ff9640 10 client.16524 handle_command_reply: tid=0 2024-03-04T12:27:55.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f4a055740 1 client.16524 shutdown 2024-03-04T12:27:55.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f4a055740 1 -- 172.21.15.60:0/142693661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 msgr2=0x55b42c6a0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f4a055740 1 --2- 172.21.15.60:0/142693661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 0x55b42c6a0480 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f9f30037040 tx=0x7f9f30009490 comp rx=0 tx=0).stop 2024-03-04T12:27:55.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f4a055740 1 -- 172.21.15.60:0/142693661 shutdown_connections 2024-03-04T12:27:55.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f4a055740 1 --2- 172.21.15.60:0/142693661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c6a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f4a055740 1 --2- 172.21.15.60:0/142693661 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55b42c6b3240 0x55b42c6b5630 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f4a055740 1 --2- 172.21.15.60:0/142693661 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 0x55b42c6a0480 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.222 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f4a055740 1 --2- 172.21.15.60:0/142693661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c6999d0 0x55b42c69ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f4a055740 1 -- 172.21.15.60:0/142693661 >> 172.21.15.60:0/142693661 conn(0x55b42c694100 msgr2=0x55b42c695e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:55.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f28ff9640 0 client.16524 ms_handle_reset on v2:172.21.15.86:6838/3347800322 2024-03-04T12:27:55.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f4a055740 1 -- 172.21.15.60:0/142693661 shutdown_connections 2024-03-04T12:27:55.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f4a055740 1 -- 172.21.15.60:0/142693661 wait complete. 2024-03-04T12:27:55.223 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.208+0000 7f9f4a055740 20 client.16524 trim_cache size 0 max 16384 2024-03-04T12:27:55.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.224+0000 7f9f4a055740 1 Processor -- start 2024-03-04T12:27:55.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.225+0000 7f9f4a055740 1 -- start start 2024-03-04T12:27:55.232 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.226+0000 7f9f4a055740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 0x55b42c6a2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.226+0000 7f9f4a055740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c69a2f0 0x55b42c69e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.226+0000 7f9f4a055740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c69ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.226+0000 7f9f4a055740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f401ac3d0 con 0x55b42c6999d0 2024-03-04T12:27:55.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.226+0000 7f9f4a055740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f401addb0 con 0x55b42c69a2f0 2024-03-04T12:27:55.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.226+0000 7f9f4a055740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f401aca30 con 0x55b42c6a2ed0 2024-03-04T12:27:55.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.227+0000 7f9f45a10640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 0x55b42c6a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.227+0000 7f9f45a10640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 0x55b42c6a2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53056/0 (socket says 172.21.15.60:53056) 2024-03-04T12:27:55.234 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.227+0000 7f9f45a10640 1 -- 172.21.15.60:0/2397921771 learned_addr learned my addr 172.21.15.60:0/2397921771 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:55.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.227+0000 7f9f45a10640 1 -- 172.21.15.60:0/2397921771 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 msgr2=0x55b42c69ed50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T12:27:55.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.227+0000 7f9f45a10640 1 --2- 172.21.15.60:0/2397921771 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c69ed50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.227+0000 7f9f45a10640 1 -- 172.21.15.60:0/2397921771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c69a2f0 msgr2=0x55b42c69e810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-04T12:27:55.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.227+0000 7f9f45a10640 1 --2- 172.21.15.60:0/2397921771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c69a2f0 0x55b42c69e810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.227+0000 7f9f45a10640 1 -- 172.21.15.60:0/2397921771 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b42c69bb40 con 0x55b42c6999d0 2024-03-04T12:27:55.235 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.227+0000 7f9f45a10640 1 --2- 172.21.15.60:0/2397921771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 0x55b42c6a2600 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800fa40 tx=0x7f9f38011040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c254cb419089624 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:55.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.228+0000 7f9f2affd640 1 -- 172.21.15.60:0/2397921771 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f38005bd0 con 0x55b42c6999d0 2024-03-04T12:27:55.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.228+0000 7f9f2affd640 1 -- 172.21.15.60:0/2397921771 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f3802f400 con 0x55b42c6999d0 2024-03-04T12:27:55.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.228+0000 7f9f2affd640 1 -- 172.21.15.60:0/2397921771 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f38004da0 con 0x55b42c6999d0 2024-03-04T12:27:55.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.229+0000 7f9f4a055740 1 -- 172.21.15.60:0/2397921771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 msgr2=0x55b42c6a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.236 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.229+0000 7f9f4a055740 1 --2- 172.21.15.60:0/2397921771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 0x55b42c6a2600 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800fa40 tx=0x7f9f38011040 comp rx=0 tx=0).stop 2024-03-04T12:27:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.229+0000 7f9f4a055740 1 -- 172.21.15.60:0/2397921771 shutdown_connections 2024-03-04T12:27:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.229+0000 7f9f4a055740 1 --2- 172.21.15.60:0/2397921771 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c69ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.229+0000 7f9f4a055740 1 --2- 172.21.15.60:0/2397921771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c69a2f0 0x55b42c69e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.229+0000 7f9f4a055740 1 --2- 172.21.15.60:0/2397921771 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c6999d0 0x55b42c6a2600 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.229+0000 7f9f4a055740 1 -- 172.21.15.60:0/2397921771 >> 172.21.15.60:0/2397921771 conn(0x55b42c694100 msgr2=0x55b42c6b9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.230+0000 7f9f4a055740 1 -- 172.21.15.60:0/2397921771 shutdown_connections 2024-03-04T12:27:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.230+0000 7f9f4a055740 1 -- 172.21.15.60:0/2397921771 wait complete. 2024-03-04T12:27:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.230+0000 7f9f4a055740 1 -- start start 2024-03-04T12:27:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.230+0000 7f9f4a055740 1 Processor -- start 2024-03-04T12:27:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.230+0000 7f9f4a055740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c6999d0 0x55b42c6aa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.230+0000 7f9f4a055740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 0x55b42c6a8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.231+0000 7f9f4a055740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c6a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.231+0000 7f9f4a055740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f401ad750 con 0x55b42c69a2f0 2024-03-04T12:27:55.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.231+0000 7f9f4a055740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f401ad090 con 0x55b42c6999d0 2024-03-04T12:27:55.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.231+0000 7f9f4a055740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f401aa670 con 0x55b42c6a2ed0 2024-03-04T12:27:55.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.231+0000 7f9f4a055740 10 client.0 _ll_register_callbacks cb 0x7f9f38003b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:55.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.231+0000 7f9f44a0e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 0x55b42c6a8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.231+0000 7f9f44a0e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 0x55b42c6a8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:53064/0 (socket says 172.21.15.60:53064) 2024-03-04T12:27:55.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.231+0000 7f9f44a0e640 1 -- 172.21.15.60:0/1806694651 learned_addr learned my addr 172.21.15.60:0/1806694651 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:55.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.231+0000 7f9f45a10640 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c6a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.240 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.231+0000 7f9f45a10640 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c6999d0 0x55b42c6aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.232+0000 7f9f44a0e640 1 -- 172.21.15.60:0/1806694651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 msgr2=0x55b42c6a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.232+0000 7f9f44a0e640 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c6a8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.232+0000 7f9f44a0e640 1 -- 172.21.15.60:0/1806694651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c6999d0 msgr2=0x55b42c6aa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.243 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.232+0000 7f9f44a0e640 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c6999d0 0x55b42c6aa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.232+0000 7f9f44a0e640 1 -- 172.21.15.60:0/1806694651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b42c6a91e0 con 0x55b42c69a2f0 2024-03-04T12:27:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.232+0000 7f9f44a0e640 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 0x55b42c6a8560 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9f3002c180 tx=0x7f9f30032870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f882acacb43a845 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.232+0000 7f9f28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.232+0000 7f9f28ff9640 1 -- 172.21.15.60:0/1806694651 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f30058060 con 0x55b42c69a2f0 2024-03-04T12:27:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.232+0000 7f9f28ff9640 1 -- 172.21.15.60:0/1806694651 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7f9f30032d30 con 0x55b42c69a2f0 2024-03-04T12:27:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.233+0000 7f9f4a055740 1 -- 172.21.15.60:0/1806694651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f3802f400 con 0x55b42c69a2f0 2024-03-04T12:27:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.233+0000 7f9f28ff9640 1 -- 172.21.15.60:0/1806694651 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f3005f050 con 0x55b42c69a2f0 2024-03-04T12:27:55.244 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.233+0000 7f9f4a055740 10 client.16530 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.233+0000 7f9f4a055740 1 -- 172.21.15.60:0/1806694651 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55b42c70a690 con 0x55b42c69a2f0 2024-03-04T12:27:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.233+0000 7f9f28ff9640 1 -- 172.21.15.60:0/1806694651 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f3802f400 con 0x55b42c69a2f0 2024-03-04T12:27:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.233+0000 7f9f28ff9640 1 -- 172.21.15.60:0/1806694651 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f9f300052a0 con 0x55b42c69a2f0 2024-03-04T12:27:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.233+0000 7f9f4a055740 10 client.16530 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.233+0000 7f9f4a055740 10 client.16530 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-04T12:27:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.233+0000 7f9f4a055740 20 client.16530 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.233+0000 7f9f4a055740 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55b42c70ad00 0x55b42c70d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.234+0000 7f9f4a055740 4 client.16530 mds_command: new command op to 24487 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:27:55.245 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.234+0000 7f9f4a055740 1 -- 172.21.15.60:0/1806694651 --> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b42c604750 con 0x55b42c70ad00 2024-03-04T12:27:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.234+0000 7f9f45a10640 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55b42c70ad00 0x55b42c70d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.235+0000 7f9f45a10640 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55b42c70ad00 0x55b42c70d120 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=445c236ae8c30900 server_cookie=b6197187a876314b in_seq=0 out_seq=0 2024-03-04T12:27:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.235+0000 7f9f28ff9640 10 client.16530 ms_handle_connect on v2:172.21.15.86:6838/3347800322 2024-03-04T12:27:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.235+0000 7f9f28ff9640 1 -- 172.21.15.60:0/1806694651 <== mds.1 v2:172.21.15.86:6838/3347800322 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b42c604750 con 0x55b42c70ad00 2024-03-04T12:27:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.235+0000 7f9f28ff9640 10 client.16530 handle_command_reply: tid=0 2024-03-04T12:27:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.235+0000 7f9f4a055740 1 client.16530 shutdown 2024-03-04T12:27:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.235+0000 7f9f4a055740 1 -- 172.21.15.60:0/1806694651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 msgr2=0x55b42c6a8560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.246 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.235+0000 7f9f4a055740 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 0x55b42c6a8560 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9f3002c180 tx=0x7f9f30032870 comp rx=0 tx=0).stop 2024-03-04T12:27:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.235+0000 7f9f4a055740 1 -- 172.21.15.60:0/1806694651 shutdown_connections 2024-03-04T12:27:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.235+0000 7f9f4a055740 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b42c6a2ed0 0x55b42c6a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.236+0000 7f9f4a055740 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.86:6838/3347800322,v1:172.21.15.86:6839/3347800322] conn(0x55b42c70ad00 0x55b42c70d120 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.236+0000 7f9f4a055740 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x55b42c69a2f0 0x55b42c6a8560 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.236+0000 7f9f4a055740 1 --2- 172.21.15.60:0/1806694651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b42c6999d0 0x55b42c6aa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.236+0000 7f9f4a055740 1 -- 172.21.15.60:0/1806694651 >> 172.21.15.60:0/1806694651 conn(0x55b42c694100 msgr2=0x55b42c69e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.236+0000 7f9f28ff9640 0 client.16530 ms_handle_reset on v2:172.21.15.86:6838/3347800322 2024-03-04T12:27:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.236+0000 7f9f4a055740 1 -- 172.21.15.60:0/1806694651 shutdown_connections 2024-03-04T12:27:55.247 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.236+0000 7f9f4a055740 1 -- 172.21.15.60:0/1806694651 wait complete. 2024-03-04T12:27:55.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.236+0000 7f9f4a055740 20 client.16530 trim_cache size 0 max 16384 2024-03-04T12:27:55.248 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:27:55.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.237+0000 7f9f47473640 1 -- 172.21.15.60:0/583968938 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f9f2c0a1f30 msgr2=0x7f9f2c0a43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.237+0000 7f9f47473640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f9f2c0a1f30 0x7f9f2c0a43f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c0cbb80 tx=0x7f9f3c0d4550 comp rx=0 tx=0).stop 2024-03-04T12:27:55.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.237+0000 7f9f47473640 1 -- 172.21.15.60:0/583968938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f40101540 msgr2=0x7f9f4019b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.237+0000 7f9f47473640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f40101540 0x7f9f4019b6d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c005fd0 tx=0x7f9f3c00ee50 comp rx=0 tx=0).stop 2024-03-04T12:27:55.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.237+0000 7f9f47473640 1 -- 172.21.15.60:0/583968938 shutdown_connections 2024-03-04T12:27:55.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.237+0000 7f9f47473640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f40195840 0x7f9f40195cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.237+0000 7f9f47473640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f40106610 0x7f9f4019bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.237+0000 7f9f47473640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f9f2c0a1f30 0x7f9f2c0a43f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.237+0000 7f9f47473640 1 --2- 172.21.15.60:0/583968938 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f9f40101540 0x7f9f4019b6d0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.237+0000 7f9f47473640 1 -- 172.21.15.60:0/583968938 >> 172.21.15.60:0/583968938 conn(0x7f9f400fcdc0 msgr2=0x7f9f400fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:55.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.237+0000 7f9f47473640 1 -- 172.21.15.60:0/583968938 shutdown_connections 2024-03-04T12:27:55.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.237+0000 7f9f47473640 1 -- 172.21.15.60:0/583968938 wait complete. 2024-03-04T12:27:55.259 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-04T12:27:55.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-04T12:27:55.335 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:55 smithi060 ceph-mon[32745]: pgmap v64: 129 pgs: 129 active+clean; 68 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 543 KiB/s wr, 111 op/s 2024-03-04T12:27:55.335 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:55 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3301490872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:27:55.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:55 smithi086 ceph-mon[36851]: pgmap v64: 129 pgs: 129 active+clean; 68 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 543 KiB/s wr, 111 op/s 2024-03-04T12:27:55.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:55 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3301490872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:27:55.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:55 smithi063 ceph-mon[36717]: pgmap v64: 129 pgs: 129 active+clean; 68 MiB data, 504 MiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 543 KiB/s wr, 111 op/s 2024-03-04T12:27:55.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:55 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3301490872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:27:55.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.426+0000 7fba22f53640 1 -- 172.21.15.60:0/3061712444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1c105510 msgr2=0x7fba1c10d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.426+0000 7fba22f53640 1 --2- 172.21.15.60:0/3061712444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1c105510 0x7fba1c10d0e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fba14009ca0 tx=0x7fba1402c0f0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.434 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.426+0000 7fba22f53640 1 -- 172.21.15.60:0/3061712444 shutdown_connections 2024-03-04T12:27:55.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.426+0000 7fba22f53640 1 --2- 172.21.15.60:0/3061712444 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1c105510 0x7fba1c10d0e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.426+0000 7fba22f53640 1 --2- 172.21.15.60:0/3061712444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba1c100df0 0x7fba1c104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.426+0000 7fba22f53640 1 --2- 172.21.15.60:0/3061712444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba1c100440 0x7fba1c100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.426+0000 7fba22f53640 1 -- 172.21.15.60:0/3061712444 >> 172.21.15.60:0/3061712444 conn(0x7fba1c0773e0 msgr2=0x7fba1c0777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:55.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.426+0000 7fba22f53640 1 -- 172.21.15.60:0/3061712444 shutdown_connections 2024-03-04T12:27:55.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.426+0000 7fba22f53640 1 -- 172.21.15.60:0/3061712444 wait complete. 2024-03-04T12:27:55.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.426+0000 7fba22f53640 1 Processor -- start 2024-03-04T12:27:55.435 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba22f53640 1 -- start start 2024-03-04T12:27:55.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba22f53640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1c100440 0x7fba1c19d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba22f53640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba1c105510 0x7fba1c19dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba22f53640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba1c197930 0x7fba1c197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba22f53640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1105b0 con 0x7fba1c100440 2024-03-04T12:27:55.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba22f53640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c110430 con 0x7fba1c197930 2024-03-04T12:27:55.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba22f53640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c110730 con 0x7fba1c105510 2024-03-04T12:27:55.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba20cef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1c100440 0x7fba1c19d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba1bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba1c105510 0x7fba1c19dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba20cef640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1c100440 0x7fba1c19d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55784/0 (socket says 172.21.15.60:55784) 2024-03-04T12:27:55.436 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba20cef640 1 -- 172.21.15.60:0/1347497614 learned_addr learned my addr 172.21.15.60:0/1347497614 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:55.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba214f0640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba1c197930 0x7fba1c197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba20cef640 1 -- 172.21.15.60:0/1347497614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba1c105510 msgr2=0x7fba1c19dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba20cef640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba1c105510 0x7fba1c19dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba20cef640 1 -- 172.21.15.60:0/1347497614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba1c197930 msgr2=0x7fba1c197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba20cef640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba1c197930 0x7fba1c197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba20cef640 1 -- 172.21.15.60:0/1347497614 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c198450 con 0x7fba1c100440 2024-03-04T12:27:55.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba20cef640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1c100440 0x7fba1c19d780 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fba0c00c890 tx=0x7fba0c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:55.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba19ffb640 1 -- 172.21.15.60:0/1347497614 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c004220 con 0x7fba1c100440 2024-03-04T12:27:55.437 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.427+0000 7fba19ffb640 1 -- 172.21.15.60:0/1347497614 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba0c0043c0 con 0x7fba1c100440 2024-03-04T12:27:55.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.428+0000 7fba22f53640 1 -- 172.21.15.60:0/1347497614 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba1c198740 con 0x7fba1c100440 2024-03-04T12:27:55.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.428+0000 7fba19ffb640 1 -- 172.21.15.60:0/1347497614 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0c0049e0 con 0x7fba1c100440 2024-03-04T12:27:55.438 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.428+0000 7fba22f53640 1 -- 172.21.15.60:0/1347497614 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1c1a4990 con 0x7fba1c100440 2024-03-04T12:27:55.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.430+0000 7fba19ffb640 1 -- 172.21.15.60:0/1347497614 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7fba0c0079b0 con 0x7fba1c100440 2024-03-04T12:27:55.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.430+0000 7fba25c11740 1 Processor -- start 2024-03-04T12:27:55.444 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.430+0000 7fba19ffb640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fba040a1ef0 0x7fba040a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.430+0000 7fba25c11740 1 -- start start 2024-03-04T12:27:55.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.430+0000 7fba1bfff640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fba040a1ef0 0x7fba040a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.430+0000 7fba19ffb640 1 -- 172.21.15.60:0/1347497614 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fba0c0ce4b0 con 0x7fba1c100440 2024-03-04T12:27:55.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.430+0000 7fba25c11740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd4fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.430+0000 7fba25c11740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd5cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.430+0000 7fba25c11740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd5f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.430+0000 7fba25c11740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x56182dd16dc0 con 0x56182dd5d2e0 2024-03-04T12:27:55.445 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.430+0000 7fba25c11740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x56182dcb2ec0 con 0x56182dd4f700 2024-03-04T12:27:55.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.430+0000 7fba25c11740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56182dc5bd30 con 0x56182dd50020 2024-03-04T12:27:55.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba214f0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd5f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba20cef640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd4fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba214f0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd5f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55786/0 (socket says 172.21.15.60:55786) 2024-03-04T12:27:55.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba214f0640 1 -- 172.21.15.60:0/3679560930 learned_addr learned my addr 172.21.15.60:0/3679560930 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:55.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba1bfff640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fba040a1ef0 0x7fba040a43b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fba1c0696e0 tx=0x7fba1000a2d0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:55.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba214f0640 1 -- 172.21.15.60:0/3679560930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 msgr2=0x56182dd5cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba214f0640 1 --2- 172.21.15.60:0/3679560930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd5cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.446 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba214f0640 1 -- 172.21.15.60:0/3679560930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 msgr2=0x56182dd4fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba214f0640 1 --2- 172.21.15.60:0/3679560930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd4fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba214f0640 1 -- 172.21.15.60:0/3679560930 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56182d9c73f0 con 0x56182dd5d2e0 2024-03-04T12:27:55.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba20cef640 1 --2- 172.21.15.60:0/3679560930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd4fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:55.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba214f0640 1 --2- 172.21.15.60:0/3679560930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd5f6d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fba14009ca0 tx=0x7fba14032480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f1991f0d317a2c server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:55.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba02ffd640 1 -- 172.21.15.60:0/3679560930 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba14043070 con 0x56182dd5d2e0 2024-03-04T12:27:55.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba02ffd640 1 -- 172.21.15.60:0/3679560930 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba14032c90 con 0x56182dd5d2e0 2024-03-04T12:27:55.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.431+0000 7fba02ffd640 1 -- 172.21.15.60:0/3679560930 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba14039040 con 0x56182dd5d2e0 2024-03-04T12:27:55.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.432+0000 7fba25c11740 1 -- 172.21.15.60:0/3679560930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 msgr2=0x56182dd5f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.447 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.432+0000 7fba25c11740 1 --2- 172.21.15.60:0/3679560930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd5f6d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fba14009ca0 tx=0x7fba14032480 comp rx=0 tx=0).stop 2024-03-04T12:27:55.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.432+0000 7fba25c11740 1 -- 172.21.15.60:0/3679560930 shutdown_connections 2024-03-04T12:27:55.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.432+0000 7fba25c11740 1 --2- 172.21.15.60:0/3679560930 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd5f6d0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.432+0000 7fba25c11740 1 --2- 172.21.15.60:0/3679560930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd5cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.432+0000 7fba25c11740 1 --2- 172.21.15.60:0/3679560930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd4fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.432+0000 7fba25c11740 1 -- 172.21.15.60:0/3679560930 >> 172.21.15.60:0/3679560930 conn(0x56182dd4a100 msgr2=0x56182dd4a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:55.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.432+0000 7fba25c11740 1 -- 172.21.15.60:0/3679560930 shutdown_connections 2024-03-04T12:27:55.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.432+0000 7fba25c11740 1 -- 172.21.15.60:0/3679560930 wait complete. 2024-03-04T12:27:55.448 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.432+0000 7fba25c11740 1 -- start start 2024-03-04T12:27:55.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.432+0000 7fba25c11740 1 Processor -- start 2024-03-04T12:27:55.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba25c11740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd5a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba25c11740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd5a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba25c11740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd5aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba25c11740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x56182dc514a0 con 0x56182dd5d2e0 2024-03-04T12:27:55.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba25c11740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x56182dcaef60 con 0x56182dd4f700 2024-03-04T12:27:55.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba25c11740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56182dd60040 con 0x56182dd50020 2024-03-04T12:27:55.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba25c11740 10 client.0 _ll_register_callbacks cb 0x56182dcad1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:55.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba214f0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd5aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba1bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd5a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba214f0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd5aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55794/0 (socket says 172.21.15.60:55794) 2024-03-04T12:27:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba214f0640 1 -- 172.21.15.60:0/56875719 learned_addr learned my addr 172.21.15.60:0/56875719 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba1bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd5a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.60:56416/0 (socket says 172.21.15.60:56416) 2024-03-04T12:27:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba20cef640 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd5a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba1bfff640 1 -- 172.21.15.60:0/56875719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 msgr2=0x56182dd5a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba1bfff640 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd5a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba1bfff640 1 -- 172.21.15.60:0/56875719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 msgr2=0x56182dd5aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba1bfff640 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd5aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba1bfff640 1 -- 172.21.15.60:0/56875719 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56182da917d0 con 0x56182dd50020 2024-03-04T12:27:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.433+0000 7fba214f0640 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd5aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-04T12:27:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fba1bfff640 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd5a990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fba1000afd0 tx=0x7fba100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff3d9631cbacd867 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fb9fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-03-04T12:27:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fb9fbfff640 1 -- 172.21.15.60:0/56875719 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba10016070 con 0x56182dd50020 2024-03-04T12:27:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fba25c11740 1 -- 172.21.15.60:0/56875719 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba14032c90 con 0x56182dd50020 2024-03-04T12:27:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fb9fbfff640 1 -- 172.21.15.60:0/56875719 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fba1001bc10 con 0x56182dd50020 2024-03-04T12:27:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fb9fbfff640 1 -- 172.21.15.60:0/56875719 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba10030050 con 0x56182dd50020 2024-03-04T12:27:55.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fba25c11740 10 client.26038 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fba25c11740 1 -- 172.21.15.60:0/56875719 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x56182dd6c390 con 0x56182dd50020 2024-03-04T12:27:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fb9fbfff640 1 -- 172.21.15.60:0/56875719 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba14032c90 con 0x56182dd50020 2024-03-04T12:27:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fb9fbfff640 1 -- 172.21.15.60:0/56875719 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fba1001c460 con 0x56182dd50020 2024-03-04T12:27:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fba25c11740 10 client.26038 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fba25c11740 10 client.26038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fba25c11740 20 client.26038 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fba25c11740 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56182dd6d7f0 0x56182dd6fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fba25c11740 4 client.26038 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-04T12:27:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.434+0000 7fba25c11740 1 -- 172.21.15.60:0/56875719 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56182dcc0400 con 0x56182dd6d7f0 2024-03-04T12:27:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.435+0000 7fba20cef640 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56182dd6d7f0 0x56182dd6fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.435+0000 7fba20cef640 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56182dd6d7f0 0x56182dd6fbe0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f2ce033f9b4d341 server_cookie=125a73e197754c0 in_seq=0 out_seq=0 2024-03-04T12:27:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.435+0000 7fb9fbfff640 10 client.26038 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fb9fbfff640 1 -- 172.21.15.60:0/56875719 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56182dcc0400 con 0x56182dd6d7f0 2024-03-04T12:27:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fb9fbfff640 10 client.26038 handle_command_reply: tid=0 2024-03-04T12:27:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fba25c11740 1 client.26038 shutdown 2024-03-04T12:27:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fba25c11740 1 -- 172.21.15.60:0/56875719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 msgr2=0x56182dd5a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fba25c11740 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd5a990 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fba1000afd0 tx=0x7fba100046c0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fba25c11740 1 -- 172.21.15.60:0/56875719 shutdown_connections 2024-03-04T12:27:55.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fba25c11740 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56182dd6d7f0 0x56182dd6fbe0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fba25c11740 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd5aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fba25c11740 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd5a990 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fba25c11740 1 --2- 172.21.15.60:0/56875719 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd5a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fb9fbfff640 0 client.26038 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:55.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fba25c11740 1 -- 172.21.15.60:0/56875719 >> 172.21.15.60:0/56875719 conn(0x56182dd4a100 msgr2=0x56182dd4be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:55.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fba25c11740 1 -- 172.21.15.60:0/56875719 shutdown_connections 2024-03-04T12:27:55.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fba25c11740 1 -- 172.21.15.60:0/56875719 wait complete. 2024-03-04T12:27:55.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.436+0000 7fba25c11740 20 client.26038 trim_cache size 0 max 16384 2024-03-04T12:27:55.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.448+0000 7fba25c11740 1 Processor -- start 2024-03-04T12:27:55.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.448+0000 7fba25c11740 1 -- start start 2024-03-04T12:27:55.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.448+0000 7fba25c11740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd5cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.448+0000 7fba25c11740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd50020 0x56182dd58d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.448+0000 7fba25c11740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd5d2e0 0x56182dd59260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.448+0000 7fba25c11740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1ac250 con 0x56182dd50020 2024-03-04T12:27:55.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.448+0000 7fba25c11740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1adc30 con 0x56182dd4f700 2024-03-04T12:27:55.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.448+0000 7fba25c11740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1ac8b0 con 0x56182dd5d2e0 2024-03-04T12:27:55.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.448+0000 7fba1bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd50020 0x56182dd58d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.448+0000 7fba1bfff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd50020 0x56182dd58d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55808/0 (socket says 172.21.15.60:55808) 2024-03-04T12:27:55.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.448+0000 7fba1bfff640 1 -- 172.21.15.60:0/491171396 learned_addr learned my addr 172.21.15.60:0/491171396 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:55.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.449+0000 7fba214f0640 1 --2- 172.21.15.60:0/491171396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd5d2e0 0x56182dd59260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.449+0000 7fba20cef640 1 --2- 172.21.15.60:0/491171396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd5cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.449+0000 7fba1bfff640 1 -- 172.21.15.60:0/491171396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd5d2e0 msgr2=0x56182dd59260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.449+0000 7fba1bfff640 1 --2- 172.21.15.60:0/491171396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd5d2e0 0x56182dd59260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.449+0000 7fba1bfff640 1 -- 172.21.15.60:0/491171396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 msgr2=0x56182dd5cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.449+0000 7fba1bfff640 1 --2- 172.21.15.60:0/491171396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd5cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.449+0000 7fba1bfff640 1 -- 172.21.15.60:0/491171396 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56182dd69d90 con 0x56182dd50020 2024-03-04T12:27:55.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.449+0000 7fba1bfff640 1 --2- 172.21.15.60:0/491171396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd50020 0x56182dd58d20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fba1001c8b0 tx=0x7fba1000a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e938c257f5def6cd server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:55.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.449+0000 7fba027fc640 1 -- 172.21.15.60:0/491171396 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba10016070 con 0x56182dd50020 2024-03-04T12:27:55.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.449+0000 7fba027fc640 1 -- 172.21.15.60:0/491171396 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba10039070 con 0x56182dd50020 2024-03-04T12:27:55.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.449+0000 7fba027fc640 1 -- 172.21.15.60:0/491171396 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba10005370 con 0x56182dd50020 2024-03-04T12:27:55.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 -- 172.21.15.60:0/491171396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd50020 msgr2=0x56182dd58d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 --2- 172.21.15.60:0/491171396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd50020 0x56182dd58d20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fba1001c8b0 tx=0x7fba1000a4d0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 -- 172.21.15.60:0/491171396 shutdown_connections 2024-03-04T12:27:55.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 --2- 172.21.15.60:0/491171396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd5d2e0 0x56182dd59260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 --2- 172.21.15.60:0/491171396 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd50020 0x56182dd58d20 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 --2- 172.21.15.60:0/491171396 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd5cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 -- 172.21.15.60:0/491171396 >> 172.21.15.60:0/491171396 conn(0x56182dd4a100 msgr2=0x56182dd55760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:55.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 -- 172.21.15.60:0/491171396 shutdown_connections 2024-03-04T12:27:55.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 -- 172.21.15.60:0/491171396 wait complete. 2024-03-04T12:27:55.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 -- start start 2024-03-04T12:27:55.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 Processor -- start 2024-03-04T12:27:55.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd64000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd64540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.450+0000 7fba25c11740 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd624a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba25c11740 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1ad5d0 con 0x56182dd5d2e0 2024-03-04T12:27:55.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba25c11740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1acf10 con 0x56182dd4f700 2024-03-04T12:27:55.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba25c11740 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c1aa4f0 con 0x56182dd50020 2024-03-04T12:27:55.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba25c11740 10 client.0 _ll_register_callbacks cb 0x7fba14037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-04T12:27:55.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba214f0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd624a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba214f0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd624a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55810/0 (socket says 172.21.15.60:55810) 2024-03-04T12:27:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba214f0640 1 -- 172.21.15.60:0/797813880 learned_addr learned my addr 172.21.15.60:0/797813880 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:27:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba1bfff640 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd64540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba20cef640 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd64000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba214f0640 1 -- 172.21.15.60:0/797813880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 msgr2=0x56182dd64540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba214f0640 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd64540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba214f0640 1 -- 172.21.15.60:0/797813880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 msgr2=0x56182dd64000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-04T12:27:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba214f0640 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd64000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba214f0640 1 -- 172.21.15.60:0/797813880 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56182dd62be0 con 0x56182dd5d2e0 2024-03-04T12:27:55.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba20cef640 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd64000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:27:55.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba214f0640 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd624a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fba14000f80 tx=0x7fba14032b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9cd09ba0141b921 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:27:55.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fb9fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.60:3300/0 2024-03-04T12:27:55.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fb9fbfff640 1 -- 172.21.15.60:0/797813880 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba14050060 con 0x56182dd5d2e0 2024-03-04T12:27:55.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fb9fbfff640 1 -- 172.21.15.60:0/797813880 <== mon.0 v2:172.21.15.60:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9451+0+0 (secure 0 0 0) 0x7fba14048860 con 0x56182dd5d2e0 2024-03-04T12:27:55.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba25c11740 1 -- 172.21.15.60:0/797813880 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba10039070 con 0x56182dd5d2e0 2024-03-04T12:27:55.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fb9fbfff640 1 -- 172.21.15.60:0/797813880 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba14057050 con 0x56182dd5d2e0 2024-03-04T12:27:55.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.451+0000 7fba25c11740 10 client.16548 fetch_fsmap learned FSMap version 36 2024-03-04T12:27:55.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.452+0000 7fba25c11740 1 -- 172.21.15.60:0/797813880 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x56182ddc4810 con 0x56182dd5d2e0 2024-03-04T12:27:55.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.452+0000 7fb9fbfff640 1 -- 172.21.15.60:0/797813880 <== mon.0 v2:172.21.15.60:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba10039070 con 0x56182dd5d2e0 2024-03-04T12:27:55.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.452+0000 7fb9fbfff640 1 -- 172.21.15.60:0/797813880 <== mon.0 v2:172.21.15.60:3300/0 5 ==== fsmap(e 36) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fba14034ab0 con 0x56182dd5d2e0 2024-03-04T12:27:55.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.452+0000 7fba25c11740 10 client.16548 fetch_fsmap finished waiting for FSMap version 36 2024-03-04T12:27:55.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.452+0000 7fba25c11740 10 client.16548 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-04T12:27:55.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.452+0000 7fba25c11740 20 client.16548 populate_metadata read hostname 'smithi060' 2024-03-04T12:27:55.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.452+0000 7fba25c11740 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56182ddc4ba0 0x56182ddc6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:27:55.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.452+0000 7fba25c11740 4 client.16548 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-04T12:27:55.471 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.452+0000 7fba25c11740 1 -- 172.21.15.60:0/797813880 --> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56182dcba750 con 0x56182ddc4ba0 2024-03-04T12:27:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.452+0000 7fba20cef640 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56182ddc4ba0 0x56182ddc6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:27:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fb9fbfff640 10 client.16548 ms_handle_connect on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba20cef640 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56182ddc4ba0 0x56182ddc6f90 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2329098429e4780 server_cookie=43d424c8aa2687a1 in_seq=0 out_seq=0 2024-03-04T12:27:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fb9fbfff640 1 -- 172.21.15.60:0/797813880 <== mds.0 v2:172.21.15.63:6838/2728229139 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56182dcba750 con 0x56182ddc4ba0 2024-03-04T12:27:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fb9fbfff640 10 client.16548 handle_command_reply: tid=0 2024-03-04T12:27:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba25c11740 1 client.16548 shutdown 2024-03-04T12:27:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba25c11740 1 -- 172.21.15.60:0/797813880 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 msgr2=0x56182dd624a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba25c11740 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd624a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fba14000f80 tx=0x7fba14032b80 comp rx=0 tx=0).stop 2024-03-04T12:27:55.472 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba25c11740 1 -- 172.21.15.60:0/797813880 shutdown_connections 2024-03-04T12:27:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba25c11740 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.63:6838/2728229139,v1:172.21.15.63:6839/2728229139] conn(0x56182ddc4ba0 0x56182ddc6f90 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba25c11740 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x56182dd5d2e0 0x56182dd624a0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba25c11740 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56182dd50020 0x56182dd64540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fb9fbfff640 0 client.16548 ms_handle_reset on v2:172.21.15.63:6838/2728229139 2024-03-04T12:27:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba25c11740 1 --2- 172.21.15.60:0/797813880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56182dd4f700 0x56182dd64000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba25c11740 1 -- 172.21.15.60:0/797813880 >> 172.21.15.60:0/797813880 conn(0x56182dd4a100 msgr2=0x56182dd58ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba25c11740 1 -- 172.21.15.60:0/797813880 shutdown_connections 2024-03-04T12:27:55.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba25c11740 1 -- 172.21.15.60:0/797813880 wait complete. 2024-03-04T12:27:55.474 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.453+0000 7fba25c11740 20 client.16548 trim_cache size 0 max 16384 2024-03-04T12:27:55.474 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2024-03-04T12:27:55.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.454+0000 7fba22f53640 1 -- 172.21.15.60:0/1347497614 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fba040a1ef0 msgr2=0x7fba040a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.454+0000 7fba22f53640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fba040a1ef0 0x7fba040a43b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fba1c0696e0 tx=0x7fba1000a2d0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.475 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.454+0000 7fba22f53640 1 -- 172.21.15.60:0/1347497614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1c100440 msgr2=0x7fba1c19d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:27:55.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.454+0000 7fba22f53640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1c100440 0x7fba1c19d780 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fba0c00c890 tx=0x7fba0c00cd50 comp rx=0 tx=0).stop 2024-03-04T12:27:55.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.454+0000 7fba22f53640 1 -- 172.21.15.60:0/1347497614 shutdown_connections 2024-03-04T12:27:55.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.454+0000 7fba22f53640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba1c197930 0x7fba1c197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.454+0000 7fba22f53640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba1c105510 0x7fba1c19dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.454+0000 7fba22f53640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fba040a1ef0 0x7fba040a43b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.454+0000 7fba22f53640 1 --2- 172.21.15.60:0/1347497614 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fba1c100440 0x7fba1c19d780 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:27:55.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.454+0000 7fba22f53640 1 -- 172.21.15.60:0/1347497614 >> 172.21.15.60:0/1347497614 conn(0x7fba1c0773e0 msgr2=0x7fba1c104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:27:55.476 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.454+0000 7fba22f53640 1 -- 172.21.15.60:0/1347497614 shutdown_connections 2024-03-04T12:27:55.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:27:55.454+0000 7fba22f53640 1 -- 172.21.15.60:0/1347497614 wait complete. 2024-03-04T12:27:55.478 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 15145518, 'ino': 1099511628114, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/trunc.c'}, {'damage_type': 'backtrace', 'id': 36876980, 'ino': 1099511627857, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/inject'}, {'damage_type': 'backtrace', 'id': 50067782, 'ino': 1099511628028, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/looptest.c'}, {'damage_type': 'backtrace', 'id': 60798369, 'ino': 1099511628039, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/nametest.c'}, {'damage_type': 'backtrace', 'id': 85690099, 'ino': 1099511628304, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.idx'}, {'damage_type': 'backtrace', 'id': 88145242, 'ino': 1099511627908, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/datapid.c'}, {'damage_type': 'backtrace', 'id': 88331178, 'ino': 1099511627982, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/devzero.c'}, {'damage_type': 'backtrace', 'id': 104036562, 'ino': 1099511627820, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/Makefile'}, {'damage_type': 'backtrace', 'id': 116796498, 'ino': 1099511627925, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/fsx.c'}, {'damage_type': 'backtrace', 'id': 135401196, 'ino': 1099511628329, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/example.config'}, {'damage_type': 'backtrace', 'id': 142309834, 'ino': 1099511628019, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/iopat.c'}, {'damage_type': 'backtrace', 'id': 144526712, 'ino': 1099511627891, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildgrouplist'}, {'damage_type': 'backtrace', 'id': 145937807, 'ino': 1099511628102, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_open_tmpfiles.c'}, {'damage_type': 'backtrace', 'id': 165925414, 'ino': 1099511627977, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/chprojid_fail.c'}, {'damage_type': 'backtrace', 'id': 174463506, 'ino': 1099511628032, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/md5.h'}, {'damage_type': 'backtrace', 'id': 183839499, 'ino': 1099511627988, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirstress.c'}, {'damage_type': 'backtrace', 'id': 194602196, 'ino': 1099511627861, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/module'}, {'damage_type': 'backtrace', 'id': 197431264, 'ino': 1099511627898, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/install-sh'}, {'damage_type': 'backtrace', 'id': 200511642, 'ino': 1099511627859, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/log'}, {'damage_type': 'backtrace', 'id': 236941976, 'ino': 1099511628052, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/popdir.pl'}, {'damage_type': 'backtrace', 'id': 237638720, 'ino': 1099511628333, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/CHANGES'}, {'damage_type': 'backtrace', 'id': 245010395, 'ino': 1099511628043, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/parse-extent-tree.awk'}, {'damage_type': 'backtrace', 'id': 245626071, 'ino': 1099511627950, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-cow-race.c'}, {'damage_type': 'backtrace', 'id': 248848676, 'ino': 1099511627942, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/randomize.awk'}, {'damage_type': 'backtrace', 'id': 262902837, 'ino': 1099511628076, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_attr_corruption.c'}, {'damage_type': 'backtrace', 'id': 268944219, 'ino': 1099511627915, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/str_to_bytes.c'}, {'damage_type': 'backtrace', 'id': 274913497, 'ino': 1099511627910, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/forker.c'}, {'damage_type': 'backtrace', 'id': 285157337, 'ino': 1099511628002, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2fs'}, {'damage_type': 'backtrace', 'id': 303494259, 'ino': 1099511627924, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/fsstress.c'}, {'damage_type': 'backtrace', 'id': 309293263, 'ino': 1099511627900, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/pattern.h'}, {'damage_type': 'backtrace', 'id': 334196108, 'ino': 1099511627937, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_liburing.m4'}, {'damage_type': 'backtrace', 'id': 340412425, 'ino': 1099511627786, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/prepare-commit-msg.sample'}, {'damage_type': 'backtrace', 'id': 345740528, 'ino': 1099511628055, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/punch-alternating.c'}, {'damage_type': 'backtrace', 'id': 371976726, 'ino': 1099511628064, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/scaleread.c'}, {'damage_type': 'backtrace', 'id': 376981377, 'ino': 1099511628326, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common'}, {'damage_type': 'backtrace', 'id': 378520951, 'ino': 1099511627948, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-append-write-fallocate-race.c'}, {'damage_type': 'backtrace', 'id': 382904155, 'ino': 1099511627837, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/cgroup2'}, {'damage_type': 'backtrace', 'id': 392669768, 'ino': 1099511627829, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/acinclude.m4'}, {'damage_type': 'backtrace', 'id': 403369894, 'ino': 1099511628108, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_snapshot_deleted_subvolume.c'}, {'damage_type': 'backtrace', 'id': 430760593, 'ino': 1099511627850, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/f2fs'}, {'damage_type': 'backtrace', 'id': 446850512, 'ino': 1099511627886, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/group-names.txt'}, {'damage_type': 'backtrace', 'id': 451935590, 'ino': 1099511628113, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/testx.c'}, {'damage_type': 'backtrace', 'id': 457769834, 'ino': 1099511628297, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/heads'}, {'damage_type': 'backtrace', 'id': 464604364, 'ino': 1099511627998, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fiexchange.h'}, {'damage_type': 'backtrace', 'id': 480470474, 'ino': 1099511627813, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/HEAD'}, {'damage_type': 'backtrace', 'id': 507370495, 'ino': 1099511628047, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/fio-insert-and-compare.py'}, {'damage_type': 'backtrace', 'id': 511646524, 'ino': 1099511627932, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_gdbmdev.m4'}, {'damage_type': 'backtrace', 'id': 517341081, 'ino': 1099511627894, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/dataascii.h'}, {'damage_type': 'backtrace', 'id': 523849527, 'ino': 1099511627792, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/post-update.sample'}, {'damage_type': 'backtrace', 'id': 526939251, 'ino': 1099511628345, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/Makefile'}, {'damage_type': 'backtrace', 'id': 536947007, 'ino': 1099511628004, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fs_perms.c'}, {'damage_type': 'backtrace', 'id': 539966682, 'ino': 1099511627979, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dbtest.c'}, {'damage_type': 'backtrace', 'id': 558632771, 'ino': 1099511628298, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/tags'}, {'damage_type': 'backtrace', 'id': 569002490, 'ino': 1099511628035, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmap-rw-fault.c'}, {'damage_type': 'backtrace', 'id': 587353601, 'ino': 1099511627875, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/test_names'}, {'damage_type': 'backtrace', 'id': 595273088, 'ino': 1099511628021, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/listxattr.c'}, {'damage_type': 'backtrace', 'id': 597139248, 'ino': 1099511627992, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/enospc_unlink.c'}, {'damage_type': 'backtrace', 'id': 649633086, 'ino': 1099511627928, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/rwtest.sh'}, {'damage_type': 'backtrace', 'id': 664703078, 'ino': 1099511628003, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2fs_check'}, {'damage_type': 'backtrace', 'id': 676740681, 'ino': 1099511628057, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/randholes.c'}, {'damage_type': 'backtrace', 'id': 683113306, 'ino': 1099511627822, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README'}, {'damage_type': 'backtrace', 'id': 685400260, 'ino': 1099511627916, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/string_to_tokens.c'}, {'damage_type': 'backtrace', 'id': 697263990, 'ino': 1099511627907, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/databin.c'}, {'damage_type': 'backtrace', 'id': 701451165, 'ino': 1099511627963, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aiocp.c'}, {'damage_type': 'backtrace', 'id': 703550772, 'ino': 1099511628096, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_stale_pmd.c'}, {'damage_type': 'backtrace', 'id': 735430208, 'ino': 1099511628083, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_enospc.c'}, {'damage_type': 'backtrace', 'id': 737167532, 'ino': 1099511628309, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs'}, {'damage_type': 'backtrace', 'id': 737580633, 'ino': 1099511628038, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/multi_open_unlink.c'}, {'damage_type': 'backtrace', 'id': 740376772, 'ino': 1099511628034, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mkswap.c'}, {'damage_type': 'backtrace', 'id': 741420734, 'ino': 1099511627949, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-append-write-read-race.c'}, {'damage_type': 'backtrace', 'id': 750881278, 'ino': 1099511627956, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-invalidate-failure.c'}, {'damage_type': 'backtrace', 'id': 760639474, 'ino': 1099511627784, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git'}, {'damage_type': 'backtrace', 'id': 783648266, 'ino': 1099511628027, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/loggen.c'}, {'damage_type': 'backtrace', 'id': 788307367, 'ino': 1099511628106, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_reflink_read_race.c'}, {'damage_type': 'backtrace', 'id': 791556209, 'ino': 1099511628339, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/Makefile'}, {'damage_type': 'backtrace', 'id': 792021749, 'ino': 1099511627901, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/random_range.h'}, {'damage_type': 'backtrace', 'id': 812478707, 'ino': 1099511628042, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/parse-dev-tree.awk'}, {'damage_type': 'backtrace', 'id': 812875497, 'ino': 1099511628089, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_holes.c'}, {'damage_type': 'backtrace', 'id': 814617634, 'ino': 1099511628024, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/log-writes.c'}, {'damage_type': 'backtrace', 'id': 822566053, 'ino': 1099511627941, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/new'}, {'damage_type': 'backtrace', 'id': 831096825, 'ino': 1099511627921, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/aio-stress.c'}, {'damage_type': 'backtrace', 'id': 834510631, 'ino': 1099511628010, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ftrunc.c'}, {'damage_type': 'backtrace', 'id': 857032667, 'ino': 1099511627897, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/forker.h'}, {'damage_type': 'backtrace', 'id': 858959478, 'ino': 1099511627868, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/quota'}, {'damage_type': 'backtrace', 'id': 885708074, 'ino': 1099511627835, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/casefold'}, {'damage_type': 'backtrace', 'id': 905911049, 'ino': 1099511627849, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ext4'}, {'damage_type': 'backtrace', 'id': 915298806, 'ino': 1099511627851, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/fail_make_request'}, {'damage_type': 'backtrace', 'id': 919406646, 'ino': 1099511628323, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/Makefile'}, {'damage_type': 'backtrace', 'id': 928752688, 'ino': 1099511627911, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/open_flags.c'}, {'damage_type': 'backtrace', 'id': 946178551, 'ino': 1099511628318, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/LICENSES'}, {'damage_type': 'backtrace', 'id': 956583036, 'ino': 1099511627962, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-last-ref-held-by-io.c'}, {'damage_type': 'backtrace', 'id': 962108217, 'ino': 1099511628344, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress'}, {'damage_type': 'backtrace', 'id': 967747680, 'ino': 1099511628327, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/Makefile'}, {'damage_type': 'backtrace', 'id': 974836631, 'ino': 1099511627919, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/local.config.example'}, {'damage_type': 'backtrace', 'id': 992217432, 'ino': 1099511627867, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/punch'}, {'damage_type': 'backtrace', 'id': 996357490, 'ino': 1099511627874, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/scsi_debug'}, {'damage_type': 'backtrace', 'id': 1004504255, 'ino': 1099511628071, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/stale_handle.c'}, {'damage_type': 'backtrace', 'id': 1010996799, 'ino': 1099511628059, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/rename.c'}, {'damage_type': 'backtrace', 'id': 1015005010, 'ino': 1099511627873, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/report'}, {'damage_type': 'backtrace', 'id': 1024148256, 'ino': 1099511627975, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_unlink_test_modified.c'}, {'damage_type': 'backtrace', 'id': 1030921239, 'ino': 1099511628331, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/README'}, {'damage_type': 'backtrace', 'id': 1052722801, 'ino': 1099511628294, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-commit.sample'}, {'damage_type': 'backtrace', 'id': 1065166274, 'ino': 1099511627938, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_utilies.m4'}, {'damage_type': 'backtrace', 'id': 1072318883, 'ino': 1099511628291, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/info'}, {'damage_type': 'backtrace', 'id': 1084052773, 'ino': 1099511628088, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_getcwd.c'}, {'damage_type': 'backtrace', 'id': 1090389746, 'ino': 1099511628017, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/holes.c'}, {'damage_type': 'backtrace', 'id': 1101001460, 'ino': 1099511627863, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/overlay'}, {'damage_type': 'backtrace', 'id': 1109726417, 'ino': 1099511628020, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/itrash.c'}, {'damage_type': 'backtrace', 'id': 1124813141, 'ino': 1099511628072, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/stat_test.c'}, {'damage_type': 'backtrace', 'id': 1131369212, 'ino': 1099511627976, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/checkpoint_journal.c'}, {'damage_type': 'backtrace', 'id': 1142603624, 'ino': 1099511628075, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_access_root.c'}, {'damage_type': 'backtrace', 'id': 1146241474, 'ino': 1099511628116, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/unwritten_mmap.c'}, {'damage_type': 'backtrace', 'id': 1164357481, 'ino': 1099511627877, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ubifs'}, {'damage_type': 'backtrace', 'id': 1189687453, 'ino': 1099511628041, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/open_by_handle.c'}, {'damage_type': 'backtrace', 'id': 1201006250, 'ino': 1099511627860, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/metadump'}, {'damage_type': 'backtrace', 'id': 1205126378, 'ino': 1099511627834, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/btrfs'}, {'damage_type': 'backtrace', 'id': 1211042958, 'ino': 1099511628110, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_truncate_cmtime.c'}, {'damage_type': 'backtrace', 'id': 1229411314, 'ino': 1099511627793, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/update.sample'}, {'damage_type': 'backtrace', 'id': 1234305986, 'ino': 1099511627852, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filestreams'}, {'damage_type': 'backtrace', 'id': 1243055710, 'ino': 1099511627917, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/tlibio.c'}, {'damage_type': 'backtrace', 'id': 1275224862, 'ino': 1099511628007, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fstest.c'}, {'damage_type': 'backtrace', 'id': 1308907745, 'ino': 1099511628099, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_writev_overlap.c'}, {'damage_type': 'backtrace', 'id': 1311439776, 'ino': 1099511628301, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack'}, {'damage_type': 'backtrace', 'id': 1374468917, 'ino': 1099511627961, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-io-setup-with-nonwritable-context-pointer.c'}, {'damage_type': 'backtrace', 'id': 1384768012, 'ino': 1099511628009, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fsync-tester.c'}, {'damage_type': 'backtrace', 'id': 1389521992, 'ino': 1099511627833, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/attr'}, {'damage_type': 'backtrace', 'id': 1400065255, 'ino': 1099511627819, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/MAINTAINERS'}, {'damage_type': 'backtrace', 'id': 1409895252, 'ino': 1099511628040, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/nsexec.c'}, {'damage_type': 'backtrace', 'id': 1424720120, 'ino': 1099511627880, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/zoned'}, {'damage_type': 'backtrace', 'id': 1448578678, 'ino': 1099511627923, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/doio.h'}, {'damage_type': 'backtrace', 'id': 1466189143, 'ino': 1099511627973, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_null_ocount.c'}, {'damage_type': 'backtrace', 'id': 1481907459, 'ino': 1099511627856, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/gcov'}, {'damage_type': 'backtrace', 'id': 1487432253, 'ino': 1099511628070, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/splice2pipe.c'}, {'damage_type': 'backtrace', 'id': 1490127066, 'ino': 1099511627960, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-free-ring-with-bogus-nr-pages.c'}, {'damage_type': 'backtrace', 'id': 1506481499, 'ino': 1099511627968, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/append_writer.c'}, {'damage_type': 'backtrace', 'id': 1511751689, 'ino': 1099511627974, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bulkstat_unlink_test.c'}, {'damage_type': 'backtrace', 'id': 1516416134, 'ino': 1099511627926, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/growfiles.c'}, {'damage_type': 'backtrace', 'id': 1521069307, 'ino': 1099511628343, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/Makefile'}, {'damage_type': 'backtrace', 'id': 1535957113, 'ino': 1099511627966, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/allocstale.c'}, {'damage_type': 'backtrace', 'id': 1553501209, 'ino': 1099511627972, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/btrfs_crc32c_forged_name.py'}, {'damage_type': 'backtrace', 'id': 1556666008, 'ino': 1099511628346, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes'}, {'damage_type': 'backtrace', 'id': 1558305500, 'ino': 1099511628302, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/info'}, {'damage_type': 'backtrace', 'id': 1569185443, 'ino': 1099511627797, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-merge-commit.sample'}, {'damage_type': 'backtrace', 'id': 1593071906, 'ino': 1099511628085, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ext4_dax_journal_corruption.c'}, {'damage_type': 'backtrace', 'id': 1597883187, 'ino': 1099511628300, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects'}, {'damage_type': 'backtrace', 'id': 1605746668, 'ino': 1099511627791, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-receive.sample'}, {'damage_type': 'backtrace', 'id': 1619195040, 'ino': 1099511627876, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/tracing'}, {'damage_type': 'backtrace', 'id': 1660101853, 'ino': 1099511628018, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/holetest.c'}, {'damage_type': 'backtrace', 'id': 1666641881, 'ino': 1099511627823, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.config-sections'}, {'damage_type': 'backtrace', 'id': 1676794923, 'ino': 1099511627854, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filter.btrfs'}, {'damage_type': 'backtrace', 'id': 1682794586, 'ino': 1099511628107, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_rename_overwrite.c'}, {'damage_type': 'backtrace', 'id': 1691620670, 'ino': 1099511627828, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/VERSION'}, {'damage_type': 'backtrace', 'id': 1695000015, 'ino': 1099511627866, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/preamble'}, {'damage_type': 'backtrace', 'id': 1710848717, 'ino': 1099511627818, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/index.lock'}, {'damage_type': 'backtrace', 'id': 1716180899, 'ino': 1099511628347, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/Makefile'}, {'damage_type': 'backtrace', 'id': 1718693801, 'ino': 1099511628093, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_cow_race.c'}, {'damage_type': 'backtrace', 'id': 1730540472, 'ino': 1099511628086, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_futimens.c'}, {'damage_type': 'backtrace', 'id': 1750583676, 'ino': 1099511627895, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/databin.h'}, {'damage_type': 'backtrace', 'id': 1769005384, 'ino': 1099511627964, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aiodio_sparse2.c'}, {'damage_type': 'backtrace', 'id': 1792594302, 'ino': 1099511627821, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/Makepkgs'}, {'damage_type': 'backtrace', 'id': 1800336582, 'ino': 1099511627914, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/random_range.c'}, {'damage_type': 'backtrace', 'id': 1833401501, 'ino': 1099511627853, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/filter'}, {'damage_type': 'backtrace', 'id': 1861101553, 'ino': 1099511627881, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs/localhost.config'}, {'damage_type': 'backtrace', 'id': 1861891348, 'ino': 1099511627832, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/check'}, {'damage_type': 'backtrace', 'id': 1875339343, 'ino': 1099511628119, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/usemem.c'}, {'damage_type': 'backtrace', 'id': 1880999404, 'ino': 1099511628051, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/popattr.py'}, {'damage_type': 'backtrace', 'id': 1887092554, 'ino': 1099511627922, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/doio.c'}, {'damage_type': 'backtrace', 'id': 1900411416, 'ino': 1099511628340, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4'}, {'damage_type': 'backtrace', 'id': 1907186617, 'ino': 1099511628067, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/seek_sanity_test.c'}, {'damage_type': 'backtrace', 'id': 1942938966, 'ino': 1099511628014, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/global.h'}, {'damage_type': 'backtrace', 'id': 1950789847, 'ino': 1099511628335, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/Makefile'}, {'damage_type': 'backtrace', 'id': 1953647681, 'ino': 1099511628005, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fscrypt-crypt-util.c'}, {'damage_type': 'backtrace', 'id': 1954885257, 'ino': 1099511628115, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/truncfile.c'}, {'damage_type': 'backtrace', 'id': 1959745075, 'ino': 1099511628054, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/preallo_rw_pattern_writer.c'}, {'damage_type': 'backtrace', 'id': 1972643696, 'ino': 1099511628317, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.gitignore'}, {'damage_type': 'backtrace', 'id': 1976873353, 'ino': 1099511627788, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/fsmonitor-watchman.sample'}, {'damage_type': 'backtrace', 'id': 1983957300, 'ino': 1099511627896, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/file_lock.h'}, {'damage_type': 'backtrace', 'id': 1985275399, 'ino': 1099511628600, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/FioCompare.py'}, {'damage_type': 'backtrace', 'id': 1987659131, 'ino': 1099511628029, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/lstat64.c'}, {'damage_type': 'backtrace', 'id': 1988699713, 'ino': 1099511628312, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/origin/HEAD'}, {'damage_type': 'backtrace', 'id': 1996970934, 'ino': 1099511628334, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include'}, {'damage_type': 'backtrace', 'id': 2037207467, 'ino': 1099511627838, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/config'}, {'damage_type': 'backtrace', 'id': 2047917912, 'ino': 1099511627869, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/rc'}, {'damage_type': 'backtrace', 'id': 2072022007, 'ino': 1099511627991, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/e4compact.c'}, {'damage_type': 'backtrace', 'id': 2074161152, 'ino': 1099511627824, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.device-mapper'}, {'damage_type': 'backtrace', 'id': 2083427417, 'ino': 1099511628049, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/generate-schema.py'}, {'damage_type': 'backtrace', 'id': 2089464262, 'ino': 1099511628082, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_encrypted_d_revalidate.c'}, {'damage_type': 'backtrace', 'id': 2092640605, 'ino': 1099511628308, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs'}, {'damage_type': 'backtrace', 'id': 2100479378, 'ino': 1099511628063, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/runas.c'}, {'damage_type': 'backtrace', 'id': 2111082012, 'ino': 1099511627918, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/write_log.c'}, {'damage_type': 'backtrace', 'id': 2150168531, 'ino': 1099511628338, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp'}, {'damage_type': 'backtrace', 'id': 2169531563, 'ino': 1099511627812, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/packed-refs'}, {'damage_type': 'backtrace', 'id': 2178915993, 'ino': 1099511628013, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/getpagesize.c'}, {'damage_type': 'backtrace', 'id': 2183117465, 'ino': 1099511627787, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/push-to-checkout.sample'}, {'damage_type': 'backtrace', 'id': 2184397299, 'ino': 1099511628306, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/origin'}, {'damage_type': 'backtrace', 'id': 2190136576, 'ino': 1099511628050, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/permname.c'}, {'damage_type': 'backtrace', 'id': 2203566559, 'ino': 1099511627994, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fake-dump-rootino.c'}, {'damage_type': 'backtrace', 'id': 2205318374, 'ino': 1099511627985, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-invalidate-cache.c'}, {'damage_type': 'backtrace', 'id': 2212629300, 'ino': 1099511628342, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src'}, {'damage_type': 'backtrace', 'id': 2216314706, 'ino': 1099511627887, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/requirement-checking.txt'}, {'damage_type': 'backtrace', 'id': 2248410613, 'ino': 1099511627983, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-buf-fault.c'}, {'damage_type': 'backtrace', 'id': 2251882362, 'ino': 1099511627888, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/testing-afs.txt'}, {'damage_type': 'backtrace', 'id': 2288193026, 'ino': 1099511627789, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-rebase.sample'}, {'damage_type': 'backtrace', 'id': 2296820300, 'ino': 1099511628030, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/makeextents.c'}, {'damage_type': 'backtrace', 'id': 2301856156, 'ino': 1099511627841, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmdust'}, {'damage_type': 'backtrace', 'id': 2326725402, 'ino': 1099511627912, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/pattern.c'}, {'damage_type': 'backtrace', 'id': 2327762176, 'ino': 1099511628100, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mtab.c'}, {'damage_type': 'backtrace', 'id': 2331817066, 'ino': 1099511627965, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/alloc.c'}, {'damage_type': 'backtrace', 'id': 2337066432, 'ino': 1099511628058, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/readdir-while-renames.c'}, {'damage_type': 'backtrace', 'id': 2347564201, 'ino': 1099511628111, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_truncate_self.c'}, {'damage_type': 'backtrace', 'id': 2354561068, 'ino': 1099511628599, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf'}, {'damage_type': 'backtrace', 'id': 2390138741, 'ino': 1099511627980, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/deduperace.c'}, {'damage_type': 'backtrace', 'id': 2398659611, 'ino': 1099511628062, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/rewinddir-test.c'}, {'damage_type': 'backtrace', 'id': 2402242469, 'ino': 1099511628045, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/Makefile'}, {'damage_type': 'backtrace', 'id': 2413215785, 'ino': 1099511627890, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/builddefs.in'}, {'damage_type': 'backtrace', 'id': 2416427284, 'ino': 1099511627790, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/applypatch-msg.sample'}, {'damage_type': 'backtrace', 'id': 2427353895, 'ino': 1099511628016, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/helpers.h'}, {'damage_type': 'backtrace', 'id': 2448401042, 'ino': 1099511627892, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildmacros'}, {'damage_type': 'backtrace', 'id': 2449274002, 'ino': 1099511628011, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/genhashnames.c'}, {'damage_type': 'backtrace', 'id': 2465814984, 'ino': 1099511628046, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/ResultData.py'}, {'damage_type': 'backtrace', 'id': 2470274186, 'ino': 1099511627933, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_globals.m4'}, {'damage_type': 'backtrace', 'id': 2483091912, 'ino': 1099511627899, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/open_flags.h'}, {'damage_type': 'backtrace', 'id': 2484676621, 'ino': 1099511628307, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes/origin/HEAD'}, {'damage_type': 'backtrace', 'id': 2500626330, 'ino': 1099511628319, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/LICENSES/GPL-2.0'}, {'damage_type': 'backtrace', 'id': 2520274436, 'ino': 1099511628037, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmapcat.c'}, {'damage_type': 'backtrace', 'id': 2539463223, 'ino': 1099511628008, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fsync-err.c'}, {'damage_type': 'backtrace', 'id': 2545860023, 'ino': 1099511628296, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs'}, {'damage_type': 'backtrace', 'id': 2554269433, 'ino': 1099511627936, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libcdev.m4'}, {'damage_type': 'backtrace', 'id': 2567975590, 'ino': 1099511627978, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/cloner.c'}, {'damage_type': 'backtrace', 'id': 2572926129, 'ino': 1099511627845, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmlogwrites'}, {'damage_type': 'backtrace', 'id': 2582181752, 'ino': 1099511627993, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/ext4_resize.c'}, {'damage_type': 'backtrace', 'id': 2588338894, 'ino': 1099511628022, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/locktest.c'}, {'damage_type': 'backtrace', 'id': 2597676126, 'ino': 1099511627931, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_aiodev.m4'}, {'damage_type': 'backtrace', 'id': 2605596910, 'ino': 1099511628305, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/remotes'}, {'damage_type': 'backtrace', 'id': 2619429103, 'ino': 1099511628303, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/objects/pack/pack-94293b5f01ab34c3beb7aa28b61585a6d7e5f843.pack'}, {'damage_type': 'backtrace', 'id': 2622451403, 'ino': 1099511627955, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-hole-filling-race.c'}, {'damage_type': 'backtrace', 'id': 2630336533, 'ino': 1099511627872, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/repair'}, {'damage_type': 'backtrace', 'id': 2638639856, 'ino': 1099511628048, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/fio-results.sql'}, {'damage_type': 'backtrace', 'id': 2646369343, 'ino': 1099511628095, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_fallocate.c'}, {'damage_type': 'backtrace', 'id': 2686887760, 'ino': 1099511628078, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_create_short_dirs.c'}, {'damage_type': 'backtrace', 'id': 2702380316, 'ino': 1099511628092, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_cow_memory_failure.c'}, {'damage_type': 'backtrace', 'id': 2705806874, 'ino': 1099511628332, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc'}, {'damage_type': 'backtrace', 'id': 2707270517, 'ino': 1099511628293, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks'}, {'damage_type': 'backtrace', 'id': 2716645146, 'ino': 1099511628324, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/tar'}, {'damage_type': 'backtrace', 'id': 2717690756, 'ino': 1099511627839, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/defrag'}, {'damage_type': 'backtrace', 'id': 2731991507, 'ino': 1099511627794, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-push.sample'}, {'damage_type': 'backtrace', 'id': 2742071533, 'ino': 1099511628316, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/heads/master'}, {'damage_type': 'backtrace', 'id': 2799709160, 'ino': 1099511627878, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/verity'}, {'damage_type': 'backtrace', 'id': 2808835004, 'ino': 1099511628101, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ofd_locks.c'}, {'damage_type': 'backtrace', 'id': 2811251371, 'ino': 1099511628012, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/getdevicesize.c'}, {'damage_type': 'backtrace', 'id': 2834186746, 'ino': 1099511627796, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/pre-applypatch.sample'}, {'damage_type': 'backtrace', 'id': 2835385329, 'ino': 1099511628094, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_dio.c'}, {'damage_type': 'backtrace', 'id': 2845170803, 'ino': 1099511627810, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/shallow'}, {'damage_type': 'backtrace', 'id': 2855053345, 'ino': 1099511628053, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/preallo_rw_pattern_reader.c'}, {'damage_type': 'backtrace', 'id': 2857268722, 'ino': 1099511627984, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dio-interleaved.c'}, {'damage_type': 'backtrace', 'id': 2857837464, 'ino': 1099511627939, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_uuiddev.m4'}, {'damage_type': 'backtrace', 'id': 2859909662, 'ino': 1099511628118, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/uring_read_fault.c'}, {'damage_type': 'backtrace', 'id': 2859928072, 'ino': 1099511627831, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/xfstests.spec.in'}, {'damage_type': 'backtrace', 'id': 2866811136, 'ino': 1099511628074, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/swapon.c'}, {'damage_type': 'backtrace', 'id': 2870400145, 'ino': 1099511627905, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/write_log.h'}, {'damage_type': 'backtrace', 'id': 2928213493, 'ino': 1099511628314, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/HEAD'}, {'damage_type': 'backtrace', 'id': 2935356529, 'ino': 1099511627817, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/config'}, {'damage_type': 'backtrace', 'id': 2948753456, 'ino': 1099511628001, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2attr'}, {'damage_type': 'backtrace', 'id': 2951101475, 'ino': 1099511627825, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.fuse'}, {'damage_type': 'backtrace', 'id': 2951388549, 'ino': 1099511627870, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/reflink'}, {'damage_type': 'backtrace', 'id': 2978079813, 'ino': 1099511628097, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_write_ro.c'}, {'damage_type': 'backtrace', 'id': 2980343275, 'ino': 1099511628065, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/scaleread.sh'}, {'damage_type': 'backtrace', 'id': 2989763929, 'ino': 1099511628036, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/mmap-write-concurrent.c'}, {'damage_type': 'backtrace', 'id': 2992788983, 'ino': 1099511627848, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/encrypt'}, {'damage_type': 'backtrace', 'id': 2995824591, 'ino': 1099511628081, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_type.c'}, {'damage_type': 'backtrace', 'id': 3091096923, 'ino': 1099511627871, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/renameat2'}, {'damage_type': 'backtrace', 'id': 3100143408, 'ino': 1099511628066, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/seek_copy_test.c'}, {'damage_type': 'backtrace', 'id': 3101856120, 'ino': 1099511627864, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/perf'}, {'damage_type': 'backtrace', 'id': 3106897311, 'ino': 1099511627883, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/rc.sysinit'}, {'damage_type': 'backtrace', 'id': 3127721826, 'ino': 1099511627885, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/auxiliary-programs.txt'}, {'damage_type': 'backtrace', 'id': 3128406579, 'ino': 1099511628006, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fssum.c'}, {'damage_type': 'backtrace', 'id': 3130216736, 'ino': 1099511627906, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/dataascii.c'}, {'damage_type': 'backtrace', 'id': 3176564404, 'ino': 1099511627971, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/bstat.c'}, {'damage_type': 'backtrace', 'id': 3180752784, 'ino': 1099511627934, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libbtrfs.m4'}, {'damage_type': 'backtrace', 'id': 3186562381, 'ino': 1099511627954, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-fcntl-race.c'}, {'damage_type': 'backtrace', 'id': 3201659955, 'ino': 1099511628073, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/statx.h'}, {'damage_type': 'backtrace', 'id': 3208147820, 'ino': 1099511627862, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/nfs'}, {'damage_type': 'backtrace', 'id': 3208857766, 'ino': 1099511627913, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/random.c'}, {'damage_type': 'backtrace', 'id': 3218196769, 'ino': 1099511627858, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/locktest'}, {'damage_type': 'backtrace', 'id': 3224183881, 'ino': 1099511628098, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_writev.c'}, {'damage_type': 'backtrace', 'id': 3251585662, 'ino': 1099511627865, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/populate'}, {'damage_type': 'backtrace', 'id': 3253773870, 'ino': 1099511627904, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/tlibio.h'}, {'damage_type': 'backtrace', 'id': 3262883244, 'ino': 1099511628322, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm'}, {'damage_type': 'backtrace', 'id': 3273395424, 'ino': 1099511628056, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/pwrite_mmap_blocked.c'}, {'damage_type': 'backtrace', 'id': 3299963160, 'ino': 1099511628087, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_get_file_time.c'}, {'damage_type': 'backtrace', 'id': 3326793751, 'ino': 1099511627935, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_libcap.m4'}, {'damage_type': 'backtrace', 'id': 3327733039, 'ino': 1099511627785, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/description'}, {'damage_type': 'backtrace', 'id': 3339193630, 'ino': 1099511628330, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash'}, {'damage_type': 'backtrace', 'id': 3340835770, 'ino': 1099511627989, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dmerror'}, {'damage_type': 'backtrace', 'id': 3343339479, 'ino': 1099511628061, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/resvtest.c'}, {'damage_type': 'backtrace', 'id': 3354200374, 'ino': 1099511628031, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/md5.c'}, {'damage_type': 'backtrace', 'id': 3356213728, 'ino': 1099511627842, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmerror'}, {'damage_type': 'backtrace', 'id': 3357979348, 'ino': 1099511627987, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirperf.c'}, {'damage_type': 'backtrace', 'id': 3361629239, 'ino': 1099511627959, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-write-verify.c'}, {'damage_type': 'backtrace', 'id': 3377667437, 'ino': 1099511628060, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/renameat2.c'}, {'damage_type': 'backtrace', 'id': 3389429376, 'ino': 1099511628120, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/uuid_ioctl.c'}, {'damage_type': 'backtrace', 'id': 3391331412, 'ino': 1099511628295, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/branches'}, {'damage_type': 'backtrace', 'id': 3421780079, 'ino': 1099511628077, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_create_long_dirs.c'}, {'damage_type': 'backtrace', 'id': 3429840995, 'ino': 1099511628325, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/tar/Makefile'}, {'damage_type': 'backtrace', 'id': 3453264649, 'ino': 1099511627795, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/hooks/commit-msg.sample'}, {'damage_type': 'backtrace', 'id': 3453817756, 'ino': 1099511628313, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/refs/heads/master'}, {'damage_type': 'backtrace', 'id': 3460907172, 'ino': 1099511628336, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib'}, {'damage_type': 'backtrace', 'id': 3465260133, 'ino': 1099511627879, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/xfs'}, {'damage_type': 'backtrace', 'id': 3485690069, 'ino': 1099511627893, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/buildrules'}, {'damage_type': 'backtrace', 'id': 3490249718, 'ino': 1099511627970, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/attr_replace_test.c'}, {'damage_type': 'backtrace', 'id': 3496363722, 'ino': 1099511628079, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_offset.c'}, {'damage_type': 'backtrace', 'id': 3507448255, 'ino': 1099511627945, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/soak'}, {'damage_type': 'backtrace', 'id': 3514950093, 'ino': 1099511627827, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.selftest'}, {'damage_type': 'backtrace', 'id': 3520903312, 'ino': 1099511628025, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/log-writes.h'}, {'damage_type': 'backtrace', 'id': 3528989467, 'ino': 1099511627997, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fiemap-tester.c'}, {'damage_type': 'backtrace', 'id': 3529883614, 'ino': 1099511628090, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_immutable.c'}, {'damage_type': 'backtrace', 'id': 3530971332, 'ino': 1099511628311, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes/origin'}, {'damage_type': 'backtrace', 'id': 3536932502, 'ino': 1099511628091, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_mmap_collision.c'}, {'damage_type': 'backtrace', 'id': 3547803984, 'ino': 1099511628080, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_dir_offset2.c'}, {'damage_type': 'backtrace', 'id': 3553356909, 'ino': 1099511627920, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lsqa.pl'}, {'damage_type': 'backtrace', 'id': 3554358368, 'ino': 1099511627844, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmhugedisk'}, {'damage_type': 'backtrace', 'id': 3563556352, 'ino': 1099511627927, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/ltp/iogen.c'}, {'damage_type': 'backtrace', 'id': 3585296020, 'ino': 1099511628315, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/heads'}, {'damage_type': 'backtrace', 'id': 3603659650, 'ino': 1099511627990, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dumpfile'}, {'damage_type': 'backtrace', 'id': 3620596871, 'ino': 1099511628337, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/Makefile'}, {'damage_type': 'backtrace', 'id': 3641180848, 'ino': 1099511628341, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/Makefile'}, {'damage_type': 'backtrace', 'id': 3646294562, 'ino': 1099511628328, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configs'}, {'damage_type': 'backtrace', 'id': 3705172137, 'ino': 1099511627843, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmflakey'}, {'damage_type': 'backtrace', 'id': 3729684653, 'ino': 1099511627889, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/doc/xunit.xsd'}, {'damage_type': 'backtrace', 'id': 3732462890, 'ino': 1099511627902, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/str_to_bytes.h'}, {'damage_type': 'backtrace', 'id': 3748510104, 'ino': 1099511627882, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/configure.ac'}, {'damage_type': 'backtrace', 'id': 3759180496, 'ino': 1099511627830, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/rpm/macros.template'}, {'damage_type': 'backtrace', 'id': 3763215343, 'ino': 1099511627826, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/README.overlay'}, {'damage_type': 'backtrace', 'id': 3781611481, 'ino': 1099511627840, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmdelay'}, {'damage_type': 'backtrace', 'id': 3787454378, 'ino': 1099511627909, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/lib/file_lock.c'}, {'damage_type': 'backtrace', 'id': 3820179523, 'ino': 1099511628109, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_stripealign.c'}, {'damage_type': 'backtrace', 'id': 3826331620, 'ino': 1099511628292, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/info/exclude'}, {'damage_type': 'backtrace', 'id': 3827698450, 'ino': 1099511628104, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_2.c'}, {'damage_type': 'backtrace', 'id': 3848130917, 'ino': 1099511628026, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/replay-log.c'}, {'damage_type': 'backtrace', 'id': 3859052483, 'ino': 1099511627999, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill.c'}, {'damage_type': 'backtrace', 'id': 3868895663, 'ino': 1099511627940, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_xfslibs.m4'}, {'damage_type': 'backtrace', 'id': 3871175009, 'ino': 1099511627836, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/ceph'}, {'damage_type': 'backtrace', 'id': 3880677228, 'ino': 1099511628015, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/godown.c'}, {'damage_type': 'backtrace', 'id': 3883403115, 'ino': 1099511627944, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/setup'}, {'damage_type': 'backtrace', 'id': 3904545215, 'ino': 1099511627930, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/package_acldev.m4'}, {'damage_type': 'backtrace', 'id': 3913745567, 'ino': 1099511627969, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/attr-list-by-handle-cursor-test.c'}, {'damage_type': 'backtrace', 'id': 3922500342, 'ino': 1099511627952, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-eof-race.c'}, {'damage_type': 'backtrace', 'id': 3938786484, 'ino': 1099511627903, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/include/string_to_tokens.h'}, {'damage_type': 'backtrace', 'id': 3961789900, 'ino': 1099511628000, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fill2.c'}, {'damage_type': 'backtrace', 'id': 3963874142, 'ino': 1099511627996, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/feature.c'}, {'damage_type': 'backtrace', 'id': 3964209623, 'ino': 1099511628320, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build'}, {'damage_type': 'backtrace', 'id': 3975625141, 'ino': 1099511627986, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/dirhash_collide.c'}, {'damage_type': 'backtrace', 'id': 3999156906, 'ino': 1099511627958, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-subblock-eof-read.c'}, {'damage_type': 'backtrace', 'id': 4003210665, 'ino': 1099511628310, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/.git/logs/refs/remotes'}, {'damage_type': 'backtrace', 'id': 4003322253, 'ino': 1099511627946, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/af_unix.c'}, {'damage_type': 'backtrace', 'id': 4015915125, 'ino': 1099511627957, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-invalidate-readahead.c'}, {'damage_type': 'backtrace', 'id': 4017106498, 'ino': 1099511627943, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/release.sh'}, {'damage_type': 'backtrace', 'id': 4024991526, 'ino': 1099511628069, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/splice-test.c'}, {'damage_type': 'backtrace', 'id': 4027032892, 'ino': 1099511628084, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_ext4_dax_inline_corruption.c'}, {'damage_type': 'backtrace', 'id': 4034769142, 'ino': 1099511627947, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/SOURCE'}, {'damage_type': 'backtrace', 'id': 4059425113, 'ino': 1099511628023, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/log-writes/SOURCE'}, {'damage_type': 'backtrace', 'id': 4062650391, 'ino': 1099511627995, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/fault.c'}, {'damage_type': 'backtrace', 'id': 4096812389, 'ino': 1099511627953, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-extend-stat.c'}, {'damage_type': 'backtrace', 'id': 4102732125, 'ino': 1099511628117, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/unwritten_sync.c'}, {'damage_type': 'backtrace', 'id': 4103847433, 'ino': 1099511628112, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/test-nextquota.c'}, {'damage_type': 'backtrace', 'id': 4115184060, 'ino': 1099511628068, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/soak_duration.awk'}, {'damage_type': 'backtrace', 'id': 4120013376, 'ino': 1099511627951, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/aio-dio-regress/aio-dio-cycle-write.c'}, {'damage_type': 'backtrace', 'id': 4131724534, 'ino': 1099511628103, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_1.c'}, {'damage_type': 'backtrace', 'id': 4139792175, 'ino': 1099511627967, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/append_reader.c'}, {'damage_type': 'backtrace', 'id': 4161360526, 'ino': 1099511628321, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/build/Makefile'}, {'damage_type': 'backtrace', 'id': 4186743557, 'ino': 1099511627855, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/fuzzy'}, {'damage_type': 'backtrace', 'id': 4221619859, 'ino': 1099511627847, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dump'}, {'damage_type': 'backtrace', 'id': 4225237416, 'ino': 1099511627981, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/detached_mounts_propagation.c'}, {'damage_type': 'backtrace', 'id': 4232775347, 'ino': 1099511628044, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/perf/FioResultDecoder.py'}, {'damage_type': 'backtrace', 'id': 4235409953, 'ino': 1099511627846, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/common/dmthin'}, {'damage_type': 'backtrace', 'id': 4235944732, 'ino': 1099511628105, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/t_readdir_3.c'}, {'damage_type': 'backtrace', 'id': 4248435851, 'ino': 1099511627884, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/crash/xfscrash'}, {'damage_type': 'backtrace', 'id': 4260419634, 'ino': 1099511627929, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/m4/multilib.m4'}, {'damage_type': 'backtrace', 'id': 4291632764, 'ino': 1099511628033, 'path': '/volumes/qa/sv_1/5936704d-a11e-4f77-91f6-4c6a307f08e1/client.0/tmp/xfstests-dev/src/metaperf.c'}] 2024-03-04T12:27:55.479 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-03-04T12:27:57.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:57 smithi086 ceph-mon[36851]: pgmap v65: 129 pgs: 129 active+clean; 78 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 1.5 MiB/s wr, 233 op/s 2024-03-04T12:27:57.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:57 smithi063 ceph-mon[36717]: pgmap v65: 129 pgs: 129 active+clean; 78 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 1.5 MiB/s wr, 233 op/s 2024-03-04T12:27:57.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:57 smithi060 ceph-mon[32745]: pgmap v65: 129 pgs: 129 active+clean; 78 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 1.5 MiB/s wr, 233 op/s 2024-03-04T12:27:59.351 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:27:59 smithi086 ceph-mon[36851]: pgmap v66: 129 pgs: 129 active+clean; 78 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 1.1 MiB/s wr, 156 op/s 2024-03-04T12:27:59.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:27:59 smithi063 ceph-mon[36717]: pgmap v66: 129 pgs: 129 active+clean; 78 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 1.1 MiB/s wr, 156 op/s 2024-03-04T12:27:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:27:59 smithi060 ceph-mon[32745]: pgmap v66: 129 pgs: 129 active+clean; 78 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 1.1 MiB/s wr, 156 op/s 2024-03-04T12:28:00.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:00 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:28:00.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:00 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:28:00.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:00 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:28:01.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:01 smithi086 ceph-mon[36851]: pgmap v67: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 1.2 MiB/s wr, 193 op/s 2024-03-04T12:28:01.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:01 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:28:01.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:01 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:28:01.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:01 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:28:01.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:01 smithi063 ceph-mon[36717]: pgmap v67: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 1.2 MiB/s wr, 193 op/s 2024-03-04T12:28:01.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:01 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:28:01.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:01 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:28:01.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:01 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:28:01.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:01 smithi060 ceph-mon[32745]: pgmap v67: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 1.2 MiB/s wr, 193 op/s 2024-03-04T12:28:01.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:01 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:28:01.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:01 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:28:01.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:01 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:28:03.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:03 smithi086 ceph-mon[36851]: pgmap v68: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.0 MiB/s wr, 158 op/s 2024-03-04T12:28:03.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:03 smithi063 ceph-mon[36717]: pgmap v68: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.0 MiB/s wr, 158 op/s 2024-03-04T12:28:03.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:03 smithi060 ceph-mon[32745]: pgmap v68: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.0 MiB/s wr, 158 op/s 2024-03-04T12:28:05.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:05 smithi086 ceph-mon[36851]: pgmap v69: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.0 MiB/s wr, 158 op/s 2024-03-04T12:28:05.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:05 smithi063 ceph-mon[36717]: pgmap v69: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.0 MiB/s wr, 158 op/s 2024-03-04T12:28:05.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:05 smithi060 ceph-mon[32745]: pgmap v69: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 1.0 MiB/s wr, 158 op/s 2024-03-04T12:28:07.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:07 smithi086 ceph-mon[36851]: pgmap v70: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.0 MiB/s wr, 179 op/s 2024-03-04T12:28:07.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:07 smithi063 ceph-mon[36717]: pgmap v70: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.0 MiB/s wr, 179 op/s 2024-03-04T12:28:07.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:07 smithi060 ceph-mon[32745]: pgmap v70: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 1.0 MiB/s wr, 179 op/s 2024-03-04T12:28:09.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:09 smithi086 ceph-mon[36851]: pgmap v71: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 120 KiB/s wr, 58 op/s 2024-03-04T12:28:09.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:09 smithi063 ceph-mon[36717]: pgmap v71: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 120 KiB/s wr, 58 op/s 2024-03-04T12:28:09.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:09 smithi060 ceph-mon[32745]: pgmap v71: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 120 KiB/s wr, 58 op/s 2024-03-04T12:28:11.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:11 smithi086 ceph-mon[36851]: pgmap v72: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 120 KiB/s wr, 62 op/s 2024-03-04T12:28:11.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:11 smithi063 ceph-mon[36717]: pgmap v72: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 120 KiB/s wr, 62 op/s 2024-03-04T12:28:11.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:11 smithi060 ceph-mon[32745]: pgmap v72: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 120 KiB/s wr, 62 op/s 2024-03-04T12:28:13.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:13 smithi086 ceph-mon[36851]: pgmap v73: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 38 KiB/s wr, 25 op/s 2024-03-04T12:28:13.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:13 smithi063 ceph-mon[36717]: pgmap v73: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 38 KiB/s wr, 25 op/s 2024-03-04T12:28:13.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:13 smithi060 ceph-mon[32745]: pgmap v73: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 38 KiB/s wr, 25 op/s 2024-03-04T12:28:15.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:15 smithi086 ceph-mon[36851]: pgmap v74: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 38 KiB/s wr, 25 op/s 2024-03-04T12:28:15.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:15 smithi063 ceph-mon[36717]: pgmap v74: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 38 KiB/s wr, 25 op/s 2024-03-04T12:28:15.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:15 smithi060 ceph-mon[32745]: pgmap v74: 129 pgs: 129 active+clean; 77 MiB data, 507 MiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 38 KiB/s wr, 25 op/s 2024-03-04T12:28:17.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:17 smithi086 ceph-mon[36851]: pgmap v75: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 111 KiB/s wr, 32 op/s 2024-03-04T12:28:17.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:17 smithi063 ceph-mon[36717]: pgmap v75: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 111 KiB/s wr, 32 op/s 2024-03-04T12:28:17.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:17 smithi060 ceph-mon[32745]: pgmap v75: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 111 KiB/s wr, 32 op/s 2024-03-04T12:28:19.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:19 smithi086 ceph-mon[36851]: pgmap v76: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 73 KiB/s wr, 10 op/s 2024-03-04T12:28:19.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:19 smithi063 ceph-mon[36717]: pgmap v76: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 73 KiB/s wr, 10 op/s 2024-03-04T12:28:19.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:19 smithi060 ceph-mon[32745]: pgmap v76: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 73 KiB/s wr, 10 op/s 2024-03-04T12:28:21.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:21 smithi086 ceph-mon[36851]: pgmap v77: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 73 KiB/s wr, 15 op/s 2024-03-04T12:28:21.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:21 smithi063 ceph-mon[36717]: pgmap v77: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 73 KiB/s wr, 15 op/s 2024-03-04T12:28:21.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:21 smithi060 ceph-mon[32745]: pgmap v77: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 73 KiB/s wr, 15 op/s 2024-03-04T12:28:23.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:23 smithi086 ceph-mon[36851]: pgmap v78: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 73 KiB/s wr, 10 op/s 2024-03-04T12:28:23.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:23 smithi063 ceph-mon[36717]: pgmap v78: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 73 KiB/s wr, 10 op/s 2024-03-04T12:28:23.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:23 smithi060 ceph-mon[32745]: pgmap v78: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 73 KiB/s wr, 10 op/s 2024-03-04T12:28:25.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:25 smithi086 ceph-mon[36851]: pgmap v79: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 73 KiB/s wr, 10 op/s 2024-03-04T12:28:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:25 smithi063 ceph-mon[36717]: pgmap v79: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 73 KiB/s wr, 10 op/s 2024-03-04T12:28:25.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:25 smithi060 ceph-mon[32745]: pgmap v79: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 73 KiB/s wr, 10 op/s 2024-03-04T12:28:27.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:27 smithi086 ceph-mon[36851]: pgmap v80: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 92 KiB/s wr, 15 op/s 2024-03-04T12:28:27.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:27 smithi063 ceph-mon[36717]: pgmap v80: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 92 KiB/s wr, 15 op/s 2024-03-04T12:28:27.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:27 smithi060 ceph-mon[32745]: pgmap v80: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 92 KiB/s wr, 15 op/s 2024-03-04T12:28:29.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:29 smithi086 ceph-mon[36851]: pgmap v81: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 19 KiB/s wr, 9 op/s 2024-03-04T12:28:29.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:29 smithi063 ceph-mon[36717]: pgmap v81: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 19 KiB/s wr, 9 op/s 2024-03-04T12:28:29.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:29 smithi060 ceph-mon[32745]: pgmap v81: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 19 KiB/s wr, 9 op/s 2024-03-04T12:28:31.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:31 smithi086 ceph-mon[36851]: pgmap v82: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T12:28:31.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:31 smithi063 ceph-mon[36717]: pgmap v82: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T12:28:31.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:31 smithi060 ceph-mon[32745]: pgmap v82: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 20 KiB/s wr, 13 op/s 2024-03-04T12:28:33.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:33 smithi086 ceph-mon[36851]: pgmap v83: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 9 op/s 2024-03-04T12:28:33.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:33 smithi063 ceph-mon[36717]: pgmap v83: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 9 op/s 2024-03-04T12:28:33.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:33 smithi060 ceph-mon[32745]: pgmap v83: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 9 op/s 2024-03-04T12:28:35.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:35 smithi086 ceph-mon[36851]: pgmap v84: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 9 op/s 2024-03-04T12:28:35.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:35 smithi063 ceph-mon[36717]: pgmap v84: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 9 op/s 2024-03-04T12:28:35.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:35 smithi060 ceph-mon[32745]: pgmap v84: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 9 op/s 2024-03-04T12:28:37.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:37 smithi086 ceph-mon[36851]: pgmap v85: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-04T12:28:37.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:37 smithi063 ceph-mon[36717]: pgmap v85: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-04T12:28:37.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:37 smithi060 ceph-mon[32745]: pgmap v85: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 20 KiB/s wr, 14 op/s 2024-03-04T12:28:39.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:39 smithi063 ceph-mon[36717]: pgmap v86: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-04T12:28:39.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:39 smithi060 ceph-mon[32745]: pgmap v86: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-04T12:28:39.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:39 smithi086 ceph-mon[36851]: pgmap v86: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-03-04T12:28:41.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:41 smithi060 ceph-mon[32745]: pgmap v87: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-03-04T12:28:41.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:41 smithi086 ceph-mon[36851]: pgmap v87: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-03-04T12:28:41.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:41 smithi063 ceph-mon[36717]: pgmap v87: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-03-04T12:28:43.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:43 smithi060 ceph-mon[32745]: pgmap v88: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-03-04T12:28:43.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:43 smithi086 ceph-mon[36851]: pgmap v88: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-03-04T12:28:43.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:43 smithi063 ceph-mon[36717]: pgmap v88: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-03-04T12:28:45.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:45 smithi060 ceph-mon[32745]: pgmap v89: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-03-04T12:28:45.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:45 smithi086 ceph-mon[36851]: pgmap v89: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-03-04T12:28:45.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:45 smithi063 ceph-mon[36717]: pgmap v89: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 9 op/s 2024-03-04T12:28:47.402 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:47 smithi086 ceph-mon[36851]: pgmap v90: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-03-04T12:28:47.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:47 smithi060 ceph-mon[32745]: pgmap v90: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-03-04T12:28:47.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:47 smithi063 ceph-mon[36717]: pgmap v90: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-03-04T12:28:49.405 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:49 smithi063 ceph-mon[36717]: pgmap v91: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-04T12:28:49.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:49 smithi060 ceph-mon[32745]: pgmap v91: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-04T12:28:49.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:49 smithi086 ceph-mon[36851]: pgmap v91: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-04T12:28:51.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:51 smithi060 ceph-mon[32745]: pgmap v92: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-03-04T12:28:51.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:51 smithi086 ceph-mon[36851]: pgmap v92: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-03-04T12:28:51.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:51 smithi063 ceph-mon[36717]: pgmap v92: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2024-03-04T12:28:53.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:53 smithi060 ceph-mon[32745]: pgmap v93: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-04T12:28:53.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:53 smithi086 ceph-mon[36851]: pgmap v93: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-04T12:28:53.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:53 smithi063 ceph-mon[36717]: pgmap v93: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-04T12:28:55.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:55 smithi060 ceph-mon[32745]: pgmap v94: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-04T12:28:55.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:55 smithi086 ceph-mon[36851]: pgmap v94: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-04T12:28:55.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:55 smithi063 ceph-mon[36717]: pgmap v94: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-04T12:28:57.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:57 smithi060 ceph-mon[32745]: pgmap v95: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-03-04T12:28:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:57 smithi086 ceph-mon[36851]: pgmap v95: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-03-04T12:28:57.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:57 smithi063 ceph-mon[36717]: pgmap v95: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-03-04T12:28:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:28:59 smithi060 ceph-mon[32745]: pgmap v96: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-04T12:28:59.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:28:59 smithi086 ceph-mon[36851]: pgmap v96: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-04T12:28:59.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:28:59 smithi063 ceph-mon[36717]: pgmap v96: 129 pgs: 129 active+clean; 78 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-04T12:29:01.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:01 smithi060 ceph-mon[32745]: pgmap v97: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 106 KiB/s wr, 13 op/s 2024-03-04T12:29:01.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:01 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:29:01.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:01 smithi086 ceph-mon[36851]: pgmap v97: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 106 KiB/s wr, 13 op/s 2024-03-04T12:29:01.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:01 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:29:01.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:01 smithi063 ceph-mon[36717]: pgmap v97: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 106 KiB/s wr, 13 op/s 2024-03-04T12:29:01.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:01 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:29:02.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:02 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:29:02.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:02 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:29:02.430 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:02 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:29:02.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:02 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:29:02.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:02 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:29:02.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:02 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:29:02.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:02 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:29:02.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:02 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:29:02.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:02 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:29:03.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:03 smithi060 ceph-mon[32745]: pgmap v98: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 9 op/s 2024-03-04T12:29:03.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:03 smithi086 ceph-mon[36851]: pgmap v98: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 9 op/s 2024-03-04T12:29:03.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:03 smithi063 ceph-mon[36717]: pgmap v98: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 9 op/s 2024-03-04T12:29:05.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:05 smithi060 ceph-mon[32745]: pgmap v99: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 9 op/s 2024-03-04T12:29:05.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:05 smithi086 ceph-mon[36851]: pgmap v99: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 9 op/s 2024-03-04T12:29:05.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:05 smithi063 ceph-mon[36717]: pgmap v99: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 9 op/s 2024-03-04T12:29:07.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:07 smithi060 ceph-mon[32745]: pgmap v100: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 187 KiB/s wr, 29 op/s 2024-03-04T12:29:07.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:07 smithi086 ceph-mon[36851]: pgmap v100: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 187 KiB/s wr, 29 op/s 2024-03-04T12:29:07.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:07 smithi063 ceph-mon[36717]: pgmap v100: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 187 KiB/s wr, 29 op/s 2024-03-04T12:29:08.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:08 smithi086 ceph-mon[36851]: pgmap v101: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 185 KiB/s wr, 24 op/s 2024-03-04T12:29:08.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:08 smithi063 ceph-mon[36717]: pgmap v101: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 185 KiB/s wr, 24 op/s 2024-03-04T12:29:08.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:08 smithi060 ceph-mon[32745]: pgmap v101: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 185 KiB/s wr, 24 op/s 2024-03-04T12:29:10.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:10 smithi086 ceph-mon[36851]: pgmap v102: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 191 KiB/s wr, 30 op/s 2024-03-04T12:29:10.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:10 smithi063 ceph-mon[36717]: pgmap v102: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 191 KiB/s wr, 30 op/s 2024-03-04T12:29:10.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:10 smithi060 ceph-mon[32745]: pgmap v102: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 191 KiB/s wr, 30 op/s 2024-03-04T12:29:12.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:12 smithi086 ceph-mon[36851]: pgmap v103: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 86 KiB/s wr, 25 op/s 2024-03-04T12:29:12.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:12 smithi063 ceph-mon[36717]: pgmap v103: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 86 KiB/s wr, 25 op/s 2024-03-04T12:29:12.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:12 smithi060 ceph-mon[32745]: pgmap v103: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 86 KiB/s wr, 25 op/s 2024-03-04T12:29:14.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:14 smithi086 ceph-mon[36851]: pgmap v104: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 86 KiB/s wr, 25 op/s 2024-03-04T12:29:14.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:14 smithi063 ceph-mon[36717]: pgmap v104: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 86 KiB/s wr, 25 op/s 2024-03-04T12:29:14.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:14 smithi060 ceph-mon[32745]: pgmap v104: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 86 KiB/s wr, 25 op/s 2024-03-04T12:29:16.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:16 smithi086 ceph-mon[36851]: pgmap v105: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 87 KiB/s wr, 29 op/s 2024-03-04T12:29:16.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:16 smithi063 ceph-mon[36717]: pgmap v105: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 87 KiB/s wr, 29 op/s 2024-03-04T12:29:16.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:16 smithi060 ceph-mon[32745]: pgmap v105: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 87 KiB/s wr, 29 op/s 2024-03-04T12:29:18.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:18 smithi086 ceph-mon[36851]: pgmap v106: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.4 KiB/s wr, 10 op/s 2024-03-04T12:29:18.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:18 smithi063 ceph-mon[36717]: pgmap v106: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.4 KiB/s wr, 10 op/s 2024-03-04T12:29:18.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:18 smithi060 ceph-mon[32745]: pgmap v106: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.4 KiB/s wr, 10 op/s 2024-03-04T12:29:20.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:20 smithi086 ceph-mon[36851]: pgmap v107: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.0 KiB/s wr, 14 op/s 2024-03-04T12:29:20.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:20 smithi063 ceph-mon[36717]: pgmap v107: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.0 KiB/s wr, 14 op/s 2024-03-04T12:29:20.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:20 smithi060 ceph-mon[32745]: pgmap v107: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.0 KiB/s wr, 14 op/s 2024-03-04T12:29:22.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:22 smithi086 ceph-mon[36851]: pgmap v108: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-03-04T12:29:22.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:22 smithi063 ceph-mon[36717]: pgmap v108: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-03-04T12:29:22.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:22 smithi060 ceph-mon[32745]: pgmap v108: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-03-04T12:29:24.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:24 smithi086 ceph-mon[36851]: pgmap v109: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-03-04T12:29:24.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:24 smithi063 ceph-mon[36717]: pgmap v109: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-03-04T12:29:24.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:24 smithi060 ceph-mon[32745]: pgmap v109: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-03-04T12:29:26.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:26 smithi086 ceph-mon[36851]: pgmap v110: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 3.5 KiB/s wr, 13 op/s 2024-03-04T12:29:26.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:26 smithi063 ceph-mon[36717]: pgmap v110: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 3.5 KiB/s wr, 13 op/s 2024-03-04T12:29:26.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:26 smithi060 ceph-mon[32745]: pgmap v110: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 3.5 KiB/s wr, 13 op/s 2024-03-04T12:29:28.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:28 smithi086 ceph-mon[36851]: pgmap v111: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-03-04T12:29:28.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:28 smithi063 ceph-mon[36717]: pgmap v111: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-03-04T12:29:28.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:28 smithi060 ceph-mon[32745]: pgmap v111: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2024-03-04T12:29:30.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:30 smithi086 ceph-mon[36851]: pgmap v112: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-03-04T12:29:30.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:30 smithi063 ceph-mon[36717]: pgmap v112: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-03-04T12:29:30.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:30 smithi060 ceph-mon[32745]: pgmap v112: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 2.8 KiB/s wr, 13 op/s 2024-03-04T12:29:32.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:32 smithi086 ceph-mon[36851]: pgmap v113: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-04T12:29:32.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:32 smithi063 ceph-mon[36717]: pgmap v113: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-04T12:29:32.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:32 smithi060 ceph-mon[32745]: pgmap v113: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-04T12:29:34.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:34 smithi086 ceph-mon[36851]: pgmap v114: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-04T12:29:34.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:34 smithi063 ceph-mon[36717]: pgmap v114: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-04T12:29:34.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:34 smithi060 ceph-mon[32745]: pgmap v114: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-04T12:29:36.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:36 smithi086 ceph-mon[36851]: pgmap v115: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-04T12:29:36.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:36 smithi063 ceph-mon[36717]: pgmap v115: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-04T12:29:36.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:36 smithi060 ceph-mon[32745]: pgmap v115: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-04T12:29:38.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:38 smithi086 ceph-mon[36851]: pgmap v116: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:38.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:38 smithi063 ceph-mon[36717]: pgmap v116: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:38.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:38 smithi060 ceph-mon[32745]: pgmap v116: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:40.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:40 smithi086 ceph-mon[36851]: pgmap v117: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:29:40.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:40 smithi063 ceph-mon[36717]: pgmap v117: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:29:40.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:40 smithi060 ceph-mon[32745]: pgmap v117: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:29:42.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:42 smithi086 ceph-mon[36851]: pgmap v118: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:42.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:42 smithi063 ceph-mon[36717]: pgmap v118: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:42.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:42 smithi060 ceph-mon[32745]: pgmap v118: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:44.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:44 smithi086 ceph-mon[36851]: pgmap v119: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:44.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:44 smithi063 ceph-mon[36717]: pgmap v119: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:44.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:44 smithi060 ceph-mon[32745]: pgmap v119: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:46.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:46 smithi086 ceph-mon[36851]: pgmap v120: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:29:46.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:46 smithi063 ceph-mon[36717]: pgmap v120: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:29:46.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:46 smithi060 ceph-mon[32745]: pgmap v120: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:29:48.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:48 smithi086 ceph-mon[36851]: pgmap v121: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:48.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:48 smithi063 ceph-mon[36717]: pgmap v121: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:48.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:48 smithi060 ceph-mon[32745]: pgmap v121: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:50.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:50 smithi086 ceph-mon[36851]: pgmap v122: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:29:50.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:50 smithi063 ceph-mon[36717]: pgmap v122: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:29:50.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:50 smithi060 ceph-mon[32745]: pgmap v122: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:29:51.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:51 smithi086 ceph-mon[36851]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:29:51.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:51 smithi063 ceph-mon[36717]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:29:51.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:51 smithi060 ceph-mon[32745]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:29:52.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:52 smithi086 ceph-mon[36851]: pgmap v123: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:52.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:52 smithi063 ceph-mon[36717]: pgmap v123: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:52.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:52 smithi060 ceph-mon[32745]: pgmap v123: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:54.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:54 smithi086 ceph-mon[36851]: pgmap v124: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:54.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:54 smithi063 ceph-mon[36717]: pgmap v124: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:54.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:54 smithi060 ceph-mon[32745]: pgmap v124: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:56.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:56 smithi086 ceph-mon[36851]: pgmap v125: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:29:56.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:56 smithi063 ceph-mon[36717]: pgmap v125: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:29:56.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:56 smithi060 ceph-mon[32745]: pgmap v125: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:29:58.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:29:58 smithi063 ceph-mon[36717]: pgmap v126: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:58.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:29:58 smithi060 ceph-mon[32745]: pgmap v126: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:29:58.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:29:58 smithi086 ceph-mon[36851]: pgmap v126: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:00.352 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a[32741]: 2024-03-04T12:29:59.999+0000 7f4240087700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-04T12:30:00.353 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a[32741]: 2024-03-04T12:29:59.999+0000 7f4240087700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-04T12:30:00.353 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a[32741]: 2024-03-04T12:29:59.999+0000 7f4240087700 -1 log_channel(cluster) log [ERR] : mds.h(mds.2): Metadata damage detected 2024-03-04T12:30:00.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:00 smithi060 ceph-mon[32745]: pgmap v127: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:00.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:00 smithi060 ceph-mon[32745]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-04T12:30:00.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:00 smithi060 ceph-mon[32745]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-04T12:30:00.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:00 smithi060 ceph-mon[32745]: mds.h(mds.2): Metadata damage detected 2024-03-04T12:30:00.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:00 smithi086 ceph-mon[36851]: pgmap v127: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:00.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:00 smithi086 ceph-mon[36851]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-04T12:30:00.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:00 smithi086 ceph-mon[36851]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-04T12:30:00.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:00 smithi086 ceph-mon[36851]: mds.h(mds.2): Metadata damage detected 2024-03-04T12:30:00.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:00 smithi063 ceph-mon[36717]: pgmap v127: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:00.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:00 smithi063 ceph-mon[36717]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-04T12:30:00.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:00 smithi063 ceph-mon[36717]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-04T12:30:00.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:00 smithi063 ceph-mon[36717]: mds.h(mds.2): Metadata damage detected 2024-03-04T12:30:02.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:02 smithi060 ceph-mon[32745]: pgmap v128: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:02.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:02 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:30:02.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:02 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:30:02.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:02 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:30:02.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:02 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:30:02.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:02 smithi086 ceph-mon[36851]: pgmap v128: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:02.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:02 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:30:02.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:02 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:30:02.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:02 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:30:02.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:02 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:30:02.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:02 smithi063 ceph-mon[36717]: pgmap v128: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:02.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:02 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:30:02.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:02 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:30:02.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:02 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:30:02.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:02 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:30:04.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:04 smithi060 ceph-mon[32745]: pgmap v129: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:04.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:04 smithi086 ceph-mon[36851]: pgmap v129: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:04.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:04 smithi063 ceph-mon[36717]: pgmap v129: 129 pgs: 129 active+clean; 80 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:06.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:06 smithi060 ceph-mon[32745]: pgmap v130: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:30:06.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:06 smithi086 ceph-mon[36851]: pgmap v130: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:30:06.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:06 smithi063 ceph-mon[36717]: pgmap v130: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:30:08.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:08 smithi060 ceph-mon[32745]: pgmap v131: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:30:08.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:08 smithi060 ceph-mon[32745]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:30:08.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:08 smithi086 ceph-mon[36851]: pgmap v131: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:30:08.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:08 smithi086 ceph-mon[36851]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:30:08.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:08 smithi063 ceph-mon[36717]: pgmap v131: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:30:08.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:08 smithi063 ceph-mon[36717]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:30:10.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:10 smithi060 ceph-mon[32745]: pgmap v132: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:30:10.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:10 smithi086 ceph-mon[36851]: pgmap v132: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:30:10.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:10 smithi063 ceph-mon[36717]: pgmap v132: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:30:12.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:12 smithi060 ceph-mon[32745]: pgmap v133: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:30:12.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:12 smithi086 ceph-mon[36851]: pgmap v133: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:30:12.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:12 smithi063 ceph-mon[36717]: pgmap v133: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:30:14.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:14 smithi060 ceph-mon[32745]: pgmap v134: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:30:14.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:14 smithi086 ceph-mon[36851]: pgmap v134: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:30:14.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:14 smithi063 ceph-mon[36717]: pgmap v134: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:30:16.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:16 smithi060 ceph-mon[32745]: pgmap v135: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:30:16.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:16 smithi086 ceph-mon[36851]: pgmap v135: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:30:16.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:16 smithi063 ceph-mon[36717]: pgmap v135: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:30:18.652 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:18 smithi086 ceph-mon[36851]: pgmap v136: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-04T12:30:18.652 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:18 smithi086 ceph-mon[36851]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:30:18.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:18 smithi060 ceph-mon[32745]: pgmap v136: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-04T12:30:18.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:18 smithi060 ceph-mon[32745]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:30:18.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:18 smithi063 ceph-mon[36717]: pgmap v136: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-04T12:30:18.853 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:18 smithi063 ceph-mon[36717]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:30:20.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:20 smithi063 ceph-mon[36717]: pgmap v137: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-04T12:30:20.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:20 smithi060 ceph-mon[32745]: pgmap v137: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-04T12:30:20.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:20 smithi086 ceph-mon[36851]: pgmap v137: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-04T12:30:22.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:22 smithi060 ceph-mon[32745]: pgmap v138: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:22.849 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:22 smithi063 ceph-mon[36717]: pgmap v138: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:22.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:22 smithi086 ceph-mon[36851]: pgmap v138: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:24.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:24 smithi060 ceph-mon[32745]: pgmap v139: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:24.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:24 smithi086 ceph-mon[36851]: pgmap v139: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:24.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:24 smithi063 ceph-mon[36717]: pgmap v139: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:26.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:26 smithi060 ceph-mon[32745]: pgmap v140: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:26.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:26 smithi086 ceph-mon[36851]: pgmap v140: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:26.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:26 smithi063 ceph-mon[36717]: pgmap v140: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:28.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:28 smithi060 ceph-mon[32745]: pgmap v141: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:28.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:28 smithi086 ceph-mon[36851]: pgmap v141: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:28.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:28 smithi063 ceph-mon[36717]: pgmap v141: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:30.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:30 smithi060 ceph-mon[32745]: pgmap v142: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:30.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:30 smithi086 ceph-mon[36851]: pgmap v142: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:30.855 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:30 smithi063 ceph-mon[36717]: pgmap v142: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:32.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:32 smithi063 ceph-mon[36717]: pgmap v143: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:32.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:32 smithi060 ceph-mon[32745]: pgmap v143: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:32.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:32 smithi086 ceph-mon[36851]: pgmap v143: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:34.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:34 smithi060 ceph-mon[32745]: pgmap v144: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:34.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:34 smithi086 ceph-mon[36851]: pgmap v144: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:34.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:34 smithi063 ceph-mon[36717]: pgmap v144: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:36.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:36 smithi060 ceph-mon[32745]: pgmap v145: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:36.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:36 smithi086 ceph-mon[36851]: pgmap v145: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:36.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:36 smithi063 ceph-mon[36717]: pgmap v145: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:38.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:38 smithi060 ceph-mon[32745]: pgmap v146: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:38.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:38 smithi086 ceph-mon[36851]: pgmap v146: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:38.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:38 smithi063 ceph-mon[36717]: pgmap v146: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:40.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:40 smithi060 ceph-mon[32745]: pgmap v147: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:40.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:40 smithi086 ceph-mon[36851]: pgmap v147: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:40.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:40 smithi063 ceph-mon[36717]: pgmap v147: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:41.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:41 smithi086 ceph-mon[36851]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:30:41.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:41 smithi063 ceph-mon[36717]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:30:41.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:41 smithi060 ceph-mon[32745]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-04T12:30:42.831 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:42 smithi060 ceph-mon[32745]: pgmap v148: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:42.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:42 smithi086 ceph-mon[36851]: pgmap v148: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:42.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:42 smithi063 ceph-mon[36717]: pgmap v148: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:44.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:44 smithi086 ceph-mon[36851]: pgmap v149: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:44.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:44 smithi063 ceph-mon[36717]: pgmap v149: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:44.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:44 smithi060 ceph-mon[32745]: pgmap v149: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:46.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:46 smithi086 ceph-mon[36851]: pgmap v150: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:46.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:46 smithi063 ceph-mon[36717]: pgmap v150: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:46.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:46 smithi060 ceph-mon[32745]: pgmap v150: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:48.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:48 smithi086 ceph-mon[36851]: pgmap v151: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:48.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:48 smithi063 ceph-mon[36717]: pgmap v151: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:48.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:48 smithi060 ceph-mon[32745]: pgmap v151: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:50.840 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:50 smithi086 ceph-mon[36851]: pgmap v152: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:50.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:50 smithi063 ceph-mon[36717]: pgmap v152: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:50.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:50 smithi060 ceph-mon[32745]: pgmap v152: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:52.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:52 smithi086 ceph-mon[36851]: pgmap v153: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:52.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:52 smithi063 ceph-mon[36717]: pgmap v153: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:52.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:52 smithi060 ceph-mon[32745]: pgmap v153: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:54.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:54 smithi086 ceph-mon[36851]: pgmap v154: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:54.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:54 smithi063 ceph-mon[36717]: pgmap v154: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:54.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:54 smithi060 ceph-mon[32745]: pgmap v154: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:56.851 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:56 smithi086 ceph-mon[36851]: pgmap v155: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:56.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:56 smithi063 ceph-mon[36717]: pgmap v155: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:56.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:56 smithi060 ceph-mon[32745]: pgmap v155: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:30:58.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:30:58 smithi086 ceph-mon[36851]: pgmap v156: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:58.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:30:58 smithi063 ceph-mon[36717]: pgmap v156: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:30:58.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:30:58 smithi060 ceph-mon[32745]: pgmap v156: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:00.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:00 smithi086 ceph-mon[36851]: pgmap v157: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:00.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:00 smithi063 ceph-mon[36717]: pgmap v157: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:00.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:00 smithi060 ceph-mon[32745]: pgmap v157: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:02.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:02 smithi086 ceph-mon[36851]: pgmap v158: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:02.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:02 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:31:02.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:02 smithi063 ceph-mon[36717]: pgmap v158: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:02.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:02 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:31:02.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:02 smithi060 ceph-mon[32745]: pgmap v158: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:02.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:02 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:31:03.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:03 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:31:03.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:03 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:31:03.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:03 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:31:03.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:03 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:31:03.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:03 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:31:03.852 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:03 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:31:03.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:03 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:31:03.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:03 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:31:03.929 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:03 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:31:05.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:04 smithi060 ceph-mon[32745]: pgmap v159: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:05.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:04 smithi086 ceph-mon[36851]: pgmap v159: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:05.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:04 smithi063 ceph-mon[36717]: pgmap v159: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:07.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:06 smithi060 ceph-mon[32745]: pgmap v160: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:31:07.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:06 smithi086 ceph-mon[36851]: pgmap v160: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:31:07.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:06 smithi063 ceph-mon[36717]: pgmap v160: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:31:09.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:08 smithi060 ceph-mon[32745]: pgmap v161: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:31:09.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:08 smithi086 ceph-mon[36851]: pgmap v161: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:31:09.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:08 smithi063 ceph-mon[36717]: pgmap v161: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:31:11.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:10 smithi086 ceph-mon[36851]: pgmap v162: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:31:11.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:10 smithi063 ceph-mon[36717]: pgmap v162: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:31:11.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:10 smithi060 ceph-mon[32745]: pgmap v162: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:31:13.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:12 smithi086 ceph-mon[36851]: pgmap v163: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:31:13.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:12 smithi063 ceph-mon[36717]: pgmap v163: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:31:13.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:12 smithi060 ceph-mon[32745]: pgmap v163: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:31:15.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:14 smithi086 ceph-mon[36851]: pgmap v164: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:31:15.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:14 smithi063 ceph-mon[36717]: pgmap v164: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:31:15.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:14 smithi060 ceph-mon[32745]: pgmap v164: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-04T12:31:17.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:16 smithi086 ceph-mon[36851]: pgmap v165: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:31:17.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:16 smithi063 ceph-mon[36717]: pgmap v165: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:31:17.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:16 smithi060 ceph-mon[32745]: pgmap v165: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-04T12:31:19.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:18 smithi086 ceph-mon[36851]: pgmap v166: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-04T12:31:19.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:18 smithi063 ceph-mon[36717]: pgmap v166: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-04T12:31:19.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:18 smithi060 ceph-mon[32745]: pgmap v166: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-04T12:31:21.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:20 smithi086 ceph-mon[36851]: pgmap v167: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-04T12:31:21.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:20 smithi063 ceph-mon[36717]: pgmap v167: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-04T12:31:21.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:20 smithi060 ceph-mon[32745]: pgmap v167: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-04T12:31:23.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:22 smithi086 ceph-mon[36851]: pgmap v168: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:23.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:22 smithi063 ceph-mon[36717]: pgmap v168: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:23.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:22 smithi060 ceph-mon[32745]: pgmap v168: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:25.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:24 smithi086 ceph-mon[36851]: pgmap v169: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:24 smithi063 ceph-mon[36717]: pgmap v169: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:25.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:24 smithi060 ceph-mon[32745]: pgmap v169: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:27.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:26 smithi086 ceph-mon[36851]: pgmap v170: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:27.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:26 smithi063 ceph-mon[36717]: pgmap v170: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:27.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:26 smithi060 ceph-mon[32745]: pgmap v170: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:29.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:29 smithi086 ceph-mon[36851]: pgmap v171: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:29.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:29 smithi063 ceph-mon[36717]: pgmap v171: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:29.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:29 smithi060 ceph-mon[32745]: pgmap v171: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:31.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:31 smithi086 ceph-mon[36851]: pgmap v172: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:31.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:31 smithi063 ceph-mon[36717]: pgmap v172: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:31.432 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:31 smithi060 ceph-mon[32745]: pgmap v172: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:33.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:33 smithi086 ceph-mon[36851]: pgmap v173: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:33.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:33 smithi063 ceph-mon[36717]: pgmap v173: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:33.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:33 smithi060 ceph-mon[32745]: pgmap v173: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:35.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:35 smithi086 ceph-mon[36851]: pgmap v174: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:35.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:35 smithi063 ceph-mon[36717]: pgmap v174: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:35.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:35 smithi060 ceph-mon[32745]: pgmap v174: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:37.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:37 smithi086 ceph-mon[36851]: pgmap v175: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:37.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:37 smithi063 ceph-mon[36717]: pgmap v175: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:37.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:37 smithi060 ceph-mon[32745]: pgmap v175: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:39.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:39 smithi086 ceph-mon[36851]: pgmap v176: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:39.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:39 smithi063 ceph-mon[36717]: pgmap v176: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:39.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:39 smithi060 ceph-mon[32745]: pgmap v176: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:41.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:41 smithi086 ceph-mon[36851]: pgmap v177: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:41.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:41 smithi063 ceph-mon[36717]: pgmap v177: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:41.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:41 smithi060 ceph-mon[32745]: pgmap v177: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:43.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:43 smithi086 ceph-mon[36851]: pgmap v178: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:43.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:43 smithi063 ceph-mon[36717]: pgmap v178: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:43.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:43 smithi060 ceph-mon[32745]: pgmap v178: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:45.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:45 smithi086 ceph-mon[36851]: pgmap v179: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:45.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:45 smithi063 ceph-mon[36717]: pgmap v179: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:45.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:45 smithi060 ceph-mon[32745]: pgmap v179: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:47.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:47 smithi086 ceph-mon[36851]: pgmap v180: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:47.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:47 smithi063 ceph-mon[36717]: pgmap v180: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:47.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:47 smithi060 ceph-mon[32745]: pgmap v180: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:49.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:49 smithi086 ceph-mon[36851]: pgmap v181: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:49.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:49 smithi063 ceph-mon[36717]: pgmap v181: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:49.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:49 smithi060 ceph-mon[32745]: pgmap v181: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:51.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:51 smithi086 ceph-mon[36851]: pgmap v182: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:51.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:51 smithi063 ceph-mon[36717]: pgmap v182: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:51.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:51 smithi060 ceph-mon[32745]: pgmap v182: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:53.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:53 smithi086 ceph-mon[36851]: pgmap v183: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:53.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:53 smithi063 ceph-mon[36717]: pgmap v183: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:53.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:53 smithi060 ceph-mon[32745]: pgmap v183: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:55.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:55 smithi086 ceph-mon[36851]: pgmap v184: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:55.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:55 smithi063 ceph-mon[36717]: pgmap v184: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:55.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:55 smithi060 ceph-mon[32745]: pgmap v184: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:57.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:57 smithi086 ceph-mon[36851]: pgmap v185: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:57.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:57 smithi063 ceph-mon[36717]: pgmap v185: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:57.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:57 smithi060 ceph-mon[32745]: pgmap v185: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:31:59.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:31:59 smithi086 ceph-mon[36851]: pgmap v186: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:59.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:31:59 smithi063 ceph-mon[36717]: pgmap v186: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:31:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:31:59 smithi060 ceph-mon[32745]: pgmap v186: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:01.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:01 smithi086 ceph-mon[36851]: pgmap v187: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:01.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:01 smithi063 ceph-mon[36717]: pgmap v187: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:01.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:01 smithi060 ceph-mon[32745]: pgmap v187: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:03.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:03 smithi086 ceph-mon[36851]: pgmap v188: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:03.350 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:03 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:32:03.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:03 smithi063 ceph-mon[36717]: pgmap v188: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:03.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:03 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:32:03.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:03 smithi060 ceph-mon[32745]: pgmap v188: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:03.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:03 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:32:04.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:04 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:32:04.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:04 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:32:04.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:04 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:32:04.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:04 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:32:04.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:04 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:32:04.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:04 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:32:04.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:04 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:32:04.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:04 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:32:04.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:04 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:32:05.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:05 smithi086 ceph-mon[36851]: pgmap v189: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:05.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:05 smithi063 ceph-mon[36717]: pgmap v189: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:05.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:05 smithi060 ceph-mon[32745]: pgmap v189: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:07.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:07 smithi086 ceph-mon[36851]: pgmap v190: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-04T12:32:07.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:07 smithi063 ceph-mon[36717]: pgmap v190: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-04T12:32:07.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:07 smithi060 ceph-mon[32745]: pgmap v190: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 45 KiB/s wr, 29 op/s 2024-03-04T12:32:08.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:08 smithi086 ceph-mon[36851]: osdmap e96: 12 total, 12 up, 12 in 2024-03-04T12:32:08.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:08 smithi063 ceph-mon[36717]: osdmap e96: 12 total, 12 up, 12 in 2024-03-04T12:32:08.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:08 smithi060 ceph-mon[32745]: osdmap e96: 12 total, 12 up, 12 in 2024-03-04T12:32:09.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:09 smithi086 ceph-mon[36851]: pgmap v192: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-04T12:32:09.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:09 smithi063 ceph-mon[36717]: pgmap v192: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-04T12:32:09.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:09 smithi060 ceph-mon[32745]: pgmap v192: 129 pgs: 129 active+clean; 80 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-04T12:32:11.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:11 smithi086 ceph-mon[36851]: pgmap v193: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-04T12:32:11.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:11 smithi063 ceph-mon[36717]: pgmap v193: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-04T12:32:11.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:11 smithi060 ceph-mon[32745]: pgmap v193: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-04T12:32:13.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:13 smithi086 ceph-mon[36851]: pgmap v194: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-04T12:32:13.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:13 smithi063 ceph-mon[36717]: pgmap v194: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-04T12:32:13.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:13 smithi060 ceph-mon[32745]: pgmap v194: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-04T12:32:15.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:15 smithi086 ceph-mon[36851]: pgmap v195: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-04T12:32:15.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:15 smithi086 ceph-mon[36851]: osdmap e97: 12 total, 12 up, 12 in 2024-03-04T12:32:15.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:15 smithi063 ceph-mon[36717]: pgmap v195: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-04T12:32:15.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:15 smithi063 ceph-mon[36717]: osdmap e97: 12 total, 12 up, 12 in 2024-03-04T12:32:15.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:15 smithi060 ceph-mon[32745]: pgmap v195: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 54 KiB/s wr, 30 op/s 2024-03-04T12:32:15.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:15 smithi060 ceph-mon[32745]: osdmap e97: 12 total, 12 up, 12 in 2024-03-04T12:32:17.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:17 smithi086 ceph-mon[36851]: pgmap v197: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2024-03-04T12:32:17.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:17 smithi063 ceph-mon[36717]: pgmap v197: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2024-03-04T12:32:17.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:17 smithi060 ceph-mon[32745]: pgmap v197: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2024-03-04T12:32:19.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:19 smithi086 ceph-mon[36851]: pgmap v198: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:32:19.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:19 smithi063 ceph-mon[36717]: pgmap v198: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:32:19.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:19 smithi060 ceph-mon[32745]: pgmap v198: 129 pgs: 129 active+clean; 80 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:32:21.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:21 smithi063 ceph-mon[36717]: pgmap v199: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:32:21.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:21 smithi060 ceph-mon[32745]: pgmap v199: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:32:21.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:21 smithi086 ceph-mon[36851]: pgmap v199: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:32:23.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:23 smithi060 ceph-mon[32745]: pgmap v200: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:32:23.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:23 smithi086 ceph-mon[36851]: pgmap v200: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:32:23.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:23 smithi063 ceph-mon[36717]: pgmap v200: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:32:25.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:25 smithi060 ceph-mon[32745]: pgmap v201: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:32:25.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:25 smithi086 ceph-mon[36851]: pgmap v201: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:32:25.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:25 smithi063 ceph-mon[36717]: pgmap v201: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:32:27.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:27 smithi060 ceph-mon[32745]: pgmap v202: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-04T12:32:27.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:27 smithi086 ceph-mon[36851]: pgmap v202: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-04T12:32:27.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:27 smithi063 ceph-mon[36717]: pgmap v202: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-03-04T12:32:29.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:29 smithi060 ceph-mon[32745]: pgmap v203: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:32:29.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:29 smithi086 ceph-mon[36851]: pgmap v203: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:32:29.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:29 smithi063 ceph-mon[36717]: pgmap v203: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:32:31.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:31 smithi060 ceph-mon[32745]: pgmap v204: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:31.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:31 smithi086 ceph-mon[36851]: pgmap v204: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:31.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:31 smithi063 ceph-mon[36717]: pgmap v204: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:33.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:33 smithi060 ceph-mon[32745]: pgmap v205: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:32:33.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:33 smithi086 ceph-mon[36851]: pgmap v205: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:32:33.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:33 smithi063 ceph-mon[36717]: pgmap v205: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:32:35.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:35 smithi060 ceph-mon[32745]: pgmap v206: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:32:35.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:35 smithi086 ceph-mon[36851]: pgmap v206: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:32:35.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:35 smithi063 ceph-mon[36717]: pgmap v206: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:32:37.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:37 smithi060 ceph-mon[32745]: pgmap v207: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:37.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:37 smithi086 ceph-mon[36851]: pgmap v207: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:37.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:37 smithi063 ceph-mon[36717]: pgmap v207: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:39.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:39 smithi060 ceph-mon[32745]: pgmap v208: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:39.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:39 smithi086 ceph-mon[36851]: pgmap v208: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:39.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:39 smithi063 ceph-mon[36717]: pgmap v208: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:41.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:41 smithi060 ceph-mon[32745]: pgmap v209: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:41.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:41 smithi086 ceph-mon[36851]: pgmap v209: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:41.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:41 smithi063 ceph-mon[36717]: pgmap v209: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:43.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:43 smithi060 ceph-mon[32745]: pgmap v210: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:43.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:43 smithi086 ceph-mon[36851]: pgmap v210: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:43.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:43 smithi063 ceph-mon[36717]: pgmap v210: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:45.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:45 smithi060 ceph-mon[32745]: pgmap v211: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:45.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:45 smithi086 ceph-mon[36851]: pgmap v211: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:45.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:45 smithi063 ceph-mon[36717]: pgmap v211: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:47.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:47 smithi060 ceph-mon[32745]: pgmap v212: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:47.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:47 smithi086 ceph-mon[36851]: pgmap v212: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:47.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:47 smithi063 ceph-mon[36717]: pgmap v212: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:49.401 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:49 smithi086 ceph-mon[36851]: pgmap v213: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:49.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:49 smithi060 ceph-mon[32745]: pgmap v213: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:49.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:49 smithi063 ceph-mon[36717]: pgmap v213: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:51.404 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:51 smithi063 ceph-mon[36717]: pgmap v214: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:51.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:51 smithi060 ceph-mon[32745]: pgmap v214: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:51.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:51 smithi086 ceph-mon[36851]: pgmap v214: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:53.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:53 smithi060 ceph-mon[32745]: pgmap v215: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:53.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:53 smithi086 ceph-mon[36851]: pgmap v215: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:53.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:53 smithi063 ceph-mon[36717]: pgmap v215: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:55.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:55 smithi060 ceph-mon[32745]: pgmap v216: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:55.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:55 smithi086 ceph-mon[36851]: pgmap v216: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:55.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:55 smithi063 ceph-mon[36717]: pgmap v216: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:57.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:57 smithi060 ceph-mon[32745]: pgmap v217: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:57 smithi086 ceph-mon[36851]: pgmap v217: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:57.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:57 smithi063 ceph-mon[36717]: pgmap v217: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:32:59.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:32:59 smithi060 ceph-mon[32745]: pgmap v218: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:59.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:32:59 smithi086 ceph-mon[36851]: pgmap v218: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:32:59.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:32:59 smithi063 ceph-mon[36717]: pgmap v218: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:01.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:01 smithi060 ceph-mon[32745]: pgmap v219: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:01.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:01 smithi086 ceph-mon[36851]: pgmap v219: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:01.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:01 smithi063 ceph-mon[36717]: pgmap v219: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:03.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:03 smithi060 ceph-mon[32745]: pgmap v220: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:03.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:03 smithi086 ceph-mon[36851]: pgmap v220: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:03.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:03 smithi063 ceph-mon[36717]: pgmap v220: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:04.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:04 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:33:04.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:04 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:33:04.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:04 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:33:05.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:05 smithi060 ceph-mon[32745]: pgmap v221: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:05.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:05 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:33:05.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:05 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:33:05.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:05 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:33:05.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:05 smithi086 ceph-mon[36851]: pgmap v221: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:05.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:05 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:33:05.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:05 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:33:05.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:05 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:33:05.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:05 smithi063 ceph-mon[36717]: pgmap v221: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:05.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:05 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:33:05.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:05 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:33:05.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:05 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:33:07.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:07 smithi060 ceph-mon[32745]: pgmap v222: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-04T12:33:07.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:07 smithi086 ceph-mon[36851]: pgmap v222: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-04T12:33:07.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:07 smithi063 ceph-mon[36717]: pgmap v222: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-04T12:33:08.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:08 smithi086 ceph-mon[36851]: osdmap e98: 12 total, 12 up, 12 in 2024-03-04T12:33:08.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:08 smithi063 ceph-mon[36717]: osdmap e98: 12 total, 12 up, 12 in 2024-03-04T12:33:08.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:08 smithi060 ceph-mon[32745]: osdmap e98: 12 total, 12 up, 12 in 2024-03-04T12:33:09.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:09 smithi086 ceph-mon[36851]: pgmap v224: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:33:09.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:09 smithi063 ceph-mon[36717]: pgmap v224: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:33:09.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:09 smithi060 ceph-mon[32745]: pgmap v224: 129 pgs: 129 active+clean; 80 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:33:11.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:11 smithi086 ceph-mon[36851]: pgmap v225: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:33:11.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:11 smithi063 ceph-mon[36717]: pgmap v225: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:33:11.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:11 smithi060 ceph-mon[32745]: pgmap v225: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:33:12.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:12 smithi086 ceph-mon[36851]: pgmap v226: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:33:12.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:12 smithi063 ceph-mon[36717]: pgmap v226: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:33:12.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:12 smithi060 ceph-mon[32745]: pgmap v226: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:33:14.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:14 smithi086 ceph-mon[36851]: pgmap v227: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:33:14.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:14 smithi063 ceph-mon[36717]: pgmap v227: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:33:14.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:14 smithi060 ceph-mon[32745]: pgmap v227: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:33:16.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:16 smithi086 ceph-mon[36851]: osdmap e99: 12 total, 12 up, 12 in 2024-03-04T12:33:16.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:16 smithi063 ceph-mon[36717]: osdmap e99: 12 total, 12 up, 12 in 2024-03-04T12:33:16.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:16 smithi060 ceph-mon[32745]: osdmap e99: 12 total, 12 up, 12 in 2024-03-04T12:33:17.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:17 smithi086 ceph-mon[36851]: pgmap v229: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2024-03-04T12:33:17.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:17 smithi063 ceph-mon[36717]: pgmap v229: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2024-03-04T12:33:17.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:17 smithi060 ceph-mon[32745]: pgmap v229: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2024-03-04T12:33:19.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:19 smithi086 ceph-mon[36851]: pgmap v230: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:33:19.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:19 smithi063 ceph-mon[36717]: pgmap v230: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:33:19.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:19 smithi060 ceph-mon[32745]: pgmap v230: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:33:21.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:21 smithi086 ceph-mon[36851]: pgmap v231: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:33:21.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:21 smithi063 ceph-mon[36717]: pgmap v231: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:33:21.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:21 smithi060 ceph-mon[32745]: pgmap v231: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:33:23.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:23 smithi086 ceph-mon[36851]: pgmap v232: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:33:23.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:23 smithi063 ceph-mon[36717]: pgmap v232: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:33:23.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:23 smithi060 ceph-mon[32745]: pgmap v232: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:33:25.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:25 smithi086 ceph-mon[36851]: pgmap v233: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:33:25.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:25 smithi063 ceph-mon[36717]: pgmap v233: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:33:25.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:25 smithi060 ceph-mon[32745]: pgmap v233: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:33:27.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:27 smithi086 ceph-mon[36851]: pgmap v234: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-04T12:33:27.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:27 smithi063 ceph-mon[36717]: pgmap v234: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-04T12:33:27.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:27 smithi060 ceph-mon[32745]: pgmap v234: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-04T12:33:29.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:29 smithi086 ceph-mon[36851]: pgmap v235: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:29.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:29 smithi063 ceph-mon[36717]: pgmap v235: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:29.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:29 smithi060 ceph-mon[32745]: pgmap v235: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:31.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:31 smithi086 ceph-mon[36851]: pgmap v236: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:31.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:31 smithi063 ceph-mon[36717]: pgmap v236: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:31.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:31 smithi060 ceph-mon[32745]: pgmap v236: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:33.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:33 smithi086 ceph-mon[36851]: pgmap v237: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:33.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:33 smithi063 ceph-mon[36717]: pgmap v237: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:33 smithi060 ceph-mon[32745]: pgmap v237: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:35.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:35 smithi086 ceph-mon[36851]: pgmap v238: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:35.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:35 smithi063 ceph-mon[36717]: pgmap v238: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:35.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:35 smithi060 ceph-mon[32745]: pgmap v238: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:37.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:37 smithi086 ceph-mon[36851]: pgmap v239: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:37.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:37 smithi063 ceph-mon[36717]: pgmap v239: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:37.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:37 smithi060 ceph-mon[32745]: pgmap v239: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:39.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:39 smithi086 ceph-mon[36851]: pgmap v240: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:39.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:39 smithi063 ceph-mon[36717]: pgmap v240: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:39 smithi060 ceph-mon[32745]: pgmap v240: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:41.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:41 smithi086 ceph-mon[36851]: pgmap v241: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:41.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:41 smithi063 ceph-mon[36717]: pgmap v241: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:41.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:41 smithi060 ceph-mon[32745]: pgmap v241: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:43.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:43 smithi086 ceph-mon[36851]: pgmap v242: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:43.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:43 smithi063 ceph-mon[36717]: pgmap v242: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:43.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:43 smithi060 ceph-mon[32745]: pgmap v242: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:45.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:45 smithi086 ceph-mon[36851]: pgmap v243: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:45.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:45 smithi063 ceph-mon[36717]: pgmap v243: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:45.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:45 smithi060 ceph-mon[32745]: pgmap v243: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:47.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:47 smithi086 ceph-mon[36851]: pgmap v244: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:47.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:47 smithi063 ceph-mon[36717]: pgmap v244: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:47.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:47 smithi060 ceph-mon[32745]: pgmap v244: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:49.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:49 smithi086 ceph-mon[36851]: pgmap v245: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:33:49.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:49 smithi063 ceph-mon[36717]: pgmap v245: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:33:49.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:49 smithi060 ceph-mon[32745]: pgmap v245: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:33:51.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:51 smithi086 ceph-mon[36851]: pgmap v246: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:51.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:51 smithi063 ceph-mon[36717]: pgmap v246: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:51.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:51 smithi060 ceph-mon[32745]: pgmap v246: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:53.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:53 smithi086 ceph-mon[36851]: pgmap v247: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:33:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:53 smithi063 ceph-mon[36717]: pgmap v247: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:33:53.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:53 smithi060 ceph-mon[32745]: pgmap v247: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:33:55.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:55 smithi086 ceph-mon[36851]: pgmap v248: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:33:55.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:55 smithi063 ceph-mon[36717]: pgmap v248: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:33:55.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:55 smithi060 ceph-mon[32745]: pgmap v248: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-03-04T12:33:57.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:57 smithi086 ceph-mon[36851]: pgmap v249: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:57.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:57 smithi063 ceph-mon[36717]: pgmap v249: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:57.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:57 smithi060 ceph-mon[32745]: pgmap v249: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:33:59.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:33:59 smithi086 ceph-mon[36851]: pgmap v250: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:59.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:33:59 smithi063 ceph-mon[36717]: pgmap v250: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:33:59.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:33:59 smithi060 ceph-mon[32745]: pgmap v250: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:01.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:01 smithi086 ceph-mon[36851]: pgmap v251: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:01.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:01 smithi063 ceph-mon[36717]: pgmap v251: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:01.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:01 smithi060 ceph-mon[32745]: pgmap v251: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:03.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:03 smithi086 ceph-mon[36851]: pgmap v252: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:03.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:03 smithi063 ceph-mon[36717]: pgmap v252: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:03.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:03 smithi060 ceph-mon[32745]: pgmap v252: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:05.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:05 smithi086 ceph-mon[36851]: pgmap v253: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:05.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:05 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:34:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:05 smithi063 ceph-mon[36717]: pgmap v253: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:05.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:05 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:34:05.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:05 smithi060 ceph-mon[32745]: pgmap v253: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:05.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:05 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:34:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:06 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:34:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:06 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:34:06.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:06 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:34:06.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:06 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:34:06.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:06 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:34:06.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:06 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:34:06.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:06 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:34:06.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:06 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:34:06.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:06 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:34:07.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:07 smithi063 ceph-mon[36717]: pgmap v254: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-04T12:34:07.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:07 smithi060 ceph-mon[32745]: pgmap v254: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-04T12:34:07.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:07 smithi086 ceph-mon[36851]: pgmap v254: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-04T12:34:08.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:08 smithi060 ceph-mon[32745]: osdmap e100: 12 total, 12 up, 12 in 2024-03-04T12:34:08.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:08 smithi086 ceph-mon[36851]: osdmap e100: 12 total, 12 up, 12 in 2024-03-04T12:34:08.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:08 smithi063 ceph-mon[36717]: osdmap e100: 12 total, 12 up, 12 in 2024-03-04T12:34:09.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:09 smithi060 ceph-mon[32745]: pgmap v256: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:34:09.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:09 smithi086 ceph-mon[36851]: pgmap v256: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:34:09.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:09 smithi063 ceph-mon[36717]: pgmap v256: 129 pgs: 129 active+clean; 80 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:34:11.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:11 smithi060 ceph-mon[32745]: pgmap v257: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:34:11.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:11 smithi086 ceph-mon[36851]: pgmap v257: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:34:11.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:11 smithi063 ceph-mon[36717]: pgmap v257: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:34:13.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:13 smithi060 ceph-mon[32745]: pgmap v258: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:34:13.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:13 smithi086 ceph-mon[36851]: pgmap v258: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:34:13.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:13 smithi063 ceph-mon[36717]: pgmap v258: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:34:15.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:15 smithi060 ceph-mon[32745]: pgmap v259: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:34:15.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:15 smithi060 ceph-mon[32745]: osdmap e101: 12 total, 12 up, 12 in 2024-03-04T12:34:15.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:15 smithi086 ceph-mon[36851]: pgmap v259: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:34:15.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:15 smithi086 ceph-mon[36851]: osdmap e101: 12 total, 12 up, 12 in 2024-03-04T12:34:15.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:15 smithi063 ceph-mon[36717]: pgmap v259: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:34:15.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:15 smithi063 ceph-mon[36717]: osdmap e101: 12 total, 12 up, 12 in 2024-03-04T12:34:17.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:17 smithi060 ceph-mon[32745]: pgmap v261: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2024-03-04T12:34:17.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:17 smithi086 ceph-mon[36851]: pgmap v261: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2024-03-04T12:34:17.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:17 smithi063 ceph-mon[36717]: pgmap v261: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2024-03-04T12:34:19.402 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:19 smithi086 ceph-mon[36851]: pgmap v262: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:34:19.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:19 smithi060 ceph-mon[32745]: pgmap v262: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:34:19.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:19 smithi063 ceph-mon[36717]: pgmap v262: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:34:21.403 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:21 smithi063 ceph-mon[36717]: pgmap v263: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:34:21.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:21 smithi060 ceph-mon[32745]: pgmap v263: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:34:21.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:21 smithi086 ceph-mon[36851]: pgmap v263: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:34:23.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:23 smithi060 ceph-mon[32745]: pgmap v264: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:34:23.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:23 smithi086 ceph-mon[36851]: pgmap v264: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:34:23.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:23 smithi063 ceph-mon[36717]: pgmap v264: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:34:25.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:25 smithi060 ceph-mon[32745]: pgmap v265: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:34:25.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:25 smithi086 ceph-mon[36851]: pgmap v265: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:34:25.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:25 smithi063 ceph-mon[36717]: pgmap v265: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:34:27.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:27 smithi060 ceph-mon[32745]: pgmap v266: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-04T12:34:27.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:27 smithi086 ceph-mon[36851]: pgmap v266: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-04T12:34:27.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:27 smithi063 ceph-mon[36717]: pgmap v266: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-04T12:34:29.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:29 smithi060 ceph-mon[32745]: pgmap v267: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:29.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:29 smithi086 ceph-mon[36851]: pgmap v267: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:29.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:29 smithi063 ceph-mon[36717]: pgmap v267: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:31.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:31 smithi060 ceph-mon[32745]: pgmap v268: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:31.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:31 smithi086 ceph-mon[36851]: pgmap v268: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:31.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:31 smithi063 ceph-mon[36717]: pgmap v268: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:33 smithi060 ceph-mon[32745]: pgmap v269: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:33.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:33 smithi086 ceph-mon[36851]: pgmap v269: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:33.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:33 smithi063 ceph-mon[36717]: pgmap v269: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:35.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:35 smithi060 ceph-mon[32745]: pgmap v270: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:35.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:35 smithi086 ceph-mon[36851]: pgmap v270: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:35.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:35 smithi063 ceph-mon[36717]: pgmap v270: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:37.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:37 smithi060 ceph-mon[32745]: pgmap v271: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:37.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:37 smithi086 ceph-mon[36851]: pgmap v271: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:37.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:37 smithi063 ceph-mon[36717]: pgmap v271: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:39 smithi060 ceph-mon[32745]: pgmap v272: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:39.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:39 smithi086 ceph-mon[36851]: pgmap v272: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:39.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:39 smithi063 ceph-mon[36717]: pgmap v272: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:41.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:41 smithi060 ceph-mon[32745]: pgmap v273: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:41.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:41 smithi086 ceph-mon[36851]: pgmap v273: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:41.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:41 smithi063 ceph-mon[36717]: pgmap v273: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:43.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:43 smithi060 ceph-mon[32745]: pgmap v274: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:43.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:43 smithi086 ceph-mon[36851]: pgmap v274: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:43.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:43 smithi063 ceph-mon[36717]: pgmap v274: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:45.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:45 smithi060 ceph-mon[32745]: pgmap v275: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:45.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:45 smithi086 ceph-mon[36851]: pgmap v275: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:45.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:45 smithi063 ceph-mon[36717]: pgmap v275: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:47.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:47 smithi060 ceph-mon[32745]: pgmap v276: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:47.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:47 smithi086 ceph-mon[36851]: pgmap v276: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:47.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:47 smithi063 ceph-mon[36717]: pgmap v276: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:49.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:49 smithi060 ceph-mon[32745]: pgmap v277: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:49.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:49 smithi086 ceph-mon[36851]: pgmap v277: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:49.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:49 smithi063 ceph-mon[36717]: pgmap v277: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:51.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:51 smithi060 ceph-mon[32745]: pgmap v278: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:51.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:51 smithi086 ceph-mon[36851]: pgmap v278: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:51.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:51 smithi063 ceph-mon[36717]: pgmap v278: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:53.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:53 smithi060 ceph-mon[32745]: pgmap v279: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:53.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:53 smithi086 ceph-mon[36851]: pgmap v279: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:53.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:53 smithi063 ceph-mon[36717]: pgmap v279: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:55.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:55 smithi086 ceph-mon[36851]: pgmap v280: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:55.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:55 smithi063 ceph-mon[36717]: pgmap v280: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:55.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:55 smithi060 ceph-mon[32745]: pgmap v280: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:57.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:57 smithi086 ceph-mon[36851]: pgmap v281: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:57.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:57 smithi063 ceph-mon[36717]: pgmap v281: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:57.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:57 smithi060 ceph-mon[32745]: pgmap v281: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:34:59.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:34:59 smithi086 ceph-mon[36851]: pgmap v282: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:59.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:34:59 smithi063 ceph-mon[36717]: pgmap v282: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:34:59.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:34:59 smithi060 ceph-mon[32745]: pgmap v282: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:01.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:01 smithi086 ceph-mon[36851]: pgmap v283: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:01.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:01 smithi063 ceph-mon[36717]: pgmap v283: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:01.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:01 smithi060 ceph-mon[32745]: pgmap v283: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:03.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:03 smithi086 ceph-mon[36851]: pgmap v284: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:03.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:03 smithi063 ceph-mon[36717]: pgmap v284: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:03.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:03 smithi060 ceph-mon[32745]: pgmap v284: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:05.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:05 smithi086 ceph-mon[36851]: pgmap v285: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:05.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:05 smithi063 ceph-mon[36717]: pgmap v285: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:05.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:05 smithi060 ceph-mon[32745]: pgmap v285: 129 pgs: 129 active+clean; 80 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:06.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:06 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:35:06.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:06 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:35:06.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:06 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:35:06.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:06 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:35:06.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:06 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:35:06.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:06 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:35:06.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:06 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:35:06.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:06 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:35:06.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:06 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:35:06.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:06 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:35:06.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:06 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:35:06.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:06 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:35:07.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:07 smithi086 ceph-mon[36851]: pgmap v286: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-04T12:35:07.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:07 smithi063 ceph-mon[36717]: pgmap v286: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-04T12:35:07.680 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:07 smithi060 ceph-mon[32745]: pgmap v286: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-04T12:35:08.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:08 smithi086 ceph-mon[36851]: osdmap e102: 12 total, 12 up, 12 in 2024-03-04T12:35:08.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:08 smithi086 ceph-mon[36851]: pgmap v288: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:35:08.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:08 smithi063 ceph-mon[36717]: osdmap e102: 12 total, 12 up, 12 in 2024-03-04T12:35:08.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:08 smithi063 ceph-mon[36717]: pgmap v288: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:35:08.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:08 smithi060 ceph-mon[32745]: osdmap e102: 12 total, 12 up, 12 in 2024-03-04T12:35:08.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:08 smithi060 ceph-mon[32745]: pgmap v288: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:35:11.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:11 smithi086 ceph-mon[36851]: pgmap v289: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:35:11.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:11 smithi086 ceph-mon[36851]: osdmap e103: 12 total, 12 up, 12 in 2024-03-04T12:35:11.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:11 smithi063 ceph-mon[36717]: pgmap v289: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:35:11.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:11 smithi063 ceph-mon[36717]: osdmap e103: 12 total, 12 up, 12 in 2024-03-04T12:35:11.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:11 smithi060 ceph-mon[32745]: pgmap v289: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:35:11.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:11 smithi060 ceph-mon[32745]: osdmap e103: 12 total, 12 up, 12 in 2024-03-04T12:35:13.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:13 smithi086 ceph-mon[36851]: pgmap v291: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 67 KiB/s wr, 38 op/s 2024-03-04T12:35:13.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:13 smithi063 ceph-mon[36717]: pgmap v291: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 67 KiB/s wr, 38 op/s 2024-03-04T12:35:13.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:13 smithi060 ceph-mon[32745]: pgmap v291: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 67 KiB/s wr, 38 op/s 2024-03-04T12:35:15.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:15 smithi063 ceph-mon[36717]: pgmap v292: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-04T12:35:15.352 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:15 smithi086 ceph-mon[36851]: pgmap v292: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-04T12:35:15.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:15 smithi060 ceph-mon[32745]: pgmap v292: 129 pgs: 129 active+clean; 80 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-04T12:35:17.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:17 smithi086 ceph-mon[36851]: pgmap v293: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2024-03-04T12:35:17.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:17 smithi063 ceph-mon[36717]: pgmap v293: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2024-03-04T12:35:17.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:17 smithi060 ceph-mon[32745]: pgmap v293: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 252 B/s wr, 12 op/s 2024-03-04T12:35:19.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:19 smithi086 ceph-mon[36851]: pgmap v294: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:35:19.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:19 smithi063 ceph-mon[36717]: pgmap v294: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:35:19.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:19 smithi060 ceph-mon[32745]: pgmap v294: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:35:21.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:21 smithi086 ceph-mon[36851]: pgmap v295: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:35:21.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:21 smithi063 ceph-mon[36717]: pgmap v295: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:35:21.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:21 smithi060 ceph-mon[32745]: pgmap v295: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:35:23.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:23 smithi086 ceph-mon[36851]: pgmap v296: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 90 B/s wr, 9 op/s 2024-03-04T12:35:23.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:23 smithi063 ceph-mon[36717]: pgmap v296: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 90 B/s wr, 9 op/s 2024-03-04T12:35:23.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:23 smithi060 ceph-mon[32745]: pgmap v296: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 90 B/s wr, 9 op/s 2024-03-04T12:35:25.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:25 smithi086 ceph-mon[36851]: pgmap v297: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-04T12:35:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:25 smithi063 ceph-mon[36717]: pgmap v297: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-04T12:35:25.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:25 smithi060 ceph-mon[32745]: pgmap v297: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-04T12:35:27.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:27 smithi086 ceph-mon[36851]: pgmap v298: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-04T12:35:27.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:27 smithi063 ceph-mon[36717]: pgmap v298: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-04T12:35:27.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:27 smithi060 ceph-mon[32745]: pgmap v298: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-04T12:35:29.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:29 smithi086 ceph-mon[36851]: pgmap v299: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:29.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:29 smithi063 ceph-mon[36717]: pgmap v299: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:29.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:29 smithi060 ceph-mon[32745]: pgmap v299: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:31.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:31 smithi086 ceph-mon[36851]: pgmap v300: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:31.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:31 smithi063 ceph-mon[36717]: pgmap v300: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:31.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:31 smithi060 ceph-mon[32745]: pgmap v300: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:33.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:33 smithi086 ceph-mon[36851]: pgmap v301: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:33.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:33 smithi063 ceph-mon[36717]: pgmap v301: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:33 smithi060 ceph-mon[32745]: pgmap v301: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:35.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:35 smithi086 ceph-mon[36851]: pgmap v302: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:35.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:35 smithi063 ceph-mon[36717]: pgmap v302: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:35.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:35 smithi060 ceph-mon[32745]: pgmap v302: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:37.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:37 smithi086 ceph-mon[36851]: pgmap v303: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:37.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:37 smithi063 ceph-mon[36717]: pgmap v303: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:37.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:37 smithi060 ceph-mon[32745]: pgmap v303: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:39.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:39 smithi086 ceph-mon[36851]: pgmap v304: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:39.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:39 smithi063 ceph-mon[36717]: pgmap v304: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:39.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:39 smithi060 ceph-mon[32745]: pgmap v304: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:41.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:41 smithi086 ceph-mon[36851]: pgmap v305: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:41.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:41 smithi063 ceph-mon[36717]: pgmap v305: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:41.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:41 smithi060 ceph-mon[32745]: pgmap v305: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:43.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:43 smithi086 ceph-mon[36851]: pgmap v306: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:43.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:43 smithi063 ceph-mon[36717]: pgmap v306: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:43.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:43 smithi060 ceph-mon[32745]: pgmap v306: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:45.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:45 smithi086 ceph-mon[36851]: pgmap v307: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:45.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:45 smithi063 ceph-mon[36717]: pgmap v307: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:45.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:45 smithi060 ceph-mon[32745]: pgmap v307: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:47.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:47 smithi086 ceph-mon[36851]: pgmap v308: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:47.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:47 smithi063 ceph-mon[36717]: pgmap v308: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:47.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:47 smithi060 ceph-mon[32745]: pgmap v308: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:49.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:49 smithi086 ceph-mon[36851]: pgmap v309: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:49.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:49 smithi063 ceph-mon[36717]: pgmap v309: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:49.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:49 smithi060 ceph-mon[32745]: pgmap v309: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:51.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:51 smithi063 ceph-mon[36717]: pgmap v310: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:51.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:51 smithi060 ceph-mon[32745]: pgmap v310: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:51.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:51 smithi086 ceph-mon[36851]: pgmap v310: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:53.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:53 smithi063 ceph-mon[36717]: pgmap v311: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:53.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:53 smithi060 ceph-mon[32745]: pgmap v311: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:53.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:53 smithi086 ceph-mon[36851]: pgmap v311: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:55.239 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-04T12:35:55.358 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.350+0000 7f1897af1640 1 -- 172.21.15.60:0/674506213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1890107420 msgr2=0x7f1890107800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.350+0000 7f1897af1640 1 --2- 172.21.15.60:0/674506213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1890107420 0x7f1890107800 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f188c009900 tx=0x7f188c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:35:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.350+0000 7f1897af1640 1 -- 172.21.15.60:0/674506213 shutdown_connections 2024-03-04T12:35:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.350+0000 7f1897af1640 1 --2- 172.21.15.60:0/674506213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f189006a2e0 0x7f189010e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.350+0000 7f1897af1640 1 --2- 172.21.15.60:0/674506213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1890069940 0x7f1890069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.350+0000 7f1897af1640 1 --2- 172.21.15.60:0/674506213 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1890107420 0x7f1890107800 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.350+0000 7f1897af1640 1 -- 172.21.15.60:0/674506213 >> 172.21.15.60:0/674506213 conn(0x7f1890077080 msgr2=0x7f1890077490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:55.359 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.350+0000 7f1897af1640 1 -- 172.21.15.60:0/674506213 shutdown_connections 2024-03-04T12:35:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.350+0000 7f1897af1640 1 -- 172.21.15.60:0/674506213 wait complete. 2024-03-04T12:35:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f1897af1640 1 Processor -- start 2024-03-04T12:35:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f1897af1640 1 -- start start 2024-03-04T12:35:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f1897af1640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1890069940 0x7f1890100550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f1897af1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f189006a2e0 0x7f1890100a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f1897af1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1890107420 0x7f1890100fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f1897af1640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18901116d0 con 0x7f1890069940 2024-03-04T12:35:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f1897af1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1890111550 con 0x7f1890107420 2024-03-04T12:35:55.360 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f1897af1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1890111850 con 0x7f189006a2e0 2024-03-04T12:35:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f189588d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1890069940 0x7f1890100550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f189588d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1890069940 0x7f1890100550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:55892/0 (socket says 172.21.15.60:55892) 2024-03-04T12:35:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f189588d640 1 -- 172.21.15.60:0/3379167575 learned_addr learned my addr 172.21.15.60:0/3379167575 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:35:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f189608e640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1890107420 0x7f1890100fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f189508c640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f189006a2e0 0x7f1890100a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f189588d640 1 -- 172.21.15.60:0/3379167575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f189006a2e0 msgr2=0x7f1890100a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f189588d640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f189006a2e0 0x7f1890100a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.361 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.351+0000 7f189588d640 1 -- 172.21.15.60:0/3379167575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1890107420 msgr2=0x7f1890100fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.352+0000 7f189588d640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1890107420 0x7f1890100fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.352+0000 7f189588d640 1 -- 172.21.15.60:0/3379167575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18901a6090 con 0x7f1890069940 2024-03-04T12:35:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.352+0000 7f189508c640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f189006a2e0 0x7f1890100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:35:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.352+0000 7f189608e640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1890107420 0x7f1890100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:35:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.352+0000 7f189588d640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1890069940 0x7f1890100550 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f188c006010 tx=0x7f188c02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.352+0000 7f1886ffd640 1 -- 172.21.15.60:0/3379167575 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c041070 con 0x7f1890069940 2024-03-04T12:35:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.352+0000 7f1897af1640 1 -- 172.21.15.60:0/3379167575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18901a6320 con 0x7f1890069940 2024-03-04T12:35:55.362 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.352+0000 7f1886ffd640 1 -- 172.21.15.60:0/3379167575 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f188c004450 con 0x7f1890069940 2024-03-04T12:35:55.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.352+0000 7f1886ffd640 1 -- 172.21.15.60:0/3379167575 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c003e00 con 0x7f1890069940 2024-03-04T12:35:55.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.352+0000 7f1897af1640 1 -- 172.21.15.60:0/3379167575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18901a6860 con 0x7f1890069940 2024-03-04T12:35:55.363 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.353+0000 7f1897af1640 1 -- 172.21.15.60:0/3379167575 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1890102a30 con 0x7f1890069940 2024-03-04T12:35:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.354+0000 7f1886ffd640 1 -- 172.21.15.60:0/3379167575 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f188c044070 con 0x7f1890069940 2024-03-04T12:35:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.354+0000 7f1886ffd640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f187c0a1ee0 0x7f187c0a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.355+0000 7f1886ffd640 1 -- 172.21.15.60:0/3379167575 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f188c0f02e0 con 0x7f1890069940 2024-03-04T12:35:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.355+0000 7f189508c640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f187c0a1ee0 0x7f187c0a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.355+0000 7f189508c640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f187c0a1ee0 0x7f187c0a43a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1880005d00 tx=0x7f1880009550 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:55.364 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.358+0000 7f1886ffd640 1 -- 172.21.15.60:0/3379167575 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f188c0bd0e0 con 0x7f1890069940 2024-03-04T12:35:55.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:55 smithi060 ceph-mon[32745]: pgmap v312: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:55.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.560+0000 7f1897af1640 1 -- 172.21.15.60:0/3379167575 --> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f1890061770 con 0x7f187c0a1ee0 2024-03-04T12:35:55.574 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.568+0000 7f1886ffd640 1 -- 172.21.15.60:0/3379167575 <== mgr.15735 v2:172.21.15.60:6800/248765372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7f1890061770 con 0x7f187c0a1ee0 2024-03-04T12:35:55.574 INFO:teuthology.orchestra.run.smithi060.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-04T00:00:00", "created": "2024-03-04T12:25:54", "first": "2024-03-04T12:26:00", "last": "2024-03-04T12:35:00", "last_pruned": "2024-03-04T12:35:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-04T12:35:55.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.570+0000 7f1897af1640 1 -- 172.21.15.60:0/3379167575 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f187c0a1ee0 msgr2=0x7f187c0a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:55.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.570+0000 7f1897af1640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f187c0a1ee0 0x7f187c0a43a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1880005d00 tx=0x7f1880009550 comp rx=0 tx=0).stop 2024-03-04T12:35:55.576 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.570+0000 7f1897af1640 1 -- 172.21.15.60:0/3379167575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1890069940 msgr2=0x7f1890100550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:55.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.570+0000 7f1897af1640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1890069940 0x7f1890100550 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f188c006010 tx=0x7f188c02cb30 comp rx=0 tx=0).stop 2024-03-04T12:35:55.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.570+0000 7f1897af1640 1 -- 172.21.15.60:0/3379167575 shutdown_connections 2024-03-04T12:35:55.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.570+0000 7f1897af1640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1890107420 0x7f1890100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.570+0000 7f1897af1640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f189006a2e0 0x7f1890100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.570+0000 7f1897af1640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f187c0a1ee0 0x7f187c0a43a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.570+0000 7f1897af1640 1 --2- 172.21.15.60:0/3379167575 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f1890069940 0x7f1890100550 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.570+0000 7f1897af1640 1 -- 172.21.15.60:0/3379167575 >> 172.21.15.60:0/3379167575 conn(0x7f1890077080 msgr2=0x7f18900ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:55.577 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.570+0000 7f1897af1640 1 -- 172.21.15.60:0/3379167575 shutdown_connections 2024-03-04T12:35:55.578 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.570+0000 7f1897af1640 1 -- 172.21.15.60:0/3379167575 wait complete. 2024-03-04T12:35:55.586 DEBUG:teuthology.orchestra.run.smithi060:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-04T12:35:55.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:55 smithi086 ceph-mon[36851]: pgmap v312: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:55.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:55 smithi063 ceph-mon[36717]: pgmap v312: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:35:55.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.684+0000 7efcff468640 1 -- 172.21.15.60:0/313014169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf810e030 msgr2=0x7efcf8110420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:55.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.684+0000 7efcff468640 1 --2- 172.21.15.60:0/313014169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf810e030 0x7efcf8110420 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009ca0 tx=0x7efcf002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.684+0000 7efcff468640 1 -- 172.21.15.60:0/313014169 shutdown_connections 2024-03-04T12:35:55.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.684+0000 7efcff468640 1 --2- 172.21.15.60:0/313014169 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf810e030 0x7efcf8110420 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.684+0000 7efcff468640 1 --2- 172.21.15.60:0/313014169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efcf806a260 0x7efcf810daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.684+0000 7efcff468640 1 --2- 172.21.15.60:0/313014169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf8069940 0x7efcf8069d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.684+0000 7efcff468640 1 -- 172.21.15.60:0/313014169 >> 172.21.15.60:0/313014169 conn(0x7efcf8076ff0 msgr2=0x7efcf8077400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:55.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.684+0000 7efcff468640 1 -- 172.21.15.60:0/313014169 shutdown_connections 2024-03-04T12:35:55.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.684+0000 7efcff468640 1 -- 172.21.15.60:0/313014169 wait complete. 2024-03-04T12:35:55.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.685+0000 7efcff468640 1 Processor -- start 2024-03-04T12:35:55.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.685+0000 7efcff468640 1 -- start start 2024-03-04T12:35:55.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.685+0000 7efcff468640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf8069940 0x7efcf8199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:55.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.685+0000 7efcff468640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efcf806a260 0x7efcf819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:55.694 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.685+0000 7efcff468640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf810e030 0x7efcf819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.685+0000 7efcff468640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf8113960 con 0x7efcf810e030 2024-03-04T12:35:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.685+0000 7efcff468640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf81137e0 con 0x7efcf806a260 2024-03-04T12:35:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.685+0000 7efcff468640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf8113ae0 con 0x7efcf8069940 2024-03-04T12:35:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.685+0000 7efcfda05640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf810e030 0x7efcf819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.685+0000 7efcfda05640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf810e030 0x7efcf819e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52338/0 (socket says 172.21.15.60:52338) 2024-03-04T12:35:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.685+0000 7efcfda05640 1 -- 172.21.15.60:0/1090159207 learned_addr learned my addr 172.21.15.60:0/1090159207 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:35:55.695 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcfd204640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf8069940 0x7efcf8199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:55.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcfca03640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efcf806a260 0x7efcf819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:55.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcfda05640 1 -- 172.21.15.60:0/1090159207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf8069940 msgr2=0x7efcf8199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:55.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcfda05640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf8069940 0x7efcf8199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcfda05640 1 -- 172.21.15.60:0/1090159207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efcf806a260 msgr2=0x7efcf819a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:55.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcfda05640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efcf806a260 0x7efcf819a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcfda05640 1 -- 172.21.15.60:0/1090159207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf819eda0 con 0x7efcf810e030 2024-03-04T12:35:55.696 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcfda05640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf810e030 0x7efcf819e670 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7efcf00348f0 tx=0x7efcf0032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:55.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcee7fc640 1 -- 172.21.15.60:0/1090159207 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0041020 con 0x7efcf810e030 2024-03-04T12:35:55.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcee7fc640 1 -- 172.21.15.60:0/1090159207 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efcf0034cd0 con 0x7efcf810e030 2024-03-04T12:35:55.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcff468640 1 -- 172.21.15.60:0/1090159207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf819f030 con 0x7efcf810e030 2024-03-04T12:35:55.697 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcee7fc640 1 -- 172.21.15.60:0/1090159207 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0039040 con 0x7efcf810e030 2024-03-04T12:35:55.698 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.686+0000 7efcff468640 1 -- 172.21.15.60:0/1090159207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf81a6920 con 0x7efcf810e030 2024-03-04T12:35:55.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.687+0000 7efcff468640 1 -- 172.21.15.60:0/1090159207 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf8100850 con 0x7efcf810e030 2024-03-04T12:35:55.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.688+0000 7efcee7fc640 1 -- 172.21.15.60:0/1090159207 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7efcf0044020 con 0x7efcf810e030 2024-03-04T12:35:55.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.689+0000 7efcee7fc640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7efce40a1f60 0x7efce40a4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:55.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.689+0000 7efcfd204640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7efce40a1f60 0x7efce40a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:55.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.689+0000 7efcee7fc640 1 -- 172.21.15.60:0/1090159207 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9539+0+0 (secure 0 0 0) 0x7efcf00f0730 con 0x7efcf810e030 2024-03-04T12:35:55.699 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.689+0000 7efcfd204640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7efce40a1f60 0x7efce40a4420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efcf4005fd0 tx=0x7efcf400b390 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:55.700 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.693+0000 7efcee7fc640 1 -- 172.21.15.60:0/1090159207 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efcf004b020 con 0x7efcf810e030 2024-03-04T12:35:55.902 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.897+0000 7efcff468640 1 -- 172.21.15.60:0/1090159207 --> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7efcf8109230 con 0x7efce40a1f60 2024-03-04T12:35:55.909 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.904+0000 7efcee7fc640 1 -- 172.21.15.60:0/1090159207 <== mgr.15735 v2:172.21.15.60:6800/248765372 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7efcf400b870 con 0x7efce40a1f60 2024-03-04T12:35:55.910 INFO:teuthology.orchestra.run.smithi060.stdout:/ 1m 6m3h 2024-03-04T12:35:55.912 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.906+0000 7efcff468640 1 -- 172.21.15.60:0/1090159207 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7efce40a1f60 msgr2=0x7efce40a4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:55.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.906+0000 7efcff468640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7efce40a1f60 0x7efce40a4420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efcf4005fd0 tx=0x7efcf400b390 comp rx=0 tx=0).stop 2024-03-04T12:35:55.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.906+0000 7efcff468640 1 -- 172.21.15.60:0/1090159207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf810e030 msgr2=0x7efcf819e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:55.913 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.906+0000 7efcff468640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf810e030 0x7efcf819e670 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7efcf00348f0 tx=0x7efcf0032ec0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.906+0000 7efcff468640 1 -- 172.21.15.60:0/1090159207 shutdown_connections 2024-03-04T12:35:55.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.906+0000 7efcff468640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7efce40a1f60 0x7efce40a4420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.906+0000 7efcff468640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7efcf810e030 0x7efcf819e670 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.914 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.906+0000 7efcff468640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efcf806a260 0x7efcf819a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.906+0000 7efcff468640 1 --2- 172.21.15.60:0/1090159207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf8069940 0x7efcf8199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:55.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.906+0000 7efcff468640 1 -- 172.21.15.60:0/1090159207 >> 172.21.15.60:0/1090159207 conn(0x7efcf8076ff0 msgr2=0x7efcf810f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:55.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.906+0000 7efcff468640 1 -- 172.21.15.60:0/1090159207 shutdown_connections 2024-03-04T12:35:55.915 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:55.906+0000 7efcff468640 1 -- 172.21.15.60:0/1090159207 wait complete. 2024-03-04T12:35:55.923 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-04T12:35:55.935 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-04T12:35:55.935 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_717ce3cce8d0166e4e577faa8004e6f9cb4128c0/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-04T12:35:55.942 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-04T12:35:55.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:35:56.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.054+0000 7fee8df1b640 1 -- 172.21.15.60:0/3457937696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88105520 msgr2=0x7fee8810d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.054+0000 7fee8df1b640 1 --2- 172.21.15.60:0/3457937696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88105520 0x7fee8810d0f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fee80009ca0 tx=0x7fee8002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.054+0000 7fee8df1b640 1 -- 172.21.15.60:0/3457937696 shutdown_connections 2024-03-04T12:35:56.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.054+0000 7fee8df1b640 1 --2- 172.21.15.60:0/3457937696 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88105520 0x7fee8810d0f0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.062 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.054+0000 7fee8df1b640 1 --2- 172.21.15.60:0/3457937696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee88100e00 0x7fee88104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.054+0000 7fee8df1b640 1 --2- 172.21.15.60:0/3457937696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee88100450 0x7fee88100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.054+0000 7fee8df1b640 1 -- 172.21.15.60:0/3457937696 >> 172.21.15.60:0/3457937696 conn(0x7fee88077470 msgr2=0x7fee88077880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.054+0000 7fee8df1b640 1 -- 172.21.15.60:0/3457937696 shutdown_connections 2024-03-04T12:35:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.054+0000 7fee8df1b640 1 -- 172.21.15.60:0/3457937696 wait complete. 2024-03-04T12:35:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.054+0000 7fee8df1b640 1 Processor -- start 2024-03-04T12:35:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee8df1b640 1 -- start start 2024-03-04T12:35:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee8df1b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee88100450 0x7fee88198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.063 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee8df1b640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88105520 0x7fee88198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee8df1b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee881993e0 0x7fee881a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee8df1b640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee881105b0 con 0x7fee88105520 2024-03-04T12:35:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee8df1b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee88110430 con 0x7fee88100450 2024-03-04T12:35:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee8df1b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee88110730 con 0x7fee881993e0 2024-03-04T12:35:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee86ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88105520 0x7fee88198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee86ffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88105520 0x7fee88198e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52352/0 (socket says 172.21.15.60:52352) 2024-03-04T12:35:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee86ffd640 1 -- 172.21.15.60:0/3394917884 learned_addr learned my addr 172.21.15.60:0/3394917884 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:35:56.064 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee877fe640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee88100450 0x7fee88198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee87fff640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee881993e0 0x7fee881a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee86ffd640 1 -- 172.21.15.60:0/3394917884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee881993e0 msgr2=0x7fee881a3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee86ffd640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee881993e0 0x7fee881a3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee86ffd640 1 -- 172.21.15.60:0/3394917884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee88100450 msgr2=0x7fee88198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee86ffd640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee88100450 0x7fee88198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee86ffd640 1 -- 172.21.15.60:0/3394917884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee881a4160 con 0x7fee88105520 2024-03-04T12:35:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.055+0000 7fee86ffd640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88105520 0x7fee88198e50 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00c970 tx=0x7fee7c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:56.065 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.056+0000 7fee84ff9640 1 -- 172.21.15.60:0/3394917884 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c007bb0 con 0x7fee88105520 2024-03-04T12:35:56.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.056+0000 7fee84ff9640 1 -- 172.21.15.60:0/3394917884 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee7c007d50 con 0x7fee88105520 2024-03-04T12:35:56.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.056+0000 7fee8df1b640 1 -- 172.21.15.60:0/3394917884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee881a4450 con 0x7fee88105520 2024-03-04T12:35:56.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.056+0000 7fee84ff9640 1 -- 172.21.15.60:0/3394917884 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee7c00f450 con 0x7fee88105520 2024-03-04T12:35:56.066 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.056+0000 7fee8df1b640 1 -- 172.21.15.60:0/3394917884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee881a49a0 con 0x7fee88105520 2024-03-04T12:35:56.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.057+0000 7fee8df1b640 1 -- 172.21.15.60:0/3394917884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee88101920 con 0x7fee88105520 2024-03-04T12:35:56.067 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.058+0000 7fee84ff9640 1 -- 172.21.15.60:0/3394917884 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7fee7c0046c0 con 0x7fee88105520 2024-03-04T12:35:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.059+0000 7fee84ff9640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fee700a1ef0 0x7fee700a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.059+0000 7fee877fe640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fee700a1ef0 0x7fee700a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.059+0000 7fee84ff9640 1 -- 172.21.15.60:0/3394917884 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9539+0+0 (secure 0 0 0) 0x7fee7c0ce650 con 0x7fee88105520 2024-03-04T12:35:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.059+0000 7fee877fe640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fee700a1ef0 0x7fee700a43b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fee78005e10 tx=0x7fee78005da0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:56.068 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.063+0000 7fee84ff9640 1 -- 172.21.15.60:0/3394917884 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fee7c014030 con 0x7fee88105520 2024-03-04T12:35:56.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.247+0000 7fee8df1b640 1 -- 172.21.15.60:0/3394917884 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee88068600 con 0x7fee88105520 2024-03-04T12:35:56.254 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.248+0000 7fee84ff9640 1 -- 172.21.15.60:0/3394917884 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25187 (secure 0 0 0) 0x7fee7c09b340 con 0x7fee88105520 2024-03-04T12:35:56.254 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:35:56.254 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":103,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:35:10.030774+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2167195614":"2024-03-05T12:25:53.138536+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/1914863179":"2024-03-05T12:25:53.138536+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3302898178":"2024-03-05T12:25:53.138536+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/2604977412":"2024-03-05T12:25:53.138536+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/1914863179":"2024-03-05T12:25:53.138536+0000","172.21.15.60:0/366921369":"2024-03-05T12:25:53.138536+0000","172.21.15.60:0/1342937290":"2024-03-05T12:25:53.138536+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:35:56.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.250+0000 7fee8df1b640 1 -- 172.21.15.60:0/3394917884 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fee700a1ef0 msgr2=0x7fee700a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.250+0000 7fee8df1b640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fee700a1ef0 0x7fee700a43b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fee78005e10 tx=0x7fee78005da0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.250+0000 7fee8df1b640 1 -- 172.21.15.60:0/3394917884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88105520 msgr2=0x7fee88198e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.250+0000 7fee8df1b640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88105520 0x7fee88198e50 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00c970 tx=0x7fee7c00ce30 comp rx=0 tx=0).stop 2024-03-04T12:35:56.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.250+0000 7fee8df1b640 1 -- 172.21.15.60:0/3394917884 shutdown_connections 2024-03-04T12:35:56.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.250+0000 7fee8df1b640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee881993e0 0x7fee881a3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.250+0000 7fee8df1b640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fee700a1ef0 0x7fee700a43b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.250+0000 7fee8df1b640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fee88105520 0x7fee88198e50 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.250+0000 7fee8df1b640 1 --2- 172.21.15.60:0/3394917884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee88100450 0x7fee88198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.250+0000 7fee8df1b640 1 -- 172.21.15.60:0/3394917884 >> 172.21.15.60:0/3394917884 conn(0x7fee88077470 msgr2=0x7fee880fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:56.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.250+0000 7fee8df1b640 1 -- 172.21.15.60:0/3394917884 shutdown_connections 2024-03-04T12:35:56.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.250+0000 7fee8df1b640 1 -- 172.21.15.60:0/3394917884 wait complete. 2024-03-04T12:35:56.267 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T12:23:10.438928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T12:23:11.472998+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T12:23:15.162500+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T12:35:56.268 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:35:56.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.368+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2965509999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c4c0ff7a0 msgr2=0x7f2c4c0ffc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.368+0000 7f2c50fb2640 1 --2- 172.21.15.60:0/2965509999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c4c0ff7a0 0x7f2c4c0ffc20 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009930 tx=0x7f2c3402bc30 comp rx=0 tx=0).stop 2024-03-04T12:35:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.368+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2965509999 shutdown_connections 2024-03-04T12:35:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.368+0000 7f2c50fb2640 1 --2- 172.21.15.60:0/2965509999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c4c100160 0x7f2c4c10e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.368+0000 7f2c50fb2640 1 --2- 172.21.15.60:0/2965509999 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c4c0ff7a0 0x7f2c4c0ffc20 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.368+0000 7f2c50fb2640 1 --2- 172.21.15.60:0/2965509999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c4c106300 0x7f2c4c1066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.368+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2965509999 >> 172.21.15.60:0/2965509999 conn(0x7f2c4c076730 msgr2=0x7f2c4c0fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:56.377 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.368+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2965509999 shutdown_connections 2024-03-04T12:35:56.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.368+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2965509999 wait complete. 2024-03-04T12:35:56.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c50fb2640 1 Processor -- start 2024-03-04T12:35:56.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c50fb2640 1 -- start start 2024-03-04T12:35:56.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c50fb2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c4c0ff7a0 0x7f2c4c197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c50fb2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c4c100160 0x7f2c4c198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c50fb2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c4c106300 0x7f2c4c19c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.378 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c50fb2640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c1117d0 con 0x7f2c4c100160 2024-03-04T12:35:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c50fb2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c111650 con 0x7f2c4c106300 2024-03-04T12:35:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c50fb2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c111950 con 0x7f2c4c0ff7a0 2024-03-04T12:35:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c43fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c4c100160 0x7f2c4c198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c43fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c4c100160 0x7f2c4c198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52368/0 (socket says 172.21.15.60:52368) 2024-03-04T12:35:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c43fff640 1 -- 172.21.15.60:0/2099703023 learned_addr learned my addr 172.21.15.60:0/2099703023 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:35:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c4ad9d640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c4c106300 0x7f2c4c19c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.369+0000 7f2c4a59c640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c4c0ff7a0 0x7f2c4c197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.370+0000 7f2c43fff640 1 -- 172.21.15.60:0/2099703023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c4c0ff7a0 msgr2=0x7f2c4c197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.379 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.370+0000 7f2c43fff640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c4c0ff7a0 0x7f2c4c197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.370+0000 7f2c43fff640 1 -- 172.21.15.60:0/2099703023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c4c106300 msgr2=0x7f2c4c19c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.370+0000 7f2c43fff640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c4c106300 0x7f2c4c19c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.370+0000 7f2c43fff640 1 -- 172.21.15.60:0/2099703023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c19cca0 con 0x7f2c4c100160 2024-03-04T12:35:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.370+0000 7f2c43fff640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c4c100160 0x7f2c4c198290 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009900 tx=0x7f2c340043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.370+0000 7f2c437fe640 1 -- 172.21.15.60:0/2099703023 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34041070 con 0x7f2c4c100160 2024-03-04T12:35:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.370+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2099703023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c4c19cf30 con 0x7f2c4c100160 2024-03-04T12:35:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.370+0000 7f2c437fe640 1 -- 172.21.15.60:0/2099703023 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c3402ccc0 con 0x7f2c4c100160 2024-03-04T12:35:56.380 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.370+0000 7f2c437fe640 1 -- 172.21.15.60:0/2099703023 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34003d80 con 0x7f2c4c100160 2024-03-04T12:35:56.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.370+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2099703023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4c1a4820 con 0x7f2c4c100160 2024-03-04T12:35:56.381 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.371+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2099703023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4c068600 con 0x7f2c4c100160 2024-03-04T12:35:56.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.372+0000 7f2c437fe640 1 -- 172.21.15.60:0/2099703023 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f2c34044070 con 0x7f2c4c100160 2024-03-04T12:35:56.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.372+0000 7f2c437fe640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2c380a1ef0 0x7f2c380a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.373+0000 7f2c4a59c640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2c380a1ef0 0x7f2c380a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:56.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.373+0000 7f2c437fe640 1 -- 172.21.15.60:0/2099703023 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f2c340f0240 con 0x7f2c4c100160 2024-03-04T12:35:56.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.373+0000 7f2c4a59c640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2c380a1ef0 0x7f2c380a43b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c005fd0 tx=0x7f2c3c00b390 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:56.382 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.376+0000 7f2c437fe640 1 -- 172.21.15.60:0/2099703023 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2c340bcab0 con 0x7f2c4c100160 2024-03-04T12:35:56.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.560+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2099703023 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c4c1097c0 con 0x7f2c4c100160 2024-03-04T12:35:56.567 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.560+0000 7f2c437fe640 1 -- 172.21.15.60:0/2099703023 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c340c0b50 con 0x7f2c4c100160 2024-03-04T12:35:56.567 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:35:56.568 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.562+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2099703023 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2c380a1ef0 msgr2=0x7f2c380a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.562+0000 7f2c50fb2640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2c380a1ef0 0x7f2c380a43b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c005fd0 tx=0x7f2c3c00b390 comp rx=0 tx=0).stop 2024-03-04T12:35:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.562+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2099703023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c4c100160 msgr2=0x7f2c4c198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.562+0000 7f2c50fb2640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c4c100160 0x7f2c4c198290 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2c34009900 tx=0x7f2c340043b0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.562+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2099703023 shutdown_connections 2024-03-04T12:35:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.562+0000 7f2c50fb2640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2c4c106300 0x7f2c4c19c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.562+0000 7f2c50fb2640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2c380a1ef0 0x7f2c380a43b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.562+0000 7f2c50fb2640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2c4c100160 0x7f2c4c198290 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.569 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.562+0000 7f2c50fb2640 1 --2- 172.21.15.60:0/2099703023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c4c0ff7a0 0x7f2c4c197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.562+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2099703023 >> 172.21.15.60:0/2099703023 conn(0x7f2c4c076730 msgr2=0x7f2c4c104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:56.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.563+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2099703023 shutdown_connections 2024-03-04T12:35:56.570 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.563+0000 7f2c50fb2640 1 -- 172.21.15.60:0/2099703023 wait complete. 2024-03-04T12:35:56.579 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T12:35:56.687 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.679+0000 7fbe4e960640 1 -- 172.21.15.60:0/749375163 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe48101e20 msgr2=0x7fbe48102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.679+0000 7fbe4e960640 1 --2- 172.21.15.60:0/749375163 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe48101e20 0x7fbe48102280 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c009930 tx=0x7fbe3c02bc20 comp rx=0 tx=0).stop 2024-03-04T12:35:56.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.679+0000 7fbe4e960640 1 -- 172.21.15.60:0/749375163 shutdown_connections 2024-03-04T12:35:56.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.679+0000 7fbe4e960640 1 --2- 172.21.15.60:0/749375163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe481027c0 0x7fbe4810d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.679+0000 7fbe4e960640 1 --2- 172.21.15.60:0/749375163 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe48101e20 0x7fbe48102280 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.679+0000 7fbe4e960640 1 --2- 172.21.15.60:0/749375163 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe48100540 0x7fbe48100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.679+0000 7fbe4e960640 1 -- 172.21.15.60:0/749375163 >> 172.21.15.60:0/749375163 conn(0x7fbe48077410 msgr2=0x7fbe48077820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:56.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.679+0000 7fbe4e960640 1 -- 172.21.15.60:0/749375163 shutdown_connections 2024-03-04T12:35:56.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.679+0000 7fbe4e960640 1 -- 172.21.15.60:0/749375163 wait complete. 2024-03-04T12:35:56.688 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe4e960640 1 Processor -- start 2024-03-04T12:35:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe4e960640 1 -- start start 2024-03-04T12:35:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe4e960640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe48100540 0x7fbe480fd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe4e960640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe48101e20 0x7fbe480fd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe4e960640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe481027c0 0x7fbe480fdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe4e960640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe48110690 con 0x7fbe48101e20 2024-03-04T12:35:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe4e960640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe48110510 con 0x7fbe481027c0 2024-03-04T12:35:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe4e960640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe48110810 con 0x7fbe48100540 2024-03-04T12:35:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe48101e20 0x7fbe480fd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:56.689 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe477fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe48101e20 0x7fbe480fd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52390/0 (socket says 172.21.15.60:52390) 2024-03-04T12:35:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe477fe640 1 -- 172.21.15.60:0/2516661854 learned_addr learned my addr 172.21.15.60:0/2516661854 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:35:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe4cefd640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe481027c0 0x7fbe480fdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe47fff640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe48100540 0x7fbe480fd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe477fe640 1 -- 172.21.15.60:0/2516661854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe48100540 msgr2=0x7fbe480fd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe477fe640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe48100540 0x7fbe480fd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.680+0000 7fbe477fe640 1 -- 172.21.15.60:0/2516661854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe481027c0 msgr2=0x7fbe480fdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.681+0000 7fbe477fe640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe481027c0 0x7fbe480fdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.681+0000 7fbe477fe640 1 -- 172.21.15.60:0/2516661854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe481a6020 con 0x7fbe48101e20 2024-03-04T12:35:56.690 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.681+0000 7fbe47fff640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe48100540 0x7fbe480fd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:35:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.681+0000 7fbe477fe640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe48101e20 0x7fbe480fd6b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c005ab0 tx=0x7fbe3c02cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.681+0000 7fbe457fa640 1 -- 172.21.15.60:0/2516661854 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c041070 con 0x7fbe48101e20 2024-03-04T12:35:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.681+0000 7fbe4e960640 1 -- 172.21.15.60:0/2516661854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe481a62b0 con 0x7fbe48101e20 2024-03-04T12:35:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.681+0000 7fbe457fa640 1 -- 172.21.15.60:0/2516661854 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe3c004450 con 0x7fbe48101e20 2024-03-04T12:35:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.681+0000 7fbe457fa640 1 -- 172.21.15.60:0/2516661854 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c034440 con 0x7fbe48101e20 2024-03-04T12:35:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.681+0000 7fbe4e960640 1 -- 172.21.15.60:0/2516661854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe481a67f0 con 0x7fbe48101e20 2024-03-04T12:35:56.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.682+0000 7fbe4e960640 1 -- 172.21.15.60:0/2516661854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe48068600 con 0x7fbe48101e20 2024-03-04T12:35:56.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.683+0000 7fbe457fa640 1 -- 172.21.15.60:0/2516661854 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7fbe3c044070 con 0x7fbe48101e20 2024-03-04T12:35:56.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.683+0000 7fbe457fa640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbe300a1ef0 0x7fbe300a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.692 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.684+0000 7fbe47fff640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbe300a1ef0 0x7fbe300a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:56.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.684+0000 7fbe457fa640 1 -- 172.21.15.60:0/2516661854 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9539+0+0 (secure 0 0 0) 0x7fbe3c0f0130 con 0x7fbe48101e20 2024-03-04T12:35:56.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.684+0000 7fbe47fff640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbe300a1ef0 0x7fbe300a43b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800abe0 tx=0x7fbe3800a5c0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:56.693 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.688+0000 7fbe457fa640 1 -- 172.21.15.60:0/2516661854 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fbe3c0bcf30 con 0x7fbe48101e20 2024-03-04T12:35:56.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.869+0000 7fbe4e960640 1 -- 172.21.15.60:0/2516661854 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbe48109820 con 0x7fbe48101e20 2024-03-04T12:35:56.875 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.870+0000 7fbe457fa640 1 -- 172.21.15.60:0/2516661854 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbe3c0bc830 con 0x7fbe48101e20 2024-03-04T12:35:56.875 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:35:56.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.872+0000 7fbe4e960640 1 -- 172.21.15.60:0/2516661854 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbe300a1ef0 msgr2=0x7fbe300a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.872+0000 7fbe4e960640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbe300a1ef0 0x7fbe300a43b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbe3800abe0 tx=0x7fbe3800a5c0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.872+0000 7fbe4e960640 1 -- 172.21.15.60:0/2516661854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe48101e20 msgr2=0x7fbe480fd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.872+0000 7fbe4e960640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe48101e20 0x7fbe480fd6b0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c005ab0 tx=0x7fbe3c02cca0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.872+0000 7fbe4e960640 1 -- 172.21.15.60:0/2516661854 shutdown_connections 2024-03-04T12:35:56.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.872+0000 7fbe4e960640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe481027c0 0x7fbe480fdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.872+0000 7fbe4e960640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fbe300a1ef0 0x7fbe300a43b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.878 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.872+0000 7fbe4e960640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fbe48101e20 0x7fbe480fd6b0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.872+0000 7fbe4e960640 1 --2- 172.21.15.60:0/2516661854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe48100540 0x7fbe480fd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.872+0000 7fbe4e960640 1 -- 172.21.15.60:0/2516661854 >> 172.21.15.60:0/2516661854 conn(0x7fbe48077410 msgr2=0x7fbe48069bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:56.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.872+0000 7fbe4e960640 1 -- 172.21.15.60:0/2516661854 shutdown_connections 2024-03-04T12:35:56.879 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.872+0000 7fbe4e960640 1 -- 172.21.15.60:0/2516661854 wait complete. 2024-03-04T12:35:56.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T12:35:56.997 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.989+0000 7f4644273640 1 -- 172.21.15.60:0/3135933238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f463c100440 msgr2=0x7f463c100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:56.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.989+0000 7f4644273640 1 --2- 172.21.15.60:0/3135933238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f463c100440 0x7f463c100820 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f4638009ca0 tx=0x7f463802c0f0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.989+0000 7f4644273640 1 -- 172.21.15.60:0/3135933238 shutdown_connections 2024-03-04T12:35:56.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.989+0000 7f4644273640 1 --2- 172.21.15.60:0/3135933238 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f463c105510 0x7f463c10d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.989+0000 7f4644273640 1 --2- 172.21.15.60:0/3135933238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f463c100df0 0x7f463c104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.989+0000 7f4644273640 1 --2- 172.21.15.60:0/3135933238 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f463c100440 0x7f463c100820 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:56.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.989+0000 7f4644273640 1 -- 172.21.15.60:0/3135933238 >> 172.21.15.60:0/3135933238 conn(0x7f463c0773e0 msgr2=0x7f463c0777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:56.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.989+0000 7f4644273640 1 -- 172.21.15.60:0/3135933238 shutdown_connections 2024-03-04T12:35:56.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f4644273640 1 -- 172.21.15.60:0/3135933238 wait complete. 2024-03-04T12:35:56.998 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f4644273640 1 Processor -- start 2024-03-04T12:35:56.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f4644273640 1 -- start start 2024-03-04T12:35:56.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f4644273640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f463c100440 0x7f463c19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f4644273640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f463c105510 0x7f463c19cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f4644273640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f463c196900 0x7f463c196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:56.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f4644273640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c1105a0 con 0x7f463c105510 2024-03-04T12:35:56.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f4644273640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c110420 con 0x7f463c196900 2024-03-04T12:35:56.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f4644273640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463c110720 con 0x7f463c100440 2024-03-04T12:35:56.999 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f464180e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f463c105510 0x7f463c19cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:57.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f464180e640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f463c105510 0x7f463c19cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52418/0 (socket says 172.21.15.60:52418) 2024-03-04T12:35:57.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f464180e640 1 -- 172.21.15.60:0/2834585757 learned_addr learned my addr 172.21.15.60:0/2834585757 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:35:57.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f464200f640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f463c100440 0x7f463c19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:57.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.990+0000 7f4642810640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f463c196900 0x7f463c196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:57.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.991+0000 7f464180e640 1 -- 172.21.15.60:0/2834585757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f463c100440 msgr2=0x7f463c19c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:57.000 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.991+0000 7f464180e640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f463c100440 0x7f463c19c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.991+0000 7f464180e640 1 -- 172.21.15.60:0/2834585757 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f463c196900 msgr2=0x7f463c196db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:57.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.991+0000 7f464180e640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f463c196900 0x7f463c196db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.991+0000 7f464180e640 1 -- 172.21.15.60:0/2834585757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c197420 con 0x7f463c105510 2024-03-04T12:35:57.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.991+0000 7f4642810640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f463c196900 0x7f463c196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:35:57.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.991+0000 7f464180e640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f463c105510 0x7f463c19cc90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f462c00e9c0 tx=0x7f462c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:57.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.991+0000 7f46337fe640 1 -- 172.21.15.60:0/2834585757 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c00cd80 con 0x7f463c105510 2024-03-04T12:35:57.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.991+0000 7f46337fe640 1 -- 172.21.15.60:0/2834585757 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f462c005500 con 0x7f463c105510 2024-03-04T12:35:57.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.991+0000 7f4644273640 1 -- 172.21.15.60:0/2834585757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f463c197710 con 0x7f463c105510 2024-03-04T12:35:57.001 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.991+0000 7f46337fe640 1 -- 172.21.15.60:0/2834585757 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462c010690 con 0x7f463c105510 2024-03-04T12:35:57.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.991+0000 7f4644273640 1 -- 172.21.15.60:0/2834585757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463c1a3960 con 0x7f463c105510 2024-03-04T12:35:57.002 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.992+0000 7f4644273640 1 -- 172.21.15.60:0/2834585757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f463c0681b0 con 0x7f463c105510 2024-03-04T12:35:57.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.993+0000 7f46337fe640 1 -- 172.21.15.60:0/2834585757 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f462c0046c0 con 0x7f463c105510 2024-03-04T12:35:57.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.994+0000 7f46337fe640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f46280a1ef0 0x7f46280a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:57.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.994+0000 7f464200f640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f46280a1ef0 0x7f46280a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:57.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.994+0000 7f46337fe640 1 -- 172.21.15.60:0/2834585757 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f462c0ce9e0 con 0x7f463c105510 2024-03-04T12:35:57.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.994+0000 7f464200f640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f46280a1ef0 0x7f46280a43b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4638004af0 tx=0x7f4638005c50 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:57.003 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:56.998+0000 7f46337fe640 1 -- 172.21.15.60:0/2834585757 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f462c09b6d0 con 0x7f463c105510 2024-03-04T12:35:57.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:57 smithi060 ceph-mon[32745]: pgmap v313: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:57.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:57 smithi060 ceph-mon[32745]: from='client.16554 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:35:57.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:57 smithi060 ceph-mon[32745]: from='client.16560 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:35:57.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:57 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3394917884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:35:57.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:57 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2099703023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:35:57.179 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:57 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2516661854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:35:57.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.181+0000 7f4644273640 1 -- 172.21.15.60:0/2834585757 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f463c100df0 con 0x7f463c105510 2024-03-04T12:35:57.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.181+0000 7f46337fe640 1 -- 172.21.15.60:0/2834585757 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) v1 ==== 100+0+11 (secure 0 0 0) 0x7f462c09afd0 con 0x7f463c105510 2024-03-04T12:35:57.187 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-03-04T12:35:57.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.183+0000 7f4644273640 1 -- 172.21.15.60:0/2834585757 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f46280a1ef0 msgr2=0x7f46280a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:57.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.183+0000 7f4644273640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f46280a1ef0 0x7f46280a43b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4638004af0 tx=0x7f4638005c50 comp rx=0 tx=0).stop 2024-03-04T12:35:57.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.183+0000 7f4644273640 1 -- 172.21.15.60:0/2834585757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f463c105510 msgr2=0x7f463c19cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:57.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.183+0000 7f4644273640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f463c105510 0x7f463c19cc90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f462c00e9c0 tx=0x7f462c00ee80 comp rx=0 tx=0).stop 2024-03-04T12:35:57.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.183+0000 7f4644273640 1 -- 172.21.15.60:0/2834585757 shutdown_connections 2024-03-04T12:35:57.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.183+0000 7f4644273640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f463c196900 0x7f463c196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.183+0000 7f4644273640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f46280a1ef0 0x7f46280a43b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.183+0000 7f4644273640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f463c105510 0x7f463c19cc90 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.183+0000 7f4644273640 1 --2- 172.21.15.60:0/2834585757 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f463c100440 0x7f463c19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.183+0000 7f4644273640 1 -- 172.21.15.60:0/2834585757 >> 172.21.15.60:0/2834585757 conn(0x7f463c0773e0 msgr2=0x7f463c0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:57.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.183+0000 7f4644273640 1 -- 172.21.15.60:0/2834585757 shutdown_connections 2024-03-04T12:35:57.190 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.183+0000 7f4644273640 1 -- 172.21.15.60:0/2834585757 wait complete. 2024-03-04T12:35:57.199 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T12:35:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.299+0000 7f1451569640 1 -- 172.21.15.60:0/3166439639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f144c0ffa90 msgr2=0x7f144c0ffe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.299+0000 7f1451569640 1 --2- 172.21.15.60:0/3166439639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f144c0ffa90 0x7f144c0ffe70 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f143c009900 tx=0x7f143c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:35:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.299+0000 7f1451569640 1 -- 172.21.15.60:0/3166439639 shutdown_connections 2024-03-04T12:35:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.299+0000 7f1451569640 1 --2- 172.21.15.60:0/3166439639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f144c106650 0x7f144c10e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.299+0000 7f1451569640 1 --2- 172.21.15.60:0/3166439639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f144c100440 0x7f144c105f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.299+0000 7f1451569640 1 --2- 172.21.15.60:0/3166439639 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f144c0ffa90 0x7f144c0ffe70 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.307 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.299+0000 7f1451569640 1 -- 172.21.15.60:0/3166439639 >> 172.21.15.60:0/3166439639 conn(0x7f144c076730 msgr2=0x7f144c0fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.299+0000 7f1451569640 1 -- 172.21.15.60:0/3166439639 shutdown_connections 2024-03-04T12:35:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.299+0000 7f1451569640 1 -- 172.21.15.60:0/3166439639 wait complete. 2024-03-04T12:35:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f1451569640 1 Processor -- start 2024-03-04T12:35:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f1451569640 1 -- start start 2024-03-04T12:35:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f1451569640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f144c0ffa90 0x7f144c19d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f1451569640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f144c106650 0x7f144c19dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f1451569640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f144c197920 0x7f144c197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:57.308 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f1451569640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c111720 con 0x7f144c106650 2024-03-04T12:35:57.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f1451569640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c1115a0 con 0x7f144c197920 2024-03-04T12:35:57.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f1451569640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144c1118a0 con 0x7f144c0ffa90 2024-03-04T12:35:57.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f14427fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f144c106650 0x7f144c19dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:57.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f14427fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f144c106650 0x7f144c19dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52434/0 (socket says 172.21.15.60:52434) 2024-03-04T12:35:57.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f14427fc640 1 -- 172.21.15.60:0/2847694656 learned_addr learned my addr 172.21.15.60:0/2847694656 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:35:57.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f144b7fe640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f144c197920 0x7f144c197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:57.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f144affd640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f144c0ffa90 0x7f144c19d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:57.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f14427fc640 1 -- 172.21.15.60:0/2847694656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f144c0ffa90 msgr2=0x7f144c19d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:57.309 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f14427fc640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f144c0ffa90 0x7f144c19d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f14427fc640 1 -- 172.21.15.60:0/2847694656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f144c197920 msgr2=0x7f144c197dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:57.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f14427fc640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f144c197920 0x7f144c197dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f14427fc640 1 -- 172.21.15.60:0/2847694656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c198440 con 0x7f144c106650 2024-03-04T12:35:57.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.300+0000 7f144b7fe640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f144c197920 0x7f144c197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:35:57.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.301+0000 7f14427fc640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f144c106650 0x7f144c19dcb0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f143400c970 tx=0x7f143400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:57.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.301+0000 7f1448ff9640 1 -- 172.21.15.60:0/2847694656 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1434014070 con 0x7f144c106650 2024-03-04T12:35:57.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.301+0000 7f1448ff9640 1 -- 172.21.15.60:0/2847694656 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1434004440 con 0x7f144c106650 2024-03-04T12:35:57.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.301+0000 7f1448ff9640 1 -- 172.21.15.60:0/2847694656 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143400f450 con 0x7f144c106650 2024-03-04T12:35:57.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.301+0000 7f1451569640 1 -- 172.21.15.60:0/2847694656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144c198730 con 0x7f144c106650 2024-03-04T12:35:57.311 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.301+0000 7f1451569640 1 -- 172.21.15.60:0/2847694656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144c1a4980 con 0x7f144c106650 2024-03-04T12:35:57.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.302+0000 7f1451569640 1 -- 172.21.15.60:0/2847694656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f144c068600 con 0x7f144c106650 2024-03-04T12:35:57.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.304+0000 7f1448ff9640 1 -- 172.21.15.60:0/2847694656 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f1434017070 con 0x7f144c106650 2024-03-04T12:35:57.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.304+0000 7f1448ff9640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f14380a1ef0 0x7f14380a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:57.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.304+0000 7f144affd640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f14380a1ef0 0x7f14380a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:57.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.304+0000 7f1448ff9640 1 -- 172.21.15.60:0/2847694656 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f14340ce540 con 0x7f144c106650 2024-03-04T12:35:57.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.305+0000 7f144affd640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f14380a1ef0 0x7f14380a43b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f143c036040 tx=0x7f143c005fd0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:57.313 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.308+0000 7f1448ff9640 1 -- 172.21.15.60:0/2847694656 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f143401e030 con 0x7f144c106650 2024-03-04T12:35:57.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.488+0000 7f1451569640 1 -- 172.21.15.60:0/2847694656 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f144c100440 con 0x7f144c106650 2024-03-04T12:35:57.494 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.488+0000 7f1448ff9640 1 -- 172.21.15.60:0/2847694656 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v103) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1434018e90 con 0x7f144c106650 2024-03-04T12:35:57.494 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:35:57.496 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.491+0000 7f1451569640 1 -- 172.21.15.60:0/2847694656 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f14380a1ef0 msgr2=0x7f14380a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.491+0000 7f1451569640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f14380a1ef0 0x7f14380a43b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f143c036040 tx=0x7f143c005fd0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.491+0000 7f1451569640 1 -- 172.21.15.60:0/2847694656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f144c106650 msgr2=0x7f144c19dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.491+0000 7f1451569640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f144c106650 0x7f144c19dcb0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f143400c970 tx=0x7f143400ce30 comp rx=0 tx=0).stop 2024-03-04T12:35:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.491+0000 7f1451569640 1 -- 172.21.15.60:0/2847694656 shutdown_connections 2024-03-04T12:35:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.491+0000 7f1451569640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f144c197920 0x7f144c197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.491+0000 7f1451569640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f14380a1ef0 0x7f14380a43b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.497 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.491+0000 7f1451569640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f144c106650 0x7f144c19dcb0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.491+0000 7f1451569640 1 --2- 172.21.15.60:0/2847694656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f144c0ffa90 0x7f144c19d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.491+0000 7f1451569640 1 -- 172.21.15.60:0/2847694656 >> 172.21.15.60:0/2847694656 conn(0x7f144c076730 msgr2=0x7f144c104650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.491+0000 7f1451569640 1 -- 172.21.15.60:0/2847694656 shutdown_connections 2024-03-04T12:35:57.498 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.491+0000 7f1451569640 1 -- 172.21.15.60:0/2847694656 wait complete. 2024-03-04T12:35:57.507 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-04T12:35:57.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:57 smithi086 ceph-mon[36851]: pgmap v313: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:57 smithi086 ceph-mon[36851]: from='client.16554 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:35:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:57 smithi086 ceph-mon[36851]: from='client.16560 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:35:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:57 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3394917884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:35:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:57 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2099703023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:35:57.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:57 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2516661854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:35:57.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:57 smithi063 ceph-mon[36717]: pgmap v313: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-04T12:35:57.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:57 smithi063 ceph-mon[36717]: from='client.16554 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:35:57.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:57 smithi063 ceph-mon[36717]: from='client.16560 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T12:35:57.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:57 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3394917884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:35:57.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:57 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2099703023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:35:57.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:57 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2516661854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:35:57.612 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.604+0000 7f2737d97640 1 -- 172.21.15.60:0/2030187763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27300ffe00 msgr2=0x7f27301001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:57.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.604+0000 7f2737d97640 1 --2- 172.21.15.60:0/2030187763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27300ffe00 0x7f27301001e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f272c009900 tx=0x7f272c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:35:57.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.604+0000 7f2737d97640 1 -- 172.21.15.60:0/2030187763 shutdown_connections 2024-03-04T12:35:57.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.604+0000 7f2737d97640 1 --2- 172.21.15.60:0/2030187763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2730106740 0x7f273010e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.604+0000 7f2737d97640 1 --2- 172.21.15.60:0/2030187763 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27301007b0 0x7f2730106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.604+0000 7f2737d97640 1 --2- 172.21.15.60:0/2030187763 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f27300ffe00 0x7f27301001e0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.604+0000 7f2737d97640 1 -- 172.21.15.60:0/2030187763 >> 172.21.15.60:0/2030187763 conn(0x7f2730075540 msgr2=0x7f2730075950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:35:57.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.604+0000 7f2737d97640 1 -- 172.21.15.60:0/2030187763 shutdown_connections 2024-03-04T12:35:57.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.604+0000 7f2737d97640 1 -- 172.21.15.60:0/2030187763 wait complete. 2024-03-04T12:35:57.613 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2737d97640 1 Processor -- start 2024-03-04T12:35:57.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2737d97640 1 -- start start 2024-03-04T12:35:57.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2737d97640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27300ffe00 0x7f2730100d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:57.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2737d97640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730106740 0x7f27301012b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:57.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2737d97640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27301017f0 0x7f27301a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:57.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2737d97640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2730111820 con 0x7f2730106740 2024-03-04T12:35:57.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2737d97640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27301116a0 con 0x7f27301017f0 2024-03-04T12:35:57.614 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2737d97640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27301119a0 con 0x7f27300ffe00 2024-03-04T12:35:57.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2735332640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730106740 0x7f27301012b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:57.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2735332640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730106740 0x7f27301012b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:52458/0 (socket says 172.21.15.60:52458) 2024-03-04T12:35:57.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2735332640 1 -- 172.21.15.60:0/1467235893 learned_addr learned my addr 172.21.15.60:0/1467235893 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:35:57.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2735b33640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27300ffe00 0x7f2730100d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:57.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2736334640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27301017f0 0x7f27301a5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:57.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2735332640 1 -- 172.21.15.60:0/1467235893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27300ffe00 msgr2=0x7f2730100d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:57.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2735332640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27300ffe00 0x7f2730100d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2735332640 1 -- 172.21.15.60:0/1467235893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27301017f0 msgr2=0x7f27301a5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:35:57.615 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2735332640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27301017f0 0x7f27301a5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:35:57.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2735332640 1 -- 172.21.15.60:0/1467235893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27301a6590 con 0x7f2730106740 2024-03-04T12:35:57.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.605+0000 7f2736334640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27301017f0 0x7f27301a5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:35:57.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.606+0000 7f2735332640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730106740 0x7f27301012b0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f272000d950 tx=0x7f272000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:57.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.606+0000 7f2726ffd640 1 -- 172.21.15.60:0/1467235893 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2720017070 con 0x7f2730106740 2024-03-04T12:35:57.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.606+0000 7f2726ffd640 1 -- 172.21.15.60:0/1467235893 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2720009c30 con 0x7f2730106740 2024-03-04T12:35:57.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.606+0000 7f2737d97640 1 -- 172.21.15.60:0/1467235893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27301a6880 con 0x7f2730106740 2024-03-04T12:35:57.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.606+0000 7f2726ffd640 1 -- 172.21.15.60:0/1467235893 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27200047a0 con 0x7f2730106740 2024-03-04T12:35:57.616 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.606+0000 7f2737d97640 1 -- 172.21.15.60:0/1467235893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27301a6e20 con 0x7f2730106740 2024-03-04T12:35:57.617 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.607+0000 7f2737d97640 1 -- 172.21.15.60:0/1467235893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27300695a0 con 0x7f2730106740 2024-03-04T12:35:57.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.608+0000 7f2726ffd640 1 -- 172.21.15.60:0/1467235893 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f2720004020 con 0x7f2730106740 2024-03-04T12:35:57.618 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.608+0000 7f2726ffd640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f271c0a1fa0 0x7f271c0a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:35:57.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.609+0000 7f2735b33640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f271c0a1fa0 0x7f271c0a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:35:57.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.609+0000 7f2726ffd640 1 -- 172.21.15.60:0/1467235893 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f27200ce4f0 con 0x7f2730106740 2024-03-04T12:35:57.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.609+0000 7f2735b33640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f271c0a1fa0 0x7f271c0a4460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f272c006010 tx=0x7f272c005dd0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:35:57.619 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.612+0000 7f2726ffd640 1 -- 172.21.15.60:0/1467235893 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2720020020 con 0x7f2730106740 2024-03-04T12:35:57.850 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.844+0000 7f2737d97640 1 -- 172.21.15.60:0/1467235893 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f2730109970 con 0x7f2730106740 2024-03-04T12:35:57.858 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.852+0000 7f2726ffd640 1 -- 172.21.15.60:0/1467235893 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) v1 ==== 81+0+736922 (secure 0 0 0) 0x7f272009b300 con 0x7f2730106740 2024-03-04T12:35:57.868 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:35:57.868 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:35:57.869 INFO:teuthology.orchestra.run.smithi060.stdout: "epoch": 27, 2024-03-04T12:35:57.869 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.869 INFO:teuthology.orchestra.run.smithi060.stdout: "active_gid": 15735, 2024-03-04T12:35:57.869 INFO:teuthology.orchestra.run.smithi060.stdout: "active_name": "x", 2024-03-04T12:35:57.869 INFO:teuthology.orchestra.run.smithi060.stdout: "active_addrs": { 2024-03-04T12:35:57.869 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-04T12:35:57.869 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:57.869 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-04T12:35:57.869 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6800", 2024-03-04T12:35:57.870 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 248765372 2024-03-04T12:35:57.870 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.870 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:57.870 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v1", 2024-03-04T12:35:57.870 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:6801", 2024-03-04T12:35:57.870 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 248765372 2024-03-04T12:35:57.870 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:57.870 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-04T12:35:57.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.871 INFO:teuthology.orchestra.run.smithi060.stdout: "active_addr": "172.21.15.60:6801/248765372", 2024-03-04T12:35:57.871 INFO:teuthology.orchestra.run.smithi060.stdout: "active_change": "2024-03-04T12:25:53.138758+0000", 2024-03-04T12:35:57.871 INFO:teuthology.orchestra.run.smithi060.stdout: "active_mgr_features": 4540701547738038271, 2024-03-04T12:35:57.871 INFO:teuthology.orchestra.run.smithi060.stdout: "available": true, 2024-03-04T12:35:57.871 INFO:teuthology.orchestra.run.smithi060.stdout: "standbys": [ 2024-03-04T12:35:57.871 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:57.871 INFO:teuthology.orchestra.run.smithi060.stdout: "gid": 25283, 2024-03-04T12:35:57.871 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "y", 2024-03-04T12:35:57.872 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_features": 4540701547738038271, 2024-03-04T12:35:57.872 INFO:teuthology.orchestra.run.smithi060.stdout: "available_modules": [ 2024-03-04T12:35:57.872 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:57.872 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "alerts", 2024-03-04T12:35:57.872 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:57.872 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:57.872 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:57.873 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:57.873 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:57.873 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:57.873 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.873 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.873 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:57.873 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.873 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.874 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.874 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "How frequently to reexamine health status", 2024-03-04T12:35:57.874 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.874 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.874 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.874 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:57.874 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:57.875 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.875 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.875 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.875 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:57.875 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.875 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.875 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:57.875 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:57.875 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:57.876 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:57.876 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:57.876 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:57.876 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:57.876 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:57.876 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:57.876 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.876 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.876 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.877 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:57.877 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:57.877 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.877 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.877 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.877 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:57.878 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.878 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.878 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.878 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:57.878 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.878 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.878 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.878 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.878 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:57.879 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:57.879 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.879 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.879 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.879 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:57.879 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.879 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.879 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:57.879 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:57.880 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:57.880 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:57.880 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:57.880 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:57.880 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:57.880 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:57.880 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:57.881 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.881 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.881 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.881 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:57.881 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:57.881 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.881 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.881 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.882 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:57.882 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.882 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.882 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.882 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:57.882 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.882 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.882 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.883 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_destination": { 2024-03-04T12:35:57.883 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_destination", 2024-03-04T12:35:57.883 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.883 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.883 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.883 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:57.883 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.883 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.883 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.884 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Email address to send alerts to", 2024-03-04T12:35:57.884 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.884 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.884 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.884 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_from_name": { 2024-03-04T12:35:57.884 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_from_name", 2024-03-04T12:35:57.884 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.885 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.885 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.885 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "Ceph", 2024-03-04T12:35:57.885 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.885 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.885 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.885 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Email From: name", 2024-03-04T12:35:57.885 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.885 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.886 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.886 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_host": { 2024-03-04T12:35:57.886 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_host", 2024-03-04T12:35:57.886 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.886 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.886 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.886 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:57.886 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.887 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.887 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.887 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "SMTP server", 2024-03-04T12:35:57.887 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.887 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.887 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.887 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.888 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_password": { 2024-03-04T12:35:57.888 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_password", 2024-03-04T12:35:57.888 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.888 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.888 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.888 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:57.888 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.888 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.889 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.889 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Password to authenticate with", 2024-03-04T12:35:57.889 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.889 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.889 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.889 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_port": { 2024-03-04T12:35:57.889 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_port", 2024-03-04T12:35:57.889 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:57.890 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.890 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.890 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "465", 2024-03-04T12:35:57.890 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.890 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.890 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.890 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "SMTP port", 2024-03-04T12:35:57.890 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.891 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.891 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.891 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_sender": { 2024-03-04T12:35:57.891 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_sender", 2024-03-04T12:35:57.891 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.891 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.891 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.891 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:57.892 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.892 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.892 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.892 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "SMTP envelope sender", 2024-03-04T12:35:57.892 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.892 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.892 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.893 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_ssl": { 2024-03-04T12:35:57.893 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_ssl", 2024-03-04T12:35:57.893 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.893 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.893 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.893 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:57.893 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.894 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.894 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.894 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-04T12:35:57.894 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.894 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.894 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.894 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_user": { 2024-03-04T12:35:57.894 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_user", 2024-03-04T12:35:57.895 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.895 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.895 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.895 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:57.895 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.895 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.895 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.895 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "User to authenticate as", 2024-03-04T12:35:57.895 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.896 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.896 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.896 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:57.896 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:57.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:57.896 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "balancer", 2024-03-04T12:35:57.896 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: "active": { 2024-03-04T12:35:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "active", 2024-03-04T12:35:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.898 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.898 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.898 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "automatically balance PGs across cluster", 2024-03-04T12:35:57.898 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.898 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.898 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.898 INFO:teuthology.orchestra.run.smithi060.stdout: "begin_time": { 2024-03-04T12:35:57.899 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "begin_time", 2024-03-04T12:35:57.899 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.899 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.899 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.899 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0000", 2024-03-04T12:35:57.899 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.899 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.899 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.899 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "beginning time of day to automatically balance", 2024-03-04T12:35:57.900 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-04T12:35:57.900 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.900 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.900 INFO:teuthology.orchestra.run.smithi060.stdout: "begin_weekday": { 2024-03-04T12:35:57.900 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "begin_weekday", 2024-03-04T12:35:57.900 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:57.900 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.901 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.901 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:57.901 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "0", 2024-03-04T12:35:57.901 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "6", 2024-03-04T12:35:57.901 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.901 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-04T12:35:57.901 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-04T12:35:57.901 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.901 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.902 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_compat_max_iterations": { 2024-03-04T12:35:57.902 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crush_compat_max_iterations", 2024-03-04T12:35:57.902 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:57.902 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.902 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.902 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "25", 2024-03-04T12:35:57.902 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1", 2024-03-04T12:35:57.902 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "250", 2024-03-04T12:35:57.903 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.903 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-04T12:35:57.903 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.903 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.903 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.903 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_compat_metrics": { 2024-03-04T12:35:57.903 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crush_compat_metrics", 2024-03-04T12:35:57.903 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.904 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.904 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.904 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "pgs,objects,bytes", 2024-03-04T12:35:57.904 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.904 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.904 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.904 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-04T12:35:57.905 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-04T12:35:57.905 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.905 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.905 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.905 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_compat_step": { 2024-03-04T12:35:57.905 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crush_compat_step", 2024-03-04T12:35:57.906 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:57.906 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.906 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.906 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0.5", 2024-03-04T12:35:57.906 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "0.001", 2024-03-04T12:35:57.906 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "0.999", 2024-03-04T12:35:57.906 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.906 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "aggressiveness of optimization", 2024-03-04T12:35:57.906 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-04T12:35:57.906 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.907 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.907 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.907 INFO:teuthology.orchestra.run.smithi060.stdout: "end_time": { 2024-03-04T12:35:57.907 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "end_time", 2024-03-04T12:35:57.907 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.907 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.907 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.907 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "2359", 2024-03-04T12:35:57.907 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.908 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.908 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.908 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "ending time of day to automatically balance", 2024-03-04T12:35:57.908 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-04T12:35:57.908 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.908 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.908 INFO:teuthology.orchestra.run.smithi060.stdout: "end_weekday": { 2024-03-04T12:35:57.909 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "end_weekday", 2024-03-04T12:35:57.909 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:57.909 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.909 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.909 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:57.909 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "0", 2024-03-04T12:35:57.909 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "6", 2024-03-04T12:35:57.909 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.909 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-04T12:35:57.909 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-04T12:35:57.910 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.910 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.910 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:57.910 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:57.910 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.910 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.910 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.911 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:57.911 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.911 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.911 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:57.911 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:57.911 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:57.911 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:57.911 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:57.911 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:57.912 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:57.912 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:57.912 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:57.912 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.912 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.912 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.912 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:57.913 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:57.913 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.913 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.913 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.913 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:57.913 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.913 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.913 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.914 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:57.914 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.914 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.914 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.914 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:57.914 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:57.914 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.914 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.915 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.915 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:57.915 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.915 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.915 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:57.915 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:57.915 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:57.915 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:57.915 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:57.916 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:57.916 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:57.916 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:57.916 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:57.916 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.916 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.916 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.916 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.917 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:57.917 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:57.917 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.917 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.917 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.917 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:57.917 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.917 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.917 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.918 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:57.918 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.918 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.918 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.918 INFO:teuthology.orchestra.run.smithi060.stdout: "min_score": { 2024-03-04T12:35:57.918 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "min_score", 2024-03-04T12:35:57.918 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:57.919 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.919 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.919 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:57.919 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.919 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.919 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.919 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-04T12:35:57.919 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.919 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.919 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.920 INFO:teuthology.orchestra.run.smithi060.stdout: "mode": { 2024-03-04T12:35:57.920 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mode", 2024-03-04T12:35:57.920 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.920 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.920 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.920 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "upmap", 2024-03-04T12:35:57.920 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.921 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.921 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:57.921 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-compat", 2024-03-04T12:35:57.921 INFO:teuthology.orchestra.run.smithi060.stdout: "none", 2024-03-04T12:35:57.921 INFO:teuthology.orchestra.run.smithi060.stdout: "read", 2024-03-04T12:35:57.921 INFO:teuthology.orchestra.run.smithi060.stdout: "upmap", 2024-03-04T12:35:57.921 INFO:teuthology.orchestra.run.smithi060.stdout: "upmap-read" 2024-03-04T12:35:57.921 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:57.921 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Balancer mode", 2024-03-04T12:35:57.922 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.922 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.922 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.922 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_ids": { 2024-03-04T12:35:57.922 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "pool_ids", 2024-03-04T12:35:57.922 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.922 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.922 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.923 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:57.923 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.923 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.923 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.923 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-04T12:35:57.923 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.923 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.923 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.925 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.925 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-04T12:35:57.925 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.925 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.925 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.925 INFO:teuthology.orchestra.run.smithi060.stdout: "upmap_max_deviation": { 2024-03-04T12:35:57.925 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "upmap_max_deviation", 2024-03-04T12:35:57.925 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:57.926 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.926 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.926 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5", 2024-03-04T12:35:57.926 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1", 2024-03-04T12:35:57.926 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.926 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.926 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-04T12:35:57.926 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-04T12:35:57.926 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.927 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.927 INFO:teuthology.orchestra.run.smithi060.stdout: "upmap_max_optimizations": { 2024-03-04T12:35:57.927 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "upmap_max_optimizations", 2024-03-04T12:35:57.927 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:57.927 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.927 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.927 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:57.927 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.928 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.928 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.928 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-04T12:35:57.928 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.928 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.928 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.928 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:57.928 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:57.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.929 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:57.929 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephadm", 2024-03-04T12:35:57.929 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:57.929 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:57.929 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:57.929 INFO:teuthology.orchestra.run.smithi060.stdout: "agent_down_multiplier": { 2024-03-04T12:35:57.929 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "agent_down_multiplier", 2024-03-04T12:35:57.929 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:57.930 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.930 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.930 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "3.0", 2024-03-04T12:35:57.930 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.930 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.930 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.930 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-04T12:35:57.930 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.930 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.931 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.931 INFO:teuthology.orchestra.run.smithi060.stdout: "agent_refresh_rate": { 2024-03-04T12:35:57.931 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "agent_refresh_rate", 2024-03-04T12:35:57.931 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:57.931 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.931 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.931 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "20", 2024-03-04T12:35:57.931 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.932 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.932 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.932 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-04T12:35:57.932 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.932 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.932 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.932 INFO:teuthology.orchestra.run.smithi060.stdout: "agent_starting_port": { 2024-03-04T12:35:57.933 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "agent_starting_port", 2024-03-04T12:35:57.933 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:57.933 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.933 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.933 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "4721", 2024-03-04T12:35:57.933 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.933 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.933 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.934 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-04T12:35:57.934 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.934 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.934 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.934 INFO:teuthology.orchestra.run.smithi060.stdout: "allow_ptrace": { 2024-03-04T12:35:57.934 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "allow_ptrace", 2024-03-04T12:35:57.934 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.935 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.935 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.935 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:57.935 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.935 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.935 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.935 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-04T12:35:57.935 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-04T12:35:57.935 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.936 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.936 INFO:teuthology.orchestra.run.smithi060.stdout: "autotune_interval": { 2024-03-04T12:35:57.936 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "autotune_interval", 2024-03-04T12:35:57.936 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:57.936 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.936 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.936 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:57.937 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.937 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.937 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.937 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-04T12:35:57.937 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.937 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.937 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.937 INFO:teuthology.orchestra.run.smithi060.stdout: "autotune_memory_target_ratio": { 2024-03-04T12:35:57.937 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "autotune_memory_target_ratio", 2024-03-04T12:35:57.938 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:57.938 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.938 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.938 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0.7", 2024-03-04T12:35:57.938 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.938 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.938 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.938 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-04T12:35:57.939 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.939 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.939 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.939 INFO:teuthology.orchestra.run.smithi060.stdout: "cephadm_log_destination": { 2024-03-04T12:35:57.939 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephadm_log_destination", 2024-03-04T12:35:57.939 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.939 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.939 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.940 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:57.940 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.940 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.940 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:57.940 INFO:teuthology.orchestra.run.smithi060.stdout: "file", 2024-03-04T12:35:57.940 INFO:teuthology.orchestra.run.smithi060.stdout: "file,syslog", 2024-03-04T12:35:57.940 INFO:teuthology.orchestra.run.smithi060.stdout: "syslog" 2024-03-04T12:35:57.940 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:57.941 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-04T12:35:57.941 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.941 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.941 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.941 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.941 INFO:teuthology.orchestra.run.smithi060.stdout: "cgroups_split": { 2024-03-04T12:35:57.941 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cgroups_split", 2024-03-04T12:35:57.941 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.942 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.942 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.942 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:57.942 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.942 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.942 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.942 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-04T12:35:57.942 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.942 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.942 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.943 INFO:teuthology.orchestra.run.smithi060.stdout: "config_checks_enabled": { 2024-03-04T12:35:57.943 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "config_checks_enabled", 2024-03-04T12:35:57.943 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.943 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.943 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.943 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:57.943 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.944 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.944 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.944 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-04T12:35:57.944 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.944 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.944 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.944 INFO:teuthology.orchestra.run.smithi060.stdout: "config_dashboard": { 2024-03-04T12:35:57.944 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "config_dashboard", 2024-03-04T12:35:57.945 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.945 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.945 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.945 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:57.945 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.945 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.945 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.945 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-04T12:35:57.945 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.946 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.946 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.946 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_alertmanager": { 2024-03-04T12:35:57.946 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_alertmanager", 2024-03-04T12:35:57.946 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.946 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.946 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.946 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-04T12:35:57.947 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.947 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.947 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.947 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Prometheus container image", 2024-03-04T12:35:57.947 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.947 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.947 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.947 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.947 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_base": { 2024-03-04T12:35:57.948 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_base", 2024-03-04T12:35:57.948 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.948 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.948 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.948 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-04T12:35:57.948 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.948 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.948 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.949 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Container image name, without the tag", 2024-03-04T12:35:57.949 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.949 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.949 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.949 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.949 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_elasticsearch": { 2024-03-04T12:35:57.949 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_elasticsearch", 2024-03-04T12:35:57.949 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.949 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.950 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.950 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-04T12:35:57.950 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.950 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.950 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.950 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "elasticsearch container image", 2024-03-04T12:35:57.950 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.950 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.950 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.951 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_grafana": { 2024-03-04T12:35:57.951 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_grafana", 2024-03-04T12:35:57.951 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.951 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.951 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.951 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-04T12:35:57.951 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.951 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.952 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.952 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Prometheus container image", 2024-03-04T12:35:57.952 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.952 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.952 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.952 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_haproxy": { 2024-03-04T12:35:57.952 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_haproxy", 2024-03-04T12:35:57.953 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.953 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.953 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.953 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-04T12:35:57.953 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.953 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.953 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.953 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "HAproxy container image", 2024-03-04T12:35:57.954 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.954 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.954 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.954 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_jaeger_agent": { 2024-03-04T12:35:57.954 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_jaeger_agent", 2024-03-04T12:35:57.954 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.954 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.954 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.955 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-04T12:35:57.955 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.955 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.955 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.955 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Jaeger agent container image", 2024-03-04T12:35:57.955 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.955 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.955 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.956 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_jaeger_collector": { 2024-03-04T12:35:57.956 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_jaeger_collector", 2024-03-04T12:35:57.956 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.956 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.956 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.956 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-04T12:35:57.956 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.956 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.957 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.957 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Jaeger collector container image", 2024-03-04T12:35:57.957 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.957 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.957 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.957 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_jaeger_query": { 2024-03-04T12:35:57.957 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_jaeger_query", 2024-03-04T12:35:57.957 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.958 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.958 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.958 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-04T12:35:57.958 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.958 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.958 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.958 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Jaeger query container image", 2024-03-04T12:35:57.958 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.958 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.958 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.959 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_keepalived": { 2024-03-04T12:35:57.959 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_keepalived", 2024-03-04T12:35:57.959 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.959 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.959 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.959 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-04T12:35:57.960 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.960 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.960 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.960 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Keepalived container image", 2024-03-04T12:35:57.963 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.963 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.964 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.964 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_loki": { 2024-03-04T12:35:57.964 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_loki", 2024-03-04T12:35:57.964 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.964 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.964 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.964 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-04T12:35:57.965 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.965 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.965 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.965 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Loki container image", 2024-03-04T12:35:57.965 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.965 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.965 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.965 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_node_exporter": { 2024-03-04T12:35:57.966 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_node_exporter", 2024-03-04T12:35:57.966 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.966 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.966 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.966 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-04T12:35:57.966 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.966 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.966 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.967 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Prometheus container image", 2024-03-04T12:35:57.967 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.967 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.967 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.967 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_nvmeof": { 2024-03-04T12:35:57.967 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_nvmeof", 2024-03-04T12:35:57.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.967 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.968 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.968 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-04T12:35:57.968 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.968 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.968 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.968 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Nvme-of container image", 2024-03-04T12:35:57.968 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.968 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.968 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.969 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_prometheus": { 2024-03-04T12:35:57.969 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_prometheus", 2024-03-04T12:35:57.969 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.969 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.969 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.969 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-04T12:35:57.969 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.970 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.970 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.970 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Prometheus container image", 2024-03-04T12:35:57.970 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.970 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.970 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.970 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_promtail": { 2024-03-04T12:35:57.970 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_promtail", 2024-03-04T12:35:57.971 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.971 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.971 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.971 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-04T12:35:57.971 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.971 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.971 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.971 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Promtail container image", 2024-03-04T12:35:57.972 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.972 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.972 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.972 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_snmp_gateway": { 2024-03-04T12:35:57.972 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_snmp_gateway", 2024-03-04T12:35:57.972 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.972 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.972 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.973 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-04T12:35:57.973 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.973 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.973 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.973 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "SNMP Gateway container image", 2024-03-04T12:35:57.973 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.973 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.973 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.973 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.974 INFO:teuthology.orchestra.run.smithi060.stdout: "container_init": { 2024-03-04T12:35:57.974 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_init", 2024-03-04T12:35:57.974 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.974 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.974 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.974 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:57.974 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.974 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.975 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.975 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Run podman/docker with `--init`", 2024-03-04T12:35:57.975 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.975 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.975 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.975 INFO:teuthology.orchestra.run.smithi060.stdout: "daemon_cache_timeout": { 2024-03-04T12:35:57.975 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "daemon_cache_timeout", 2024-03-04T12:35:57.976 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:57.976 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.976 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.976 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:57.976 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.976 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.976 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.976 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-04T12:35:57.977 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.977 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.977 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.977 INFO:teuthology.orchestra.run.smithi060.stdout: "default_cephadm_command_timeout": { 2024-03-04T12:35:57.977 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default_cephadm_command_timeout", 2024-03-04T12:35:57.977 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:57.977 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.977 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.978 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "900", 2024-03-04T12:35:57.978 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.978 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.978 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.978 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-04T12:35:57.978 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.978 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.978 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.979 INFO:teuthology.orchestra.run.smithi060.stdout: "default_registry": { 2024-03-04T12:35:57.979 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default_registry", 2024-03-04T12:35:57.979 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.979 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.979 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.979 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "docker.io", 2024-03-04T12:35:57.979 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.979 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.980 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.980 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-04T12:35:57.980 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.980 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.980 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.980 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.980 INFO:teuthology.orchestra.run.smithi060.stdout: "device_cache_timeout": { 2024-03-04T12:35:57.980 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "device_cache_timeout", 2024-03-04T12:35:57.981 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:57.981 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.981 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.981 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "1800", 2024-03-04T12:35:57.981 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.981 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.981 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.981 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "seconds to cache device inventory", 2024-03-04T12:35:57.982 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.982 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.982 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.982 INFO:teuthology.orchestra.run.smithi060.stdout: "device_enhanced_scan": { 2024-03-04T12:35:57.982 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "device_enhanced_scan", 2024-03-04T12:35:57.982 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.982 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.982 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.983 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:57.983 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.983 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.983 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.983 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-04T12:35:57.983 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.983 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.983 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.984 INFO:teuthology.orchestra.run.smithi060.stdout: "facts_cache_timeout": { 2024-03-04T12:35:57.984 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "facts_cache_timeout", 2024-03-04T12:35:57.984 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:57.984 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.984 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.984 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:57.984 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.985 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.985 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.985 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "seconds to cache host facts data", 2024-03-04T12:35:57.985 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.985 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.985 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.985 INFO:teuthology.orchestra.run.smithi060.stdout: "host_check_interval": { 2024-03-04T12:35:57.985 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "host_check_interval", 2024-03-04T12:35:57.986 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:57.986 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.986 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.986 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:57.986 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.986 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.986 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.986 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to perform a host check", 2024-03-04T12:35:57.986 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.987 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.987 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.987 INFO:teuthology.orchestra.run.smithi060.stdout: "hw_monitoring": { 2024-03-04T12:35:57.987 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "hw_monitoring", 2024-03-04T12:35:57.987 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.987 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.987 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.988 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:57.988 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.988 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.988 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.988 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-04T12:35:57.988 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.988 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.989 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.989 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.989 INFO:teuthology.orchestra.run.smithi060.stdout: "inventory_list_all": { 2024-03-04T12:35:57.989 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "inventory_list_all", 2024-03-04T12:35:57.989 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.989 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.989 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.989 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:57.989 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.990 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.990 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.990 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-04T12:35:57.990 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.990 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.990 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.990 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:57.990 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:57.991 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.991 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.991 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.991 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:57.991 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.991 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.991 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:57.991 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:57.991 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:57.992 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:57.992 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:57.992 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:57.992 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:57.992 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:57.992 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:57.992 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.992 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.992 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.993 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.993 INFO:teuthology.orchestra.run.smithi060.stdout: "log_refresh_metadata": { 2024-03-04T12:35:57.993 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_refresh_metadata", 2024-03-04T12:35:57.993 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.993 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.993 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.993 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:57.993 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.994 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.994 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.994 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-04T12:35:57.994 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.994 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.994 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.994 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:57.994 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:57.994 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.995 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.995 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:57.995 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:57.995 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.995 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.995 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:57.995 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-04T12:35:57.996 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.996 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.996 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.996 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:57.996 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:57.996 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:57.996 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.997 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.997 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:57.997 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.997 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:57.997 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:57.997 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:57.997 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:57.997 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:57.998 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:57.998 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:57.998 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:57.998 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:57.998 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:57.998 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:57.998 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:57.998 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:57.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:57.999 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:57.999 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:57.999 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:57.999 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:57.999 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:57.999 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:57.999 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:57.999 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.000 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.000 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.000 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.000 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.000 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.000 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.000 INFO:teuthology.orchestra.run.smithi060.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-04T12:35:58.000 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-04T12:35:58.001 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.001 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.001 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.001 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.001 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.001 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.001 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.001 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-04T12:35:58.002 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.002 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.002 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.002 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.002 INFO:teuthology.orchestra.run.smithi060.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-04T12:35:58.002 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-04T12:35:58.002 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.002 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.002 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.003 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "*", 2024-03-04T12:35:58.003 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.003 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.003 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.003 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-04T12:35:58.003 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.003 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.003 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.004 INFO:teuthology.orchestra.run.smithi060.stdout: "max_count_per_host": { 2024-03-04T12:35:58.004 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_count_per_host", 2024-03-04T12:35:58.004 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.004 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.004 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.004 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.004 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.004 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.005 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.005 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "max number of daemons per service per host", 2024-03-04T12:35:58.005 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.005 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.005 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.005 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.005 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd_draining_count": { 2024-03-04T12:35:58.005 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_osd_draining_count", 2024-03-04T12:35:58.006 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.006 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.006 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.006 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.006 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.006 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.006 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.006 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-04T12:35:58.007 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.007 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.007 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.007 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.007 INFO:teuthology.orchestra.run.smithi060.stdout: "migration_current": { 2024-03-04T12:35:58.007 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "migration_current", 2024-03-04T12:35:58.007 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.007 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.008 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.008 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.008 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.008 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.008 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.008 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "internal - do not modify", 2024-03-04T12:35:58.008 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.008 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.008 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.009 INFO:teuthology.orchestra.run.smithi060.stdout: "mode": { 2024-03-04T12:35:58.009 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mode", 2024-03-04T12:35:58.009 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.009 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.009 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.009 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "root", 2024-03-04T12:35:58.009 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.009 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.010 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.010 INFO:teuthology.orchestra.run.smithi060.stdout: "cephadm-package", 2024-03-04T12:35:58.010 INFO:teuthology.orchestra.run.smithi060.stdout: "root" 2024-03-04T12:35:58.010 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.010 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "mode for remote execution of cephadm", 2024-03-04T12:35:58.010 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.010 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.010 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.011 INFO:teuthology.orchestra.run.smithi060.stdout: "oob_default_addr": { 2024-03-04T12:35:58.011 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "oob_default_addr", 2024-03-04T12:35:58.011 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.011 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.011 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.011 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "169.254.1.1", 2024-03-04T12:35:58.011 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.012 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.012 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.012 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-04T12:35:58.012 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.012 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.012 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.012 INFO:teuthology.orchestra.run.smithi060.stdout: "prometheus_alerts_path": { 2024-03-04T12:35:58.013 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "prometheus_alerts_path", 2024-03-04T12:35:58.013 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.013 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.013 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.013 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-04T12:35:58.013 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.013 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.013 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.014 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-04T12:35:58.014 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.014 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.014 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.014 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.014 INFO:teuthology.orchestra.run.smithi060.stdout: "registry_insecure": { 2024-03-04T12:35:58.014 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "registry_insecure", 2024-03-04T12:35:58.014 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.014 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.015 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.015 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.015 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.015 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.015 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.015 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-04T12:35:58.015 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.015 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.016 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.016 INFO:teuthology.orchestra.run.smithi060.stdout: "registry_password": { 2024-03-04T12:35:58.016 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "registry_password", 2024-03-04T12:35:58.016 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.016 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.016 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.016 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.016 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.017 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.017 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.017 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-04T12:35:58.017 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.017 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.017 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.017 INFO:teuthology.orchestra.run.smithi060.stdout: "registry_url": { 2024-03-04T12:35:58.018 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "registry_url", 2024-03-04T12:35:58.018 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.018 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.018 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.018 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.018 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.018 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.018 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.019 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-04T12:35:58.019 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.019 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.019 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.019 INFO:teuthology.orchestra.run.smithi060.stdout: "registry_username": { 2024-03-04T12:35:58.020 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "registry_username", 2024-03-04T12:35:58.020 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.020 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.020 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.020 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.020 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.020 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.020 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.021 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-04T12:35:58.021 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.021 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.021 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.021 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.021 INFO:teuthology.orchestra.run.smithi060.stdout: "secure_monitoring_stack": { 2024-03-04T12:35:58.021 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "secure_monitoring_stack", 2024-03-04T12:35:58.021 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.021 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.022 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.022 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.022 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.022 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.022 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.022 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-04T12:35:58.022 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.022 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.022 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.023 INFO:teuthology.orchestra.run.smithi060.stdout: "service_discovery_port": { 2024-03-04T12:35:58.023 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "service_discovery_port", 2024-03-04T12:35:58.023 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.023 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.023 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.023 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8765", 2024-03-04T12:35:58.023 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.023 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.024 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.024 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "cephadm service discovery port", 2024-03-04T12:35:58.024 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.024 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.024 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.024 INFO:teuthology.orchestra.run.smithi060.stdout: "ssh_config_file": { 2024-03-04T12:35:58.025 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ssh_config_file", 2024-03-04T12:35:58.025 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.025 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.025 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.025 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.025 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.025 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.025 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.025 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-04T12:35:58.026 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.026 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.026 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.026 INFO:teuthology.orchestra.run.smithi060.stdout: "use_agent": { 2024-03-04T12:35:58.026 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "use_agent", 2024-03-04T12:35:58.026 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.026 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.027 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.027 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.027 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.027 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.027 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.027 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-04T12:35:58.027 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.027 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.027 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.028 INFO:teuthology.orchestra.run.smithi060.stdout: "use_repo_digest": { 2024-03-04T12:35:58.028 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "use_repo_digest", 2024-03-04T12:35:58.028 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.028 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.028 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.028 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.028 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.029 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.029 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.029 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-04T12:35:58.029 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.029 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.029 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.029 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.029 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_on_failed_host_check": { 2024-03-04T12:35:58.029 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_on_failed_host_check", 2024-03-04T12:35:58.030 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.030 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.030 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.030 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.030 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.030 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.030 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.030 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "raise a health warning if the host check fails", 2024-03-04T12:35:58.030 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.031 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.031 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.031 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_on_stray_daemons": { 2024-03-04T12:35:58.031 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_on_stray_daemons", 2024-03-04T12:35:58.031 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.031 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.031 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.032 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.032 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.032 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.032 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.032 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-04T12:35:58.032 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.032 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.032 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.033 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_on_stray_hosts": { 2024-03-04T12:35:58.033 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_on_stray_hosts", 2024-03-04T12:35:58.033 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.033 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.033 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.033 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.033 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.034 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.034 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.034 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-04T12:35:58.034 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.034 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.034 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.034 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.034 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.035 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.035 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crash", 2024-03-04T12:35:58.035 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.035 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.035 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.035 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.035 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.036 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.036 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.036 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.036 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.036 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.036 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.036 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.036 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.037 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.037 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.037 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.037 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.037 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.037 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.037 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.037 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.038 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.038 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.038 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.038 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.038 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.038 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.038 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.039 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.039 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.039 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.039 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.039 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.039 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.040 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.040 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.040 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.040 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.040 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.040 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.040 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.041 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.041 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.041 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.041 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.041 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.041 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.041 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.041 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.042 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.042 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.042 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.042 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.042 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.042 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.042 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.042 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.043 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.043 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.043 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.043 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.043 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.043 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.043 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.043 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.044 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.044 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.044 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.044 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.044 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.044 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.044 INFO:teuthology.orchestra.run.smithi060.stdout: "retain_interval": { 2024-03-04T12:35:58.044 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "retain_interval", 2024-03-04T12:35:58.045 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.045 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.045 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.045 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "31536000", 2024-03-04T12:35:58.045 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.045 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.045 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.045 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-04T12:35:58.045 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.046 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.046 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.046 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_recent_interval": { 2024-03-04T12:35:58.046 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_recent_interval", 2024-03-04T12:35:58.046 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.046 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.046 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.047 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "1209600", 2024-03-04T12:35:58.047 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.047 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.047 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.047 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-04T12:35:58.047 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.047 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.047 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.048 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.048 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.048 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.048 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "dashboard", 2024-03-04T12:35:58.048 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.048 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.048 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.048 INFO:teuthology.orchestra.run.smithi060.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-04T12:35:58.049 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-04T12:35:58.049 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.049 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.049 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.049 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.049 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.049 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.049 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.050 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.050 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.050 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.050 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.050 INFO:teuthology.orchestra.run.smithi060.stdout: "ALERTMANAGER_API_HOST": { 2024-03-04T12:35:58.050 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-04T12:35:58.050 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.051 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.051 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.051 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.051 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.051 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.051 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.051 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.052 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.052 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.052 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.052 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.052 INFO:teuthology.orchestra.run.smithi060.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-04T12:35:58.052 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-04T12:35:58.052 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.052 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.053 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.053 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.053 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.053 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.053 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.053 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.053 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.053 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.053 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.054 INFO:teuthology.orchestra.run.smithi060.stdout: "AUDIT_API_ENABLED": { 2024-03-04T12:35:58.054 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "AUDIT_API_ENABLED", 2024-03-04T12:35:58.054 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.054 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.054 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.054 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.054 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.055 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.055 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.055 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.055 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.055 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.055 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.055 INFO:teuthology.orchestra.run.smithi060.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-04T12:35:58.055 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-04T12:35:58.056 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.056 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.056 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.056 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.056 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.056 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.056 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.056 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.056 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.057 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.057 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.057 INFO:teuthology.orchestra.run.smithi060.stdout: "ENABLE_BROWSABLE_API": { 2024-03-04T12:35:58.057 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-04T12:35:58.057 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.057 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.057 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.057 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.058 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.058 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.058 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.058 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.058 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.058 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.058 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.058 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-04T12:35:58.059 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-04T12:35:58.059 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.059 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.059 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.059 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.059 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.059 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.059 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.059 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.060 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.060 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.060 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.060 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-04T12:35:58.060 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-04T12:35:58.060 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.060 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.061 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.061 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.061 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.061 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.061 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.061 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.061 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.061 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.061 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.062 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-04T12:35:58.062 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-04T12:35:58.062 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.062 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.062 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.062 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.062 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.063 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.063 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.063 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.063 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.063 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.063 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.063 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-04T12:35:58.064 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-04T12:35:58.064 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.064 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.064 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.064 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.064 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.064 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.064 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.065 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.065 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.065 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.065 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.065 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-04T12:35:58.065 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-04T12:35:58.065 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.066 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.066 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.066 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.066 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.066 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.066 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.066 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.066 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.067 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.067 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.067 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-04T12:35:58.067 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-04T12:35:58.067 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.067 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.067 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.068 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.068 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.068 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.068 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.068 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.068 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.068 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.069 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.069 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-04T12:35:58.069 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-04T12:35:58.069 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.069 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.069 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.069 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.069 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.070 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.070 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.070 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.070 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.070 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.070 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.070 INFO:teuthology.orchestra.run.smithi060.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-04T12:35:58.071 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-04T12:35:58.071 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.071 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.071 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.071 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.071 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.071 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.071 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.071 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.072 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.072 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.072 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.072 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_API_PASSWORD": { 2024-03-04T12:35:58.072 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-04T12:35:58.072 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.073 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.073 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.073 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "admin", 2024-03-04T12:35:58.073 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.073 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.073 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.073 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.073 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.073 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.074 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.074 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.074 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-04T12:35:58.074 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-04T12:35:58.074 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.074 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.074 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.074 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.074 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.075 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.075 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.075 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.075 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.075 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.075 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.075 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_API_URL": { 2024-03-04T12:35:58.076 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_API_URL", 2024-03-04T12:35:58.076 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.076 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.076 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.076 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.076 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.076 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.076 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.077 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.077 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.077 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.077 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.077 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_API_USERNAME": { 2024-03-04T12:35:58.077 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-04T12:35:58.077 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.077 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.078 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.078 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "admin", 2024-03-04T12:35:58.078 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.078 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.078 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.078 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.078 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.078 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.079 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.079 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-04T12:35:58.079 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-04T12:35:58.079 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.079 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.079 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.079 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.080 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.080 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.080 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.080 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.080 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.080 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.080 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.081 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-04T12:35:58.081 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-04T12:35:58.081 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.081 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.081 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.081 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.081 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.081 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.081 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.082 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.082 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.082 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.082 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.082 INFO:teuthology.orchestra.run.smithi060.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-04T12:35:58.082 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-04T12:35:58.082 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.082 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.083 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.083 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.083 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.083 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.083 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.083 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.083 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.083 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.083 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.084 INFO:teuthology.orchestra.run.smithi060.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-04T12:35:58.084 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-04T12:35:58.084 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.084 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.084 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.085 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.085 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.085 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.085 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.085 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.085 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.085 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.085 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.085 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.086 INFO:teuthology.orchestra.run.smithi060.stdout: "PROMETHEUS_API_HOST": { 2024-03-04T12:35:58.086 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-04T12:35:58.086 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.086 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.086 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.086 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.086 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.086 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.086 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.087 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.087 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.087 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.087 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.087 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.087 INFO:teuthology.orchestra.run.smithi060.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-04T12:35:58.087 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-04T12:35:58.087 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.087 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.088 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.088 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.088 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.088 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.088 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.088 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.088 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.088 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.088 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.089 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.089 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-04T12:35:58.089 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-04T12:35:58.089 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.089 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.089 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.089 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.089 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.090 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.090 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.090 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.090 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.090 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.090 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.090 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-04T12:35:58.090 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-04T12:35:58.091 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.091 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.091 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.091 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.091 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.091 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.091 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.091 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.091 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.092 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.092 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.092 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-04T12:35:58.092 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-04T12:35:58.092 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.092 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.092 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.093 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.093 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.093 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.093 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.093 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.093 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.093 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.093 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.094 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-04T12:35:58.094 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-04T12:35:58.094 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.094 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.094 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.094 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.094 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.094 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.095 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.095 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.095 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.095 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.095 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.095 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.095 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-04T12:35:58.095 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-04T12:35:58.096 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.096 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.096 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.096 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.096 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.096 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.096 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.096 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.097 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.097 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.097 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.097 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.097 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-04T12:35:58.097 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-04T12:35:58.097 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.097 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.098 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.098 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.098 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.098 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.098 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.098 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.098 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.098 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.099 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.099 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-04T12:35:58.099 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-04T12:35:58.099 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.099 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.099 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.099 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.099 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.100 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.100 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.100 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.100 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.100 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.100 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.100 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_ENABLED": { 2024-03-04T12:35:58.101 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-04T12:35:58.101 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.101 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.101 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.101 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.101 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.101 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.101 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.101 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.102 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.102 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.102 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.102 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.102 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-04T12:35:58.102 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-04T12:35:58.102 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.102 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.102 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.103 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-04T12:35:58.103 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.103 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.103 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.103 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.103 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.103 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.103 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.104 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.104 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-04T12:35:58.104 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-04T12:35:58.104 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.104 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.104 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.104 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.104 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.105 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.105 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.105 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.105 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.105 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.105 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.105 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-04T12:35:58.106 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-04T12:35:58.106 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.106 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.106 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.106 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8", 2024-03-04T12:35:58.106 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.106 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.107 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.107 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.107 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.107 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.107 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.107 INFO:teuthology.orchestra.run.smithi060.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-04T12:35:58.107 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-04T12:35:58.107 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.108 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.108 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.108 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "45", 2024-03-04T12:35:58.108 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.108 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.108 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.108 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.108 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.109 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.109 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.109 INFO:teuthology.orchestra.run.smithi060.stdout: "RGW_API_ACCESS_KEY": { 2024-03-04T12:35:58.109 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-04T12:35:58.109 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.109 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.109 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.109 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.110 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.110 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.110 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.110 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.110 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.110 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.110 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.110 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.110 INFO:teuthology.orchestra.run.smithi060.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-04T12:35:58.111 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-04T12:35:58.111 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.111 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.111 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.111 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "admin", 2024-03-04T12:35:58.111 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.111 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.111 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.112 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.112 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.112 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.112 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.112 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.112 INFO:teuthology.orchestra.run.smithi060.stdout: "RGW_API_SECRET_KEY": { 2024-03-04T12:35:58.112 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-04T12:35:58.112 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.113 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.113 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.113 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.113 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.113 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.113 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.113 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.113 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.113 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.114 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.114 INFO:teuthology.orchestra.run.smithi060.stdout: "RGW_API_SSL_VERIFY": { 2024-03-04T12:35:58.114 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-04T12:35:58.114 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.114 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.114 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.114 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.114 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.115 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.115 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.115 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.115 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.115 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.115 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.115 INFO:teuthology.orchestra.run.smithi060.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-04T12:35:58.115 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-04T12:35:58.116 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.116 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.116 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.116 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:58.116 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.116 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.116 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.116 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.117 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.117 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.117 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.117 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.117 INFO:teuthology.orchestra.run.smithi060.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-04T12:35:58.117 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-04T12:35:58.117 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.117 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.117 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.118 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.118 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.118 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.118 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.118 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.118 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.118 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.118 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.119 INFO:teuthology.orchestra.run.smithi060.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-04T12:35:58.119 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-04T12:35:58.119 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.119 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.119 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.119 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5", 2024-03-04T12:35:58.119 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.120 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.120 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.120 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.120 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.120 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.120 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.120 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.120 INFO:teuthology.orchestra.run.smithi060.stdout: "cross_origin_url": { 2024-03-04T12:35:58.120 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cross_origin_url", 2024-03-04T12:35:58.121 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.121 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.121 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.121 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.121 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.121 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.121 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.121 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.122 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.122 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.122 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.122 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.122 INFO:teuthology.orchestra.run.smithi060.stdout: "crt_file": { 2024-03-04T12:35:58.122 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crt_file", 2024-03-04T12:35:58.122 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.122 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.123 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.123 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.123 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.123 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.123 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.123 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.123 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.123 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.123 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: "debug": { 2024-03-04T12:35:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "debug", 2024-03-04T12:35:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.124 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Enable/disable debug options", 2024-03-04T12:35:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: "jwt_token_ttl": { 2024-03-04T12:35:58.125 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "jwt_token_ttl", 2024-03-04T12:35:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "28800", 2024-03-04T12:35:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.126 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: "key_file": { 2024-03-04T12:35:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "key_file", 2024-03-04T12:35:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.127 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.128 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.129 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.130 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.131 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.132 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.133 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.134 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.135 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.136 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.137 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.137 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.137 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.137 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.137 INFO:teuthology.orchestra.run.smithi060.stdout: "motd": { 2024-03-04T12:35:58.137 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "motd", 2024-03-04T12:35:58.137 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.138 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.138 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.138 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.138 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.138 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.138 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.138 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "The message of the day", 2024-03-04T12:35:58.138 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.139 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.139 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.139 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_resolve_ip_addr": { 2024-03-04T12:35:58.139 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "redirect_resolve_ip_addr", 2024-03-04T12:35:58.139 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.139 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.140 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.140 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.140 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.140 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.140 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.140 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.140 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.140 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.141 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.141 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.141 INFO:teuthology.orchestra.run.smithi060.stdout: "server_addr": { 2024-03-04T12:35:58.141 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_addr", 2024-03-04T12:35:58.141 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.141 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.141 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.141 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "::", 2024-03-04T12:35:58.141 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.142 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.142 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.142 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.142 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.142 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.142 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.142 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.142 INFO:teuthology.orchestra.run.smithi060.stdout: "server_port": { 2024-03-04T12:35:58.143 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_port", 2024-03-04T12:35:58.143 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.143 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.143 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.143 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8080", 2024-03-04T12:35:58.143 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.143 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.143 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.143 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.144 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.144 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.144 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.144 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.144 INFO:teuthology.orchestra.run.smithi060.stdout: "ssl": { 2024-03-04T12:35:58.144 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ssl", 2024-03-04T12:35:58.144 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.144 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.145 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.145 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.145 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.145 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.145 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.145 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.145 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.145 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.146 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.146 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.146 INFO:teuthology.orchestra.run.smithi060.stdout: "ssl_server_port": { 2024-03-04T12:35:58.146 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ssl_server_port", 2024-03-04T12:35:58.146 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.146 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.146 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.146 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8443", 2024-03-04T12:35:58.147 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.147 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.147 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.147 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.147 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.147 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.147 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.147 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.147 INFO:teuthology.orchestra.run.smithi060.stdout: "standby_behaviour": { 2024-03-04T12:35:58.148 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "standby_behaviour", 2024-03-04T12:35:58.148 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.148 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.148 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.148 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "redirect", 2024-03-04T12:35:58.148 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.148 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.148 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.149 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.149 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect" 2024-03-04T12:35:58.149 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.149 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.149 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.149 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.149 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.149 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.150 INFO:teuthology.orchestra.run.smithi060.stdout: "standby_error_status_code": { 2024-03-04T12:35:58.150 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "standby_error_status_code", 2024-03-04T12:35:58.150 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.150 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.150 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.150 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "500", 2024-03-04T12:35:58.150 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "400", 2024-03-04T12:35:58.151 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "599", 2024-03-04T12:35:58.151 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.151 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.151 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.151 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.151 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.151 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.151 INFO:teuthology.orchestra.run.smithi060.stdout: "url_prefix": { 2024-03-04T12:35:58.152 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "url_prefix", 2024-03-04T12:35:58.152 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.152 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.152 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.152 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.152 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.152 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.152 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.153 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.153 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.153 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.153 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.153 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.153 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.153 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.153 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.154 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "devicehealth", 2024-03-04T12:35:58.154 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.154 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.154 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.154 INFO:teuthology.orchestra.run.smithi060.stdout: "enable_monitoring": { 2024-03-04T12:35:58.154 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "enable_monitoring", 2024-03-04T12:35:58.154 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.154 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.154 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.155 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.155 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.155 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.155 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.155 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "monitor device health metrics", 2024-03-04T12:35:58.155 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.155 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.155 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.156 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.156 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.156 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.156 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.156 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.156 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.156 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.156 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.157 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.157 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.157 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.157 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.157 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.157 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.157 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.157 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.158 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.158 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.158 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.158 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.158 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.158 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.158 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.158 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.158 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.159 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.159 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.159 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.159 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.159 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.159 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.159 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.159 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.159 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.160 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.160 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.160 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.160 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.160 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.160 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.160 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.160 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.161 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.161 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.161 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.161 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.161 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.161 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.161 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.161 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.161 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.162 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.162 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.162 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.162 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.162 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.162 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.162 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.162 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.163 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.163 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.163 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.163 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.163 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.163 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.163 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.163 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.163 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.164 INFO:teuthology.orchestra.run.smithi060.stdout: "mark_out_threshold": { 2024-03-04T12:35:58.164 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mark_out_threshold", 2024-03-04T12:35:58.164 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.164 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.164 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.164 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "2419200", 2024-03-04T12:35:58.164 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.165 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.165 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.165 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-04T12:35:58.165 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.165 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.165 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.165 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.166 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": { 2024-03-04T12:35:58.166 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "pool_name", 2024-03-04T12:35:58.166 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.166 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.166 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.166 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "device_health_metrics", 2024-03-04T12:35:58.166 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.166 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.166 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.167 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-04T12:35:58.167 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.167 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.167 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.167 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.167 INFO:teuthology.orchestra.run.smithi060.stdout: "retention_period": { 2024-03-04T12:35:58.167 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "retention_period", 2024-03-04T12:35:58.167 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.167 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.168 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.168 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "15552000", 2024-03-04T12:35:58.168 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.168 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.168 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.168 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how long to retain device health metrics", 2024-03-04T12:35:58.168 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.168 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.169 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.169 INFO:teuthology.orchestra.run.smithi060.stdout: "scrape_frequency": { 2024-03-04T12:35:58.169 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "scrape_frequency", 2024-03-04T12:35:58.169 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.169 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.169 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.169 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "86400", 2024-03-04T12:35:58.170 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.170 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.170 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.170 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-04T12:35:58.170 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.170 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.170 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.171 INFO:teuthology.orchestra.run.smithi060.stdout: "self_heal": { 2024-03-04T12:35:58.171 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "self_heal", 2024-03-04T12:35:58.171 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.171 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.171 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.171 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.171 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.171 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.171 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-04T12:35:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.172 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to wake up and check device health", 2024-03-04T12:35:58.173 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_threshold": { 2024-03-04T12:35:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_threshold", 2024-03-04T12:35:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.174 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "7257600", 2024-03-04T12:35:58.175 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.175 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.175 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.175 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-04T12:35:58.175 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.175 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.175 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.175 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.176 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.176 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "diskprediction_local", 2024-03-04T12:35:58.176 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.176 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.176 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.176 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.176 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.176 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.177 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.177 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.177 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.177 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.177 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.177 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.177 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.177 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.178 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.178 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.178 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.178 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.178 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.178 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.178 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.178 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.178 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.179 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.179 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.179 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.179 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.179 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.179 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.179 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.180 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.180 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.180 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.180 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.180 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.180 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.180 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.181 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.181 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.181 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.181 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.181 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.181 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.181 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.181 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.182 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.182 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.182 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.182 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.182 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.182 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.182 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.182 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.183 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.183 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.183 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.183 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.183 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.183 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.183 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.183 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.183 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.184 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.184 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.184 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.184 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.184 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.184 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.184 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.184 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: "predict_interval": { 2024-03-04T12:35:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "predict_interval", 2024-03-04T12:35:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "86400", 2024-03-04T12:35:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.185 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: "predictor_model": { 2024-03-04T12:35:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "predictor_model", 2024-03-04T12:35:58.186 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "prophetstor", 2024-03-04T12:35:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.187 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.188 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.188 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:58.188 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:58.188 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.188 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.188 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.188 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.189 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "influx", 2024-03-04T12:35:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": false, 2024-03-04T12:35:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "influxdb python module not found", 2024-03-04T12:35:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: "batch_size": { 2024-03-04T12:35:58.190 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "batch_size", 2024-03-04T12:35:58.191 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.191 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.191 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.191 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5000", 2024-03-04T12:35:58.191 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.191 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.191 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.191 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-04T12:35:58.192 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.192 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.192 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.192 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.192 INFO:teuthology.orchestra.run.smithi060.stdout: "database": { 2024-03-04T12:35:58.192 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "database", 2024-03-04T12:35:58.192 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.192 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "ceph", 2024-03-04T12:35:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-04T12:35:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.193 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.194 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.194 INFO:teuthology.orchestra.run.smithi060.stdout: "hostname": { 2024-03-04T12:35:58.194 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "hostname", 2024-03-04T12:35:58.194 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.194 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.194 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.194 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "InfluxDB server hostname", 2024-03-04T12:35:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.195 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:58.196 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:58.196 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.196 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.196 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.196 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "30", 2024-03-04T12:35:58.196 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "5", 2024-03-04T12:35:58.196 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-04T12:35:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.197 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.198 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.198 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.198 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.198 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.198 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.198 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.198 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.198 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.199 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.199 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.199 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.199 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.199 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.199 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.199 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.199 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.200 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.200 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.200 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.200 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.200 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.200 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.200 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.201 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.201 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.201 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.201 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.201 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.201 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.201 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.201 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.202 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.202 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.202 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.202 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.202 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.202 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.202 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.204 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.204 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.204 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.204 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.204 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.204 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.204 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.205 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.205 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.205 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.205 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.205 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.205 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.205 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.205 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.206 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.206 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.206 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.206 INFO:teuthology.orchestra.run.smithi060.stdout: "password": { 2024-03-04T12:35:58.206 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "password", 2024-03-04T12:35:58.206 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.206 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.207 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.207 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.207 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.207 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.207 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.207 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "password of InfluxDB server user", 2024-03-04T12:35:58.207 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.207 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: "port": { 2024-03-04T12:35:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "port", 2024-03-04T12:35:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.208 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8086", 2024-03-04T12:35:58.209 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.209 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.209 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.209 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "InfluxDB server port", 2024-03-04T12:35:58.209 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.209 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.209 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.209 INFO:teuthology.orchestra.run.smithi060.stdout: "ssl": { 2024-03-04T12:35:58.210 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ssl", 2024-03-04T12:35:58.210 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.210 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.210 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.210 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "false", 2024-03-04T12:35:58.210 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.210 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.210 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.210 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-04T12:35:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: "threads": { 2024-03-04T12:35:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "threads", 2024-03-04T12:35:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.211 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.212 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5", 2024-03-04T12:35:58.212 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1", 2024-03-04T12:35:58.212 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "32", 2024-03-04T12:35:58.212 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.212 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-04T12:35:58.212 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.212 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: "username": { 2024-03-04T12:35:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "username", 2024-03-04T12:35:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.213 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.214 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.214 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.214 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "username of InfluxDB server user", 2024-03-04T12:35:58.214 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.214 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.214 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.214 INFO:teuthology.orchestra.run.smithi060.stdout: "verify_ssl": { 2024-03-04T12:35:58.214 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "verify_ssl", 2024-03-04T12:35:58.215 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.215 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.215 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.215 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "true", 2024-03-04T12:35:58.215 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.215 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.215 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.215 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-04T12:35:58.215 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.216 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.216 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.216 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.216 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.216 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.216 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "insights", 2024-03-04T12:35:58.216 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.217 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.217 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.217 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.217 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.217 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.217 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.217 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.217 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.217 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.218 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.218 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.218 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.218 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.218 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.218 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.218 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.218 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.219 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.219 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.219 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.219 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.219 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.219 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.219 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.219 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.220 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.220 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.220 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.220 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.220 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.220 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.220 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.220 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.221 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.221 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.221 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.221 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.221 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.221 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.221 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.221 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.222 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.222 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.222 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.222 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.222 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.222 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.222 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.222 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.223 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.223 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.223 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.223 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.223 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.223 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.223 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.223 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.224 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.224 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.224 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.224 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.224 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.224 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.224 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.225 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.225 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.225 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.225 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.225 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.225 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.225 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.225 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.225 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "iostat", 2024-03-04T12:35:58.226 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.226 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.226 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.226 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.226 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.226 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.226 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.226 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.227 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.227 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.227 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.227 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.227 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.227 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.227 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.227 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.227 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.227 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.228 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.228 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.228 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.228 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.228 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.228 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.228 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.228 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.229 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.229 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.229 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.229 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.229 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.229 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.229 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.229 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.229 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.230 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.230 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.230 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.230 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.230 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.230 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.230 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.230 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.230 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.230 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.231 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.231 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.231 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.231 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.231 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.231 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.231 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.231 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.232 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.232 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.232 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.232 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.232 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.232 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.232 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.232 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.233 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.233 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.233 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.233 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.233 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.233 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.233 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.233 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.233 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.234 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.234 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.234 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.234 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.234 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.234 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "k8sevents", 2024-03-04T12:35:58.234 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.234 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.235 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.235 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_event_retention_days": { 2024-03-04T12:35:58.235 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ceph_event_retention_days", 2024-03-04T12:35:58.235 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.235 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.235 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.235 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "7", 2024-03-04T12:35:58.235 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.235 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.236 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.236 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-04T12:35:58.236 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.236 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.236 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.236 INFO:teuthology.orchestra.run.smithi060.stdout: "config_check_secs": { 2024-03-04T12:35:58.236 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "config_check_secs", 2024-03-04T12:35:58.237 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.237 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.237 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.237 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.237 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "10", 2024-03-04T12:35:58.237 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.237 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.237 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-04T12:35:58.237 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.238 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.238 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.238 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.238 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.238 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.238 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.238 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.238 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.239 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.239 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.239 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.239 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.239 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.239 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.239 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.239 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.239 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.240 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.240 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.240 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.240 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.240 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.240 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.240 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.240 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.241 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.241 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.241 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.241 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.241 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.241 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.241 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.241 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.242 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.242 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.242 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.242 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.242 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.242 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.242 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.242 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.242 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.243 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.243 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.243 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.243 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.243 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.243 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.243 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.243 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.243 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.244 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.244 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.244 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.244 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.244 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.244 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.244 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.244 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.245 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.245 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.245 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.245 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.245 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.245 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.245 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.245 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.246 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.246 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.246 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.246 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.246 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.246 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.246 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.246 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.246 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "localpool", 2024-03-04T12:35:58.247 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.247 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.247 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.247 INFO:teuthology.orchestra.run.smithi060.stdout: "failure_domain": { 2024-03-04T12:35:58.247 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "failure_domain", 2024-03-04T12:35:58.247 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.248 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.248 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.248 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "host", 2024-03-04T12:35:58.248 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.248 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.248 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.248 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "failure domain for any created local pool", 2024-03-04T12:35:58.249 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-04T12:35:58.249 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.249 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.249 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.249 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.249 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.249 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.249 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.249 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.250 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.250 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.250 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.250 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.250 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.250 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.250 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.250 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.250 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.251 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.251 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.251 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.251 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.251 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.251 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.251 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.251 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.251 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.252 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.252 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.252 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.252 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.252 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.252 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.253 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.253 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.253 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.253 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.253 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.253 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.253 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.254 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.254 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.254 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.254 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.254 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.254 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.254 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.254 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.255 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.255 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.255 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.255 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.255 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.255 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.255 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.255 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.255 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.256 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.256 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.256 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.256 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.256 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.256 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.256 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.256 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.257 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.257 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.257 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.257 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.257 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.257 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.258 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": { 2024-03-04T12:35:58.258 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "min_size", 2024-03-04T12:35:58.258 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.258 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.258 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.258 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.258 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.258 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.258 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.259 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "default min_size for any created local pool", 2024-03-04T12:35:58.259 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-04T12:35:58.259 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.259 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.259 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.259 INFO:teuthology.orchestra.run.smithi060.stdout: "num_rep": { 2024-03-04T12:35:58.259 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "num_rep", 2024-03-04T12:35:58.259 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.260 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.260 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.260 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "3", 2024-03-04T12:35:58.260 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.260 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.260 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.260 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "default replica count for any created local pool", 2024-03-04T12:35:58.260 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.260 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.261 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.261 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": { 2024-03-04T12:35:58.261 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "pg_num", 2024-03-04T12:35:58.261 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.261 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.261 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.261 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "128", 2024-03-04T12:35:58.262 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.262 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.262 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.262 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "default pg_num for any created local pool", 2024-03-04T12:35:58.262 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.262 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.262 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.262 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.262 INFO:teuthology.orchestra.run.smithi060.stdout: "prefix": { 2024-03-04T12:35:58.263 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "prefix", 2024-03-04T12:35:58.263 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.263 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.263 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.263 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.263 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.263 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.263 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.264 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "name prefix for any created local pool", 2024-03-04T12:35:58.264 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.264 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.264 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.264 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.264 INFO:teuthology.orchestra.run.smithi060.stdout: "subtree": { 2024-03-04T12:35:58.264 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "subtree", 2024-03-04T12:35:58.265 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.265 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.265 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.265 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "rack", 2024-03-04T12:35:58.265 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.265 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.265 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.265 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-04T12:35:58.266 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-04T12:35:58.266 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.266 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.266 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.266 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.266 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.266 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.266 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mds_autoscaler", 2024-03-04T12:35:58.267 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.267 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.267 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.267 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.267 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.267 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.267 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.267 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.267 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.268 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.268 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.268 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.268 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.268 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.268 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.268 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.268 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.269 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.269 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.269 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.269 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.269 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.269 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.269 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.269 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.270 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.270 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.270 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.270 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.270 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.270 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.270 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.270 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.270 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.271 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.271 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.271 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.271 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.271 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.271 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.271 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.271 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.272 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.272 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.272 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.272 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.272 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.272 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.272 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.272 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.272 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.273 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.273 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.273 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.273 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.273 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.273 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.273 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.274 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.274 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.274 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.274 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.274 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.274 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.274 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.274 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.274 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.275 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.275 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.275 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.275 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.275 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.275 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.275 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mirroring", 2024-03-04T12:35:58.276 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.276 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.276 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.276 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.276 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.276 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.276 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.276 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.276 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.277 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.277 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.277 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.277 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.277 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.277 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.277 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.277 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.277 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.278 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.278 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.278 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.278 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.278 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.278 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.278 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.278 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.278 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.279 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.279 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.279 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.279 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.279 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.279 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.279 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.280 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.280 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.280 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.280 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.280 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.280 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.280 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.280 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.281 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.281 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.281 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.281 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.281 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.281 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.281 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.281 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.281 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.282 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.282 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.282 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.282 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.282 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.282 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.282 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.282 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.283 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.283 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.283 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.283 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.283 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.283 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.283 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.283 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.283 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.284 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.284 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.284 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.284 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.284 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.284 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.284 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "nfs", 2024-03-04T12:35:58.284 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.284 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.285 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.285 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.285 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.285 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.285 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.285 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.285 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.285 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.286 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.286 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.286 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.286 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.286 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.286 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.286 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.286 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.286 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.287 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.287 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.287 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.287 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.287 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.287 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.287 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.288 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.288 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.288 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.288 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.288 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.288 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.288 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.288 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.289 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.289 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.289 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.289 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.289 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.289 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.289 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.290 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.290 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.290 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.290 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.290 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.290 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.290 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.290 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.290 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.291 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.291 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.291 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.291 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.291 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.291 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.291 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.291 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.292 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.292 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.292 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.292 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.292 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.292 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.292 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.292 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.292 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.293 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.293 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.293 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.293 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.293 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "orchestrator", 2024-03-04T12:35:58.293 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.294 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.294 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.294 INFO:teuthology.orchestra.run.smithi060.stdout: "fail_fs": { 2024-03-04T12:35:58.294 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "fail_fs", 2024-03-04T12:35:58.294 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.294 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.294 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.294 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.295 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.295 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.295 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.295 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-04T12:35:58.295 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.295 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.295 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.295 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.296 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.296 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.296 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.296 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.296 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.296 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.296 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.297 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.297 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.297 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.297 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.297 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.297 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.297 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.297 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.297 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.298 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.298 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.298 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.298 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.298 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.298 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.298 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.298 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.298 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.299 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.299 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.299 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.299 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.299 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.299 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.299 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.299 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.300 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.300 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.300 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.300 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.300 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.300 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.300 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.300 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.301 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.301 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.301 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.301 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.301 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.301 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.301 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.301 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.302 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.302 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.302 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.302 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.302 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.302 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.302 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.302 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.302 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.303 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.303 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.303 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.303 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.303 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.303 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.303 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.303 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.303 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.304 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.304 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.304 INFO:teuthology.orchestra.run.smithi060.stdout: "orchestrator": { 2024-03-04T12:35:58.304 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "orchestrator", 2024-03-04T12:35:58.304 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.304 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.304 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.304 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.305 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.305 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.305 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.305 INFO:teuthology.orchestra.run.smithi060.stdout: "cephadm", 2024-03-04T12:35:58.305 INFO:teuthology.orchestra.run.smithi060.stdout: "rook", 2024-03-04T12:35:58.305 INFO:teuthology.orchestra.run.smithi060.stdout: "test_orchestrator" 2024-03-04T12:35:58.305 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.305 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Orchestrator backend", 2024-03-04T12:35:58.305 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.306 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.306 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.306 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.306 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.306 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.306 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.306 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "osd_perf_query", 2024-03-04T12:35:58.306 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.307 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.307 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.307 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.307 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.307 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.307 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.308 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.308 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.308 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.308 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.308 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.308 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.308 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.308 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.308 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.309 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.309 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.309 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.309 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.309 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.309 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.309 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.309 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.310 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.310 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.310 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.310 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.310 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.310 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.310 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.310 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.311 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.311 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.311 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.311 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.311 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.311 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.311 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.312 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.312 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.312 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.312 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.312 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.312 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.312 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.312 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.313 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.313 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.313 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.313 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.313 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.313 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.313 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.313 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.313 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.314 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.314 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.314 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.314 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.314 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.314 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.314 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.314 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.315 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.315 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.315 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.315 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.315 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.315 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.315 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.316 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.316 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "osd_support", 2024-03-04T12:35:58.316 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.316 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.316 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.316 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.316 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.316 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.317 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.317 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.317 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.317 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.317 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.317 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.317 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.317 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.317 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.318 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.318 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.318 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.318 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.318 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.318 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.318 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.318 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.319 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.319 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.319 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.319 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.319 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.319 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.319 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.320 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.320 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.320 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.320 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.320 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.320 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.320 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.320 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.321 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.321 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.321 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.321 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.321 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.321 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.321 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.321 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.321 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.322 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.322 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.322 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.322 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.322 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.322 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.322 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.322 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.323 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.323 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.323 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.323 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.323 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.323 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.323 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.323 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.323 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.324 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.324 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.324 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.324 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.324 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.324 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.324 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.325 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.325 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.325 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.325 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.325 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "pg_autoscaler", 2024-03-04T12:35:58.325 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.325 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.325 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.325 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.326 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.326 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.326 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.326 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.326 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.326 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.326 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.326 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.326 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.327 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.327 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.327 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.327 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.327 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.327 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.327 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.327 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.327 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.328 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.328 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.328 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.328 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.328 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.328 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.328 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.328 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.328 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.329 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.329 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.329 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.329 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.329 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.329 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.329 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.329 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.330 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.330 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.330 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.330 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.330 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.330 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.330 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.330 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.331 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.331 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.331 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.331 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.331 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.331 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.331 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.331 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.331 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.332 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.332 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.332 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.332 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.332 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.332 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.332 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.333 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.333 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.333 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.333 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.333 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.333 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.333 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.333 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.333 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:58.334 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:58.334 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.334 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.334 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.334 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:58.334 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.334 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.334 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.334 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.335 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.335 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.335 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.335 INFO:teuthology.orchestra.run.smithi060.stdout: "threshold": { 2024-03-04T12:35:58.335 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "threshold", 2024-03-04T12:35:58.335 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:58.335 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.335 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.336 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "3.0", 2024-03-04T12:35:58.336 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1.0", 2024-03-04T12:35:58.336 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.336 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.336 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "scaling threshold", 2024-03-04T12:35:58.336 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-04T12:35:58.336 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.336 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.336 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.337 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.337 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.337 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "progress", 2024-03-04T12:35:58.337 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.337 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.337 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.337 INFO:teuthology.orchestra.run.smithi060.stdout: "allow_pg_recovery_event": { 2024-03-04T12:35:58.337 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "allow_pg_recovery_event", 2024-03-04T12:35:58.338 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.338 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.338 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.338 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.338 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.338 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.338 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.338 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-04T12:35:58.338 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.339 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.339 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.339 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": { 2024-03-04T12:35:58.339 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "enabled", 2024-03-04T12:35:58.339 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.339 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.339 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.339 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.340 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.340 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.340 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.340 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.340 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.340 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.340 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.340 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.340 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.341 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.341 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.341 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.341 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.341 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.341 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.341 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.341 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.341 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.342 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.342 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.342 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.342 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.342 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.342 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.342 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.342 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.342 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.343 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.343 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.343 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.343 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.343 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.343 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.343 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.343 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.344 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.344 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.344 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.344 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.344 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.344 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.344 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.344 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.345 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.345 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.345 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.345 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.345 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.345 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.345 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.345 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.345 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.346 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.346 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.346 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.346 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.346 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.346 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.346 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.346 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.346 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.347 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.347 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.347 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.347 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.347 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.347 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.347 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.347 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.348 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.348 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.348 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.348 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.348 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.348 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.348 INFO:teuthology.orchestra.run.smithi060.stdout: "max_completed_events": { 2024-03-04T12:35:58.349 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_completed_events", 2024-03-04T12:35:58.349 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.349 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.349 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.349 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "50", 2024-03-04T12:35:58.349 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.349 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.349 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.349 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "number of past completed events to remember", 2024-03-04T12:35:58.350 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.350 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.350 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.350 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:58.350 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:58.350 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.350 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.350 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.351 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5", 2024-03-04T12:35:58.351 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.351 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.351 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.351 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how long the module is going to sleep", 2024-03-04T12:35:58.351 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.351 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.351 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.351 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.352 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.352 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.352 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "prometheus", 2024-03-04T12:35:58.352 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.352 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.352 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.352 INFO:teuthology.orchestra.run.smithi060.stdout: "cache": { 2024-03-04T12:35:58.352 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cache", 2024-03-04T12:35:58.353 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.353 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.353 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.353 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.353 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.353 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.353 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.353 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.353 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.354 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.354 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.354 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.354 INFO:teuthology.orchestra.run.smithi060.stdout: "exclude_perf_counters": { 2024-03-04T12:35:58.354 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "exclude_perf_counters", 2024-03-04T12:35:58.354 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.354 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.354 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.354 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.355 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.355 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.355 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.355 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-04T12:35:58.355 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-04T12:35:58.355 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.355 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.356 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.356 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.356 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.356 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.356 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.356 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.356 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.356 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.357 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.357 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.357 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.357 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.357 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.357 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.357 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.357 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.357 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.357 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.358 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.358 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.358 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.358 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.358 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.358 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.358 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.358 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.359 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.359 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.359 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.359 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.359 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.359 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.359 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.359 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.360 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.360 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.360 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.360 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.360 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.360 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.360 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.361 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.361 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.361 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.361 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.361 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.361 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.361 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.361 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.361 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.362 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.362 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.362 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.362 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.362 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.362 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.362 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.362 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.363 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.363 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.363 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.363 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.363 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.363 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.363 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.363 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.364 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.364 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.364 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_stats_pools": { 2024-03-04T12:35:58.364 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd_stats_pools", 2024-03-04T12:35:58.364 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.364 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.364 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.364 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.364 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.365 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.365 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.365 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.365 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.365 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.365 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.365 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-04T12:35:58.365 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-04T12:35:58.366 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.366 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.366 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.366 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "300", 2024-03-04T12:35:58.366 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.366 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.366 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.366 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.366 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.367 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.367 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.367 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.367 INFO:teuthology.orchestra.run.smithi060.stdout: "scrape_interval": { 2024-03-04T12:35:58.367 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "scrape_interval", 2024-03-04T12:35:58.367 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:58.367 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.367 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.367 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "15.0", 2024-03-04T12:35:58.368 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.368 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.368 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.368 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.368 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.368 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.368 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.368 INFO:teuthology.orchestra.run.smithi060.stdout: "server_addr": { 2024-03-04T12:35:58.369 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_addr", 2024-03-04T12:35:58.369 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.369 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.369 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.369 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "::", 2024-03-04T12:35:58.369 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.369 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.369 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.369 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-04T12:35:58.370 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.370 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.370 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.370 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.370 INFO:teuthology.orchestra.run.smithi060.stdout: "server_port": { 2024-03-04T12:35:58.370 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_port", 2024-03-04T12:35:58.370 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.370 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.371 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.371 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "9283", 2024-03-04T12:35:58.371 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.371 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.371 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.371 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-04T12:35:58.371 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.371 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.371 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.372 INFO:teuthology.orchestra.run.smithi060.stdout: "stale_cache_strategy": { 2024-03-04T12:35:58.372 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "stale_cache_strategy", 2024-03-04T12:35:58.372 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.372 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.372 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.372 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "log", 2024-03-04T12:35:58.372 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.372 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.373 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.373 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.373 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.373 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.373 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.373 INFO:teuthology.orchestra.run.smithi060.stdout: "standby_behaviour": { 2024-03-04T12:35:58.373 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "standby_behaviour", 2024-03-04T12:35:58.374 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.374 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.374 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.374 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "default", 2024-03-04T12:35:58.374 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.374 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.374 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.374 INFO:teuthology.orchestra.run.smithi060.stdout: "default", 2024-03-04T12:35:58.374 INFO:teuthology.orchestra.run.smithi060.stdout: "error" 2024-03-04T12:35:58.375 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.375 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.375 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.375 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.375 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.375 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.375 INFO:teuthology.orchestra.run.smithi060.stdout: "standby_error_status_code": { 2024-03-04T12:35:58.375 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "standby_error_status_code", 2024-03-04T12:35:58.376 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.376 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.376 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.376 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "500", 2024-03-04T12:35:58.376 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "400", 2024-03-04T12:35:58.376 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "599", 2024-03-04T12:35:58.376 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.376 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.377 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.377 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.377 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.377 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.377 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.377 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.377 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.377 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd_support", 2024-03-04T12:35:58.378 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.378 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.378 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.378 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.378 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.378 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.378 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.379 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.379 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.379 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.379 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.379 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.379 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.379 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.379 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.379 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.380 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.380 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.380 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.380 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.380 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.380 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.380 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.381 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.382 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.382 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.382 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.382 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.382 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.382 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.382 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.383 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.384 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.384 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.384 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.384 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.384 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.384 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.384 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.384 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.385 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.386 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.386 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.386 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.386 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.386 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.386 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.386 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.386 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.386 INFO:teuthology.orchestra.run.smithi060.stdout: "max_concurrent_snap_create": { 2024-03-04T12:35:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_concurrent_snap_create", 2024-03-04T12:35:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.387 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: "mirror_snapshot_schedule": { 2024-03-04T12:35:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mirror_snapshot_schedule", 2024-03-04T12:35:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.388 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.389 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.389 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.389 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.389 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.389 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.389 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.389 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.389 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.390 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.390 INFO:teuthology.orchestra.run.smithi060.stdout: "trash_purge_schedule": { 2024-03-04T12:35:58.390 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "trash_purge_schedule", 2024-03-04T12:35:58.390 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.390 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.390 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.390 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.390 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.390 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.391 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.391 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.391 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.391 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.391 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.391 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.391 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.391 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.392 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "restful", 2024-03-04T12:35:58.392 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.392 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.392 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.392 INFO:teuthology.orchestra.run.smithi060.stdout: "enable_auth": { 2024-03-04T12:35:58.392 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "enable_auth", 2024-03-04T12:35:58.392 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.392 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.392 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.393 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.393 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.393 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.393 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.393 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.393 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.393 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.393 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.394 INFO:teuthology.orchestra.run.smithi060.stdout: "key_file": { 2024-03-04T12:35:58.394 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "key_file", 2024-03-04T12:35:58.394 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.394 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.394 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.394 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.394 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.394 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.394 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.394 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.395 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.395 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.395 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.395 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.395 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.395 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.395 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.395 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.396 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.396 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.396 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.396 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.396 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.396 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.396 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.396 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.397 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.397 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.397 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.397 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.397 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.397 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.397 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.398 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.398 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.398 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.398 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.398 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.398 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.398 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.398 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.398 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.398 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.399 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.399 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.399 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.399 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.399 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.400 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.400 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.400 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.400 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.400 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.400 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.400 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.400 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.401 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.401 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.401 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.401 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.401 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.401 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.401 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.401 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.401 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.402 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.402 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.402 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.402 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.402 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.402 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.402 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.402 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.403 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.403 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.403 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.403 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.403 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.403 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.403 INFO:teuthology.orchestra.run.smithi060.stdout: "server_addr": { 2024-03-04T12:35:58.404 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_addr", 2024-03-04T12:35:58.404 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.404 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.404 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.404 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.404 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.404 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.404 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.405 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.405 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.405 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.405 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.405 INFO:teuthology.orchestra.run.smithi060.stdout: "server_port": { 2024-03-04T12:35:58.405 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_port", 2024-03-04T12:35:58.405 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.405 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.406 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.406 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.406 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.406 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.406 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.406 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.406 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.406 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.407 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.407 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.407 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.407 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.407 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rgw", 2024-03-04T12:35:58.407 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.407 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.407 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.408 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.408 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.408 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.408 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.408 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.408 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.408 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.408 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.408 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.409 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.409 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.409 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.409 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.409 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.409 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.409 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.409 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.409 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.410 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.410 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.410 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.410 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.410 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.410 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.410 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.411 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.411 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.411 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.411 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.411 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.411 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.411 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.411 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.412 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.412 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.412 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.412 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.412 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.412 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.412 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.412 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.413 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.413 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.413 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.413 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.413 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.413 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.413 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.413 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.413 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.414 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.414 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.414 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.414 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.414 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.414 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.414 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.415 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.415 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.415 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.415 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.415 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.415 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.415 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.415 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.415 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.416 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.416 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.416 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.416 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.416 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rook", 2024-03-04T12:35:58.416 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.416 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.416 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.417 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.417 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.417 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.417 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.417 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.417 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.417 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.417 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.417 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.418 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.418 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.418 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.418 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.418 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.418 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.418 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.418 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.418 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.419 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.419 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.419 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.419 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.419 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.419 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.419 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.419 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.419 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.420 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.420 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.420 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.420 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.420 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.420 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.420 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.420 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.421 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.421 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.421 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.421 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.421 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.421 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.421 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.421 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.421 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.422 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.422 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.422 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.422 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.422 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.422 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.422 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.422 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.422 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.423 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.423 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.423 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.423 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.423 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.423 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.423 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.424 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.424 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.424 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.424 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.424 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.424 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.424 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.425 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_class": { 2024-03-04T12:35:58.425 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "storage_class", 2024-03-04T12:35:58.425 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.425 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.425 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.425 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "local", 2024-03-04T12:35:58.425 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.425 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.425 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.426 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-04T12:35:58.426 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.426 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.426 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.426 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.426 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.426 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.426 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.426 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "selftest", 2024-03-04T12:35:58.427 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.427 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.427 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.427 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.427 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.427 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.427 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.427 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.427 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.428 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.428 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.428 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.428 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.428 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.428 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.428 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.428 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.428 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.429 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.429 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.429 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.429 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.429 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.429 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.429 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.429 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.429 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.430 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.430 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.430 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.430 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.430 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.430 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.430 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.430 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.430 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.431 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.431 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.431 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.431 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.431 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.431 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.431 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.431 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.432 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.432 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.432 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.432 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.432 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.432 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.432 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.432 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.433 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.433 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.433 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.433 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.433 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.433 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.433 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.433 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.434 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.434 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.434 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.434 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.434 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.434 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.434 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.434 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.434 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.435 INFO:teuthology.orchestra.run.smithi060.stdout: "roption1": { 2024-03-04T12:35:58.435 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "roption1", 2024-03-04T12:35:58.435 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.435 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.435 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.435 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.435 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.436 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.436 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.436 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.436 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.436 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.436 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.436 INFO:teuthology.orchestra.run.smithi060.stdout: "roption2": { 2024-03-04T12:35:58.437 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "roption2", 2024-03-04T12:35:58.437 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.437 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.437 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.437 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "xyz", 2024-03-04T12:35:58.437 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.437 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.437 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.437 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.438 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.438 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.438 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.438 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.438 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption1": { 2024-03-04T12:35:58.438 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption1", 2024-03-04T12:35:58.438 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.438 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.438 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.439 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.439 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.439 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.439 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.439 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.439 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.439 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.439 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.440 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption2": { 2024-03-04T12:35:58.440 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption2", 2024-03-04T12:35:58.440 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.440 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.440 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.440 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.440 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.440 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.441 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.441 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.441 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.441 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.441 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.441 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.441 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption3": { 2024-03-04T12:35:58.441 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption3", 2024-03-04T12:35:58.441 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:58.442 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.442 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.442 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.442 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.442 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.442 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.442 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.442 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.443 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.443 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.443 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption4": { 2024-03-04T12:35:58.443 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption4", 2024-03-04T12:35:58.443 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.443 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.443 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.443 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.444 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.444 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.444 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.444 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.444 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.444 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.444 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.445 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption5": { 2024-03-04T12:35:58.445 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption5", 2024-03-04T12:35:58.445 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.445 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.445 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.445 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.445 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.445 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.446 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.446 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.446 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.446 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.446 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.446 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.446 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption6": { 2024-03-04T12:35:58.446 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption6", 2024-03-04T12:35:58.447 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.447 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.447 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.447 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.447 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.447 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.447 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.447 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.447 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.448 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.448 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.448 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.448 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption7": { 2024-03-04T12:35:58.448 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption7", 2024-03-04T12:35:58.448 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.448 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.448 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.449 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.449 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1", 2024-03-04T12:35:58.449 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "42", 2024-03-04T12:35:58.449 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.449 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.449 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.449 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.449 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.450 INFO:teuthology.orchestra.run.smithi060.stdout: "testkey": { 2024-03-04T12:35:58.450 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "testkey", 2024-03-04T12:35:58.450 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.450 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.450 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.450 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.450 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.450 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.450 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.451 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.451 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.451 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.451 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.451 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.451 INFO:teuthology.orchestra.run.smithi060.stdout: "testlkey": { 2024-03-04T12:35:58.451 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "testlkey", 2024-03-04T12:35:58.451 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.451 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.452 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.452 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.452 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.452 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.452 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.452 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.452 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.452 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.452 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.453 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.453 INFO:teuthology.orchestra.run.smithi060.stdout: "testnewline": { 2024-03-04T12:35:58.453 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "testnewline", 2024-03-04T12:35:58.453 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.453 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.453 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.453 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.453 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.454 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.454 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.454 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.454 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.454 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.454 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.454 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.454 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.455 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.455 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "snap_schedule", 2024-03-04T12:35:58.455 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.455 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.455 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.455 INFO:teuthology.orchestra.run.smithi060.stdout: "allow_m_granularity": { 2024-03-04T12:35:58.455 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "allow_m_granularity", 2024-03-04T12:35:58.455 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.455 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.456 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.456 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.456 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.456 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.456 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.456 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "allow minute scheduled snapshots", 2024-03-04T12:35:58.456 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.456 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.456 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.457 INFO:teuthology.orchestra.run.smithi060.stdout: "dump_on_update": { 2024-03-04T12:35:58.457 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "dump_on_update", 2024-03-04T12:35:58.457 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.457 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.457 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.457 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.457 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.457 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.458 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.458 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "dump database to debug log on update", 2024-03-04T12:35:58.458 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.458 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.458 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.458 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.458 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.458 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.458 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.459 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.459 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.459 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.459 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.459 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.459 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.459 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.459 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.459 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.460 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.460 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.460 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.460 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.460 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.460 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.460 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.460 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.460 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.461 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.461 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.461 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.461 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.461 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.461 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.461 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.461 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.461 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.462 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.462 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.462 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.462 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.462 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.462 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.462 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.463 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.463 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.463 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.463 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.463 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.463 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.463 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.463 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.463 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.463 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.464 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.464 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.464 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.464 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.464 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.464 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.464 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.465 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.465 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.465 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.465 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.465 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.465 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.465 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.466 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.466 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.466 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.466 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.466 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.466 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.466 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.466 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.467 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.467 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "stats", 2024-03-04T12:35:58.467 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.467 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.467 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.467 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.467 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.467 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.468 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.468 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.468 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.468 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.468 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.468 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.468 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.468 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.469 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.469 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.469 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.469 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.469 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.469 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.469 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.469 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.469 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.470 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.470 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.470 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.470 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.470 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.470 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.470 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.470 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.471 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.471 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.471 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.471 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.471 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.471 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.471 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.471 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.472 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.472 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.472 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.472 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.472 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.472 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.472 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.472 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.472 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.473 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.473 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.473 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.473 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.473 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.473 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.473 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.473 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.474 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.474 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.474 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.474 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.474 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.474 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.474 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.474 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.475 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.475 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.475 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.475 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.475 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.475 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.475 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.475 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.476 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.476 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.476 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "status", 2024-03-04T12:35:58.476 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.476 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.476 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.476 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.476 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.476 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.477 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.477 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.477 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.477 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.477 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.477 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.477 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.477 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.478 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.478 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.478 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.478 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.478 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.478 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.478 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.479 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.479 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.479 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.479 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.479 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.479 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.479 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.479 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.480 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.480 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.480 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.480 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.480 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.480 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.480 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.481 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.482 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.482 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.482 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.482 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.482 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.482 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.482 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.482 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.483 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.483 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.483 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.483 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.483 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.483 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.483 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.483 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.483 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.484 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.484 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.484 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.484 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.484 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.484 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.484 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.484 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.485 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.485 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.485 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.485 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.485 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.485 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "telegraf", 2024-03-04T12:35:58.485 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.485 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.486 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.486 INFO:teuthology.orchestra.run.smithi060.stdout: "address": { 2024-03-04T12:35:58.486 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "address", 2024-03-04T12:35:58.486 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.486 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.486 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.486 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-04T12:35:58.487 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.487 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.487 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.487 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.487 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.487 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.487 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.488 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:58.488 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:58.488 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.488 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.488 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.488 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "15", 2024-03-04T12:35:58.488 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.488 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.489 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.489 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.489 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.489 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.489 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.489 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.489 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.489 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.490 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.491 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.492 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.493 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.493 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.493 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.493 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.493 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.494 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.494 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.494 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.494 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.495 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.495 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.495 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.495 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.495 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.496 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.496 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.496 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.496 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.496 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.496 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.496 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.497 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.497 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.497 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.497 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.497 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.497 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.498 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.498 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.498 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.498 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.498 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.498 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.498 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.498 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.498 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.499 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.499 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.499 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.499 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.499 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.499 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.499 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.500 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.500 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "telemetry", 2024-03-04T12:35:58.500 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.500 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.500 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.500 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_basic": { 2024-03-04T12:35:58.500 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_basic", 2024-03-04T12:35:58.500 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.501 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.501 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.501 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.501 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.501 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.501 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.501 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-04T12:35:58.501 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.501 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.502 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.502 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.502 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_crash": { 2024-03-04T12:35:58.502 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_crash", 2024-03-04T12:35:58.502 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.502 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.502 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.502 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.503 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.503 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.503 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.503 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-04T12:35:58.503 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.503 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.503 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.503 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_device": { 2024-03-04T12:35:58.504 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_device", 2024-03-04T12:35:58.504 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.504 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.504 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.504 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.504 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.504 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.504 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.505 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-04T12:35:58.505 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.505 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.505 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.505 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_ident": { 2024-03-04T12:35:58.505 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_ident", 2024-03-04T12:35:58.505 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.506 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.506 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.506 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.506 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.506 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.506 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.506 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-04T12:35:58.506 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.506 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.507 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.507 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.507 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_perf": { 2024-03-04T12:35:58.507 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_perf", 2024-03-04T12:35:58.507 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.507 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.507 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.507 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.507 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.508 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.508 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.508 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-04T12:35:58.508 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.508 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.508 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.508 INFO:teuthology.orchestra.run.smithi060.stdout: "contact": { 2024-03-04T12:35:58.508 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "contact", 2024-03-04T12:35:58.509 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.509 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.509 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.509 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.509 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.509 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.509 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.509 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.510 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.510 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.510 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.510 INFO:teuthology.orchestra.run.smithi060.stdout: "description": { 2024-03-04T12:35:58.510 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "description", 2024-03-04T12:35:58.510 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.510 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.511 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.511 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.511 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.511 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.511 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.511 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.511 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.511 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.511 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.512 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.512 INFO:teuthology.orchestra.run.smithi060.stdout: "device_url": { 2024-03-04T12:35:58.512 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "device_url", 2024-03-04T12:35:58.512 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.512 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.512 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.512 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-04T12:35:58.512 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.513 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.513 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.513 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.513 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.513 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.513 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.513 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": { 2024-03-04T12:35:58.514 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "enabled", 2024-03-04T12:35:58.514 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.514 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.514 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.514 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.514 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.514 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.514 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.514 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.515 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.515 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.515 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.515 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:58.515 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:58.515 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.515 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.516 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.516 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "24", 2024-03-04T12:35:58.516 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "8", 2024-03-04T12:35:58.516 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.516 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.516 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.516 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.516 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.516 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.517 INFO:teuthology.orchestra.run.smithi060.stdout: "last_opt_revision": { 2024-03-04T12:35:58.517 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "last_opt_revision", 2024-03-04T12:35:58.517 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.517 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.517 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.517 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "1", 2024-03-04T12:35:58.517 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.518 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.518 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.518 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.518 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.518 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.518 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.518 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.518 INFO:teuthology.orchestra.run.smithi060.stdout: "leaderboard": { 2024-03-04T12:35:58.518 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "leaderboard", 2024-03-04T12:35:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.519 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: "leaderboard_description": { 2024-03-04T12:35:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "leaderboard_description", 2024-03-04T12:35:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.520 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.522 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.523 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.524 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.525 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.527 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.528 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.530 INFO:teuthology.orchestra.run.smithi060.stdout: "organization": { 2024-03-04T12:35:58.530 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "organization", 2024-03-04T12:35:58.530 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.530 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.530 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.530 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.530 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.530 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.531 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.531 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.531 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.531 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.531 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.531 INFO:teuthology.orchestra.run.smithi060.stdout: "proxy": { 2024-03-04T12:35:58.531 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "proxy", 2024-03-04T12:35:58.532 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.532 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.532 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.532 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.532 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.532 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.532 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.533 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.533 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.533 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.533 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.533 INFO:teuthology.orchestra.run.smithi060.stdout: "url": { 2024-03-04T12:35:58.533 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "url", 2024-03-04T12:35:58.533 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.533 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.534 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.534 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-04T12:35:58.534 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.534 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.534 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.534 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.534 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.534 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.534 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.535 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.535 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.535 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.535 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test_orchestrator", 2024-03-04T12:35:58.535 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.535 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.535 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.535 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.536 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.536 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.536 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.536 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.536 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.536 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.536 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.536 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.537 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.537 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.537 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.537 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.537 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.537 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.537 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.538 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.538 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.538 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.538 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.538 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.538 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.538 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.539 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.539 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.539 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.539 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.539 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.539 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.539 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.539 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.539 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.540 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.540 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.540 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.540 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.540 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.540 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.540 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.540 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.541 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.541 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.541 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.541 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.541 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.541 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.541 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.541 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.542 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.542 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.542 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.542 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.542 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.542 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.542 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.542 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.543 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.543 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.543 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.543 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.543 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.543 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.543 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.543 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.543 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.544 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.544 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.544 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.544 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.544 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "volumes", 2024-03-04T12:35:58.544 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.544 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.544 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.545 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.545 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.545 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.545 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.545 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.545 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.545 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.545 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.545 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.546 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.546 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.546 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.546 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.546 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.546 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.546 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.546 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.547 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.547 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.547 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.547 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.547 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.547 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.547 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.547 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.548 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.548 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.548 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.548 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.548 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.548 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.548 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.548 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.548 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.549 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.549 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.549 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.549 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.549 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.549 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.549 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.550 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.550 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.550 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.550 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.550 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.550 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.550 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.550 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.550 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.551 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.551 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.551 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.551 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.551 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.551 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.551 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.551 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.552 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.552 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.552 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.552 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.552 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.552 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.552 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.552 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.553 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.553 INFO:teuthology.orchestra.run.smithi060.stdout: "max_concurrent_clones": { 2024-03-04T12:35:58.553 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_concurrent_clones", 2024-03-04T12:35:58.553 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.553 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.553 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.553 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "4", 2024-03-04T12:35:58.553 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.554 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.554 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.554 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-04T12:35:58.554 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.554 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.554 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.554 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.554 INFO:teuthology.orchestra.run.smithi060.stdout: "periodic_async_work": { 2024-03-04T12:35:58.554 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "periodic_async_work", 2024-03-04T12:35:58.555 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.555 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.555 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.555 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.555 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.555 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.555 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.555 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Periodically check for async work", 2024-03-04T12:35:58.555 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.555 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.556 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.556 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.556 INFO:teuthology.orchestra.run.smithi060.stdout: "snapshot_clone_delay": { 2024-03-04T12:35:58.556 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "snapshot_clone_delay", 2024-03-04T12:35:58.556 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.556 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.556 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.556 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:58.556 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.557 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.557 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.557 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-04T12:35:58.557 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.557 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.557 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.557 INFO:teuthology.orchestra.run.smithi060.stdout: "snapshot_clone_no_wait": { 2024-03-04T12:35:58.557 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "snapshot_clone_no_wait", 2024-03-04T12:35:58.558 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.558 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.558 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.558 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.558 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.558 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.558 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.558 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-04T12:35:58.558 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.559 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.559 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.559 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.559 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.559 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.559 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "zabbix", 2024-03-04T12:35:58.560 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.560 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.560 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.560 INFO:teuthology.orchestra.run.smithi060.stdout: "discovery_interval": { 2024-03-04T12:35:58.560 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "discovery_interval", 2024-03-04T12:35:58.560 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:58.560 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.560 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.561 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "100", 2024-03-04T12:35:58.561 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.561 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.561 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.561 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.561 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.561 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.561 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.562 INFO:teuthology.orchestra.run.smithi060.stdout: "identifier": { 2024-03-04T12:35:58.562 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "identifier", 2024-03-04T12:35:58.562 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.562 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.562 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.562 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.562 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.562 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.562 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.563 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.563 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.563 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.563 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.563 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:58.563 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:58.563 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.564 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.564 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.564 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:58.564 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.564 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.564 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.564 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.564 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.565 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.565 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.565 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.565 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.565 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.565 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.565 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.565 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.566 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.566 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.566 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.566 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.566 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.566 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.566 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.566 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.567 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.567 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.567 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.567 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.567 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.567 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.567 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.568 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.568 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.568 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.568 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.568 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.568 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.568 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.568 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.569 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.569 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.569 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.569 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.569 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.569 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.569 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.569 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.570 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.570 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.570 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.570 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.570 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.570 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.570 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.570 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.571 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.571 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.571 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.571 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.571 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.571 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.571 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.571 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.572 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.572 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.572 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.572 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.572 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.572 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.572 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.572 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.573 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.573 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.576 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.577 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.577 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.577 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.577 INFO:teuthology.orchestra.run.smithi060.stdout: "zabbix_host": { 2024-03-04T12:35:58.577 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "zabbix_host", 2024-03-04T12:35:58.577 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.577 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.578 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.578 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.578 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.578 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.578 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.578 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.578 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.578 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.578 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.579 INFO:teuthology.orchestra.run.smithi060.stdout: "zabbix_port": { 2024-03-04T12:35:58.579 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "zabbix_port", 2024-03-04T12:35:58.579 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.579 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.579 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.579 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10051", 2024-03-04T12:35:58.579 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.580 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.580 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.580 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.580 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.580 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.580 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.580 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.580 INFO:teuthology.orchestra.run.smithi060.stdout: "zabbix_sender": { 2024-03-04T12:35:58.581 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "zabbix_sender", 2024-03-04T12:35:58.581 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.581 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.581 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.581 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-04T12:35:58.581 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.581 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.581 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.581 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.582 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.582 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.582 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.582 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.582 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.582 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.582 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-04T12:35:58.582 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.583 INFO:teuthology.orchestra.run.smithi060.stdout: "gid": 25306, 2024-03-04T12:35:58.583 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "z", 2024-03-04T12:35:58.583 INFO:teuthology.orchestra.run.smithi060.stdout: "mgr_features": 4540701547738038271, 2024-03-04T12:35:58.583 INFO:teuthology.orchestra.run.smithi060.stdout: "available_modules": [ 2024-03-04T12:35:58.583 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.583 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "alerts", 2024-03-04T12:35:58.583 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.584 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.584 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.584 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:58.584 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:58.584 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.584 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.584 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.584 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:58.585 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.585 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.585 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.585 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "How frequently to reexamine health status", 2024-03-04T12:35:58.585 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.585 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.585 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.585 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.586 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.586 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.586 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.586 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.586 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.586 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.586 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.586 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.587 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.587 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.587 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.587 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.587 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.587 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.587 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.587 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.588 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.588 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.588 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.588 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.588 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.588 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.588 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.589 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.589 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.589 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.589 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.589 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.589 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.589 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.589 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.590 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.590 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.590 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.590 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.590 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.590 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.590 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.590 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.590 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.591 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.591 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.591 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.591 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.591 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.591 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.591 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.591 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.592 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.592 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.592 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.592 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.592 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.592 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.592 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.593 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.593 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.593 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.593 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.593 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.593 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.593 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.593 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.593 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.594 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.594 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.594 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.594 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_destination": { 2024-03-04T12:35:58.594 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_destination", 2024-03-04T12:35:58.594 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.594 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.594 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.595 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.595 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.595 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.595 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.595 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Email address to send alerts to", 2024-03-04T12:35:58.595 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.595 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.595 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.595 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.596 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_from_name": { 2024-03-04T12:35:58.596 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_from_name", 2024-03-04T12:35:58.596 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.596 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.596 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.596 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "Ceph", 2024-03-04T12:35:58.596 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.596 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.596 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.597 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Email From: name", 2024-03-04T12:35:58.597 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.597 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.597 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.597 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_host": { 2024-03-04T12:35:58.597 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_host", 2024-03-04T12:35:58.597 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.598 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.598 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.598 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.598 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.598 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.598 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.599 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "SMTP server", 2024-03-04T12:35:58.599 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.599 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.599 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.599 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.599 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_password": { 2024-03-04T12:35:58.599 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_password", 2024-03-04T12:35:58.599 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.600 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.600 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.600 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.600 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.600 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.600 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.600 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Password to authenticate with", 2024-03-04T12:35:58.600 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.600 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.601 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.601 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.601 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_port": { 2024-03-04T12:35:58.601 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_port", 2024-03-04T12:35:58.601 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.601 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.601 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.601 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "465", 2024-03-04T12:35:58.602 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.602 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.602 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.602 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "SMTP port", 2024-03-04T12:35:58.602 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.602 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.602 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.602 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.603 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_sender": { 2024-03-04T12:35:58.603 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_sender", 2024-03-04T12:35:58.603 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.603 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.603 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.603 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.603 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.603 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.603 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.604 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "SMTP envelope sender", 2024-03-04T12:35:58.604 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.604 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.604 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.604 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.604 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_ssl": { 2024-03-04T12:35:58.604 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_ssl", 2024-03-04T12:35:58.604 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.604 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.605 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.605 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.605 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.605 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.605 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.605 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-04T12:35:58.605 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.605 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.606 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.606 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.606 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_user": { 2024-03-04T12:35:58.606 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_user", 2024-03-04T12:35:58.606 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.606 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.606 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.606 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.606 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.607 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.607 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.607 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "User to authenticate as", 2024-03-04T12:35:58.607 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.607 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.607 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.607 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.607 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.608 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.608 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "balancer", 2024-03-04T12:35:58.608 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.608 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.608 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.608 INFO:teuthology.orchestra.run.smithi060.stdout: "active": { 2024-03-04T12:35:58.608 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "active", 2024-03-04T12:35:58.608 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.608 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.609 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.609 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.609 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.609 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.609 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.609 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "automatically balance PGs across cluster", 2024-03-04T12:35:58.609 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.609 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.609 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.610 INFO:teuthology.orchestra.run.smithi060.stdout: "begin_time": { 2024-03-04T12:35:58.610 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "begin_time", 2024-03-04T12:35:58.610 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.610 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.610 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.610 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0000", 2024-03-04T12:35:58.610 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.611 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.611 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.611 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "beginning time of day to automatically balance", 2024-03-04T12:35:58.611 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-04T12:35:58.611 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.611 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.611 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.611 INFO:teuthology.orchestra.run.smithi060.stdout: "begin_weekday": { 2024-03-04T12:35:58.612 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "begin_weekday", 2024-03-04T12:35:58.612 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:58.612 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.612 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.612 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:58.612 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "0", 2024-03-04T12:35:58.612 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "6", 2024-03-04T12:35:58.612 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.613 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-04T12:35:58.613 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-04T12:35:58.613 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.613 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.613 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_compat_max_iterations": { 2024-03-04T12:35:58.613 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crush_compat_max_iterations", 2024-03-04T12:35:58.613 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:58.613 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.614 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.614 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "25", 2024-03-04T12:35:58.614 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1", 2024-03-04T12:35:58.614 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "250", 2024-03-04T12:35:58.614 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.614 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-04T12:35:58.614 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.614 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.615 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.615 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_compat_metrics": { 2024-03-04T12:35:58.615 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crush_compat_metrics", 2024-03-04T12:35:58.615 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.615 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.615 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.615 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "pgs,objects,bytes", 2024-03-04T12:35:58.615 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.616 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.616 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.616 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-04T12:35:58.616 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-04T12:35:58.616 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.616 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.616 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.616 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_compat_step": { 2024-03-04T12:35:58.616 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crush_compat_step", 2024-03-04T12:35:58.617 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:58.617 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.617 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.617 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0.5", 2024-03-04T12:35:58.617 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "0.001", 2024-03-04T12:35:58.617 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "0.999", 2024-03-04T12:35:58.617 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.617 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "aggressiveness of optimization", 2024-03-04T12:35:58.618 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-04T12:35:58.618 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.618 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.618 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.618 INFO:teuthology.orchestra.run.smithi060.stdout: "end_time": { 2024-03-04T12:35:58.618 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "end_time", 2024-03-04T12:35:58.618 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.618 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.618 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.619 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "2359", 2024-03-04T12:35:58.619 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.619 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.619 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.619 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "ending time of day to automatically balance", 2024-03-04T12:35:58.619 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-04T12:35:58.619 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.619 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.620 INFO:teuthology.orchestra.run.smithi060.stdout: "end_weekday": { 2024-03-04T12:35:58.620 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "end_weekday", 2024-03-04T12:35:58.620 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:58.620 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.620 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.620 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:58.620 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "0", 2024-03-04T12:35:58.620 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "6", 2024-03-04T12:35:58.621 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.621 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-04T12:35:58.621 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-04T12:35:58.621 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.621 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.621 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.621 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.621 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.622 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.622 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.622 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.622 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.622 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.622 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.622 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.622 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.622 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.623 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.623 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.623 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.623 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.623 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.623 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.623 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.623 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.624 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.624 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.624 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.624 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.624 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.624 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.624 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.624 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.624 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.625 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.625 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.625 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.625 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.625 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.625 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.625 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.625 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.626 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.626 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.626 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.626 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.626 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.626 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.626 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.626 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.626 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.627 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.627 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.627 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.627 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.627 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.627 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.627 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.627 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.628 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.628 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.628 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.628 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.628 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.628 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.628 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.628 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.628 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.629 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.629 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.629 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.629 INFO:teuthology.orchestra.run.smithi060.stdout: "min_score": { 2024-03-04T12:35:58.629 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "min_score", 2024-03-04T12:35:58.629 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:58.629 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.630 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.630 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:58.630 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.630 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.630 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.630 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-04T12:35:58.630 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.630 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.631 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.631 INFO:teuthology.orchestra.run.smithi060.stdout: "mode": { 2024-03-04T12:35:58.631 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mode", 2024-03-04T12:35:58.631 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.631 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.631 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.631 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "upmap", 2024-03-04T12:35:58.631 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.632 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.632 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.632 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-compat", 2024-03-04T12:35:58.632 INFO:teuthology.orchestra.run.smithi060.stdout: "none", 2024-03-04T12:35:58.632 INFO:teuthology.orchestra.run.smithi060.stdout: "read", 2024-03-04T12:35:58.632 INFO:teuthology.orchestra.run.smithi060.stdout: "upmap", 2024-03-04T12:35:58.632 INFO:teuthology.orchestra.run.smithi060.stdout: "upmap-read" 2024-03-04T12:35:58.632 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.633 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Balancer mode", 2024-03-04T12:35:58.633 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.633 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.633 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.633 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.633 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_ids": { 2024-03-04T12:35:58.633 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "pool_ids", 2024-03-04T12:35:58.633 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.634 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.634 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.634 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.634 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.634 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.634 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.634 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-04T12:35:58.634 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.634 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.635 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.635 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.635 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:58.635 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:58.635 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.635 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.635 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.635 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:58.636 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.636 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.636 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.636 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-04T12:35:58.636 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.636 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.636 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.636 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.636 INFO:teuthology.orchestra.run.smithi060.stdout: "upmap_max_deviation": { 2024-03-04T12:35:58.637 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "upmap_max_deviation", 2024-03-04T12:35:58.637 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.637 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.637 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.637 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5", 2024-03-04T12:35:58.637 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1", 2024-03-04T12:35:58.637 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.637 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.638 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-04T12:35:58.638 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-04T12:35:58.638 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.638 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.638 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.638 INFO:teuthology.orchestra.run.smithi060.stdout: "upmap_max_optimizations": { 2024-03-04T12:35:58.638 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "upmap_max_optimizations", 2024-03-04T12:35:58.638 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:58.638 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.639 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.639 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.639 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.639 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.639 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.639 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-04T12:35:58.639 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.639 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.639 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.640 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.640 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.640 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.640 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.640 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephadm", 2024-03-04T12:35:58.640 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.640 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.640 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.641 INFO:teuthology.orchestra.run.smithi060.stdout: "agent_down_multiplier": { 2024-03-04T12:35:58.641 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "agent_down_multiplier", 2024-03-04T12:35:58.641 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:58.641 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.641 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.641 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "3.0", 2024-03-04T12:35:58.641 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.641 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.641 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.642 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-04T12:35:58.642 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.642 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.642 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.642 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.642 INFO:teuthology.orchestra.run.smithi060.stdout: "agent_refresh_rate": { 2024-03-04T12:35:58.642 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "agent_refresh_rate", 2024-03-04T12:35:58.642 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.642 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.643 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.643 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "20", 2024-03-04T12:35:58.643 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.643 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.643 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.643 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-04T12:35:58.643 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.643 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.644 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.644 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.644 INFO:teuthology.orchestra.run.smithi060.stdout: "agent_starting_port": { 2024-03-04T12:35:58.644 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "agent_starting_port", 2024-03-04T12:35:58.644 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.644 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.644 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.644 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "4721", 2024-03-04T12:35:58.645 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.645 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.645 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.645 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-04T12:35:58.645 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.645 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.645 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.645 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: "allow_ptrace": { 2024-03-04T12:35:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "allow_ptrace", 2024-03-04T12:35:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.646 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-04T12:35:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-04T12:35:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: "autotune_interval": { 2024-03-04T12:35:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "autotune_interval", 2024-03-04T12:35:58.647 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-04T12:35:58.648 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.649 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.649 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.649 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.649 INFO:teuthology.orchestra.run.smithi060.stdout: "autotune_memory_target_ratio": { 2024-03-04T12:35:58.649 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "autotune_memory_target_ratio", 2024-03-04T12:35:58.649 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:58.649 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.649 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.650 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0.7", 2024-03-04T12:35:58.650 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.650 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.650 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.650 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-04T12:35:58.650 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.650 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.650 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.651 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.651 INFO:teuthology.orchestra.run.smithi060.stdout: "cephadm_log_destination": { 2024-03-04T12:35:58.651 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephadm_log_destination", 2024-03-04T12:35:58.651 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.651 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.651 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.652 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.652 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.652 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.652 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.652 INFO:teuthology.orchestra.run.smithi060.stdout: "file", 2024-03-04T12:35:58.652 INFO:teuthology.orchestra.run.smithi060.stdout: "file,syslog", 2024-03-04T12:35:58.652 INFO:teuthology.orchestra.run.smithi060.stdout: "syslog" 2024-03-04T12:35:58.652 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.653 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-04T12:35:58.653 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.653 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.653 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.653 INFO:teuthology.orchestra.run.smithi060.stdout: "cgroups_split": { 2024-03-04T12:35:58.653 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cgroups_split", 2024-03-04T12:35:58.653 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.654 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.654 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.654 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.654 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.654 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.654 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.654 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-04T12:35:58.654 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.654 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.655 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.655 INFO:teuthology.orchestra.run.smithi060.stdout: "config_checks_enabled": { 2024-03-04T12:35:58.655 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "config_checks_enabled", 2024-03-04T12:35:58.655 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.655 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.655 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.656 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.656 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.656 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.656 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.656 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-04T12:35:58.656 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.656 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.656 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.656 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.657 INFO:teuthology.orchestra.run.smithi060.stdout: "config_dashboard": { 2024-03-04T12:35:58.657 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "config_dashboard", 2024-03-04T12:35:58.657 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.657 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.657 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.657 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.657 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.657 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.658 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.658 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-04T12:35:58.658 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.658 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.658 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.658 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.658 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_alertmanager": { 2024-03-04T12:35:58.658 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_alertmanager", 2024-03-04T12:35:58.658 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.659 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.659 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.659 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-04T12:35:58.659 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.659 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.659 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.659 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Prometheus container image", 2024-03-04T12:35:58.659 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.659 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.660 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.660 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.660 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_base": { 2024-03-04T12:35:58.660 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_base", 2024-03-04T12:35:58.660 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.660 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.660 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.660 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-04T12:35:58.661 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.661 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.661 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.661 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Container image name, without the tag", 2024-03-04T12:35:58.661 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.661 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.661 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.662 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_elasticsearch": { 2024-03-04T12:35:58.662 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_elasticsearch", 2024-03-04T12:35:58.662 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.662 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.662 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.662 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-04T12:35:58.662 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.662 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.662 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.663 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "elasticsearch container image", 2024-03-04T12:35:58.663 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.663 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.663 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.663 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.663 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_grafana": { 2024-03-04T12:35:58.663 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_grafana", 2024-03-04T12:35:58.663 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.663 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.663 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.664 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-04T12:35:58.664 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.664 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.664 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.664 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Prometheus container image", 2024-03-04T12:35:58.664 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.664 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.664 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.665 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.665 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_haproxy": { 2024-03-04T12:35:58.665 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_haproxy", 2024-03-04T12:35:58.665 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.665 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.665 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.665 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-04T12:35:58.665 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.665 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.666 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.666 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "HAproxy container image", 2024-03-04T12:35:58.666 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.666 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.666 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.666 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_jaeger_agent": { 2024-03-04T12:35:58.666 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_jaeger_agent", 2024-03-04T12:35:58.667 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.667 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.667 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.667 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-04T12:35:58.667 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.667 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.667 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.667 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Jaeger agent container image", 2024-03-04T12:35:58.667 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.668 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.668 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.668 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_jaeger_collector": { 2024-03-04T12:35:58.668 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_jaeger_collector", 2024-03-04T12:35:58.668 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.668 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.668 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.669 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-04T12:35:58.669 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.669 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.669 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.669 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Jaeger collector container image", 2024-03-04T12:35:58.669 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.669 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.669 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.669 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.670 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_jaeger_query": { 2024-03-04T12:35:58.670 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_jaeger_query", 2024-03-04T12:35:58.670 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.670 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.670 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.670 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-04T12:35:58.670 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.670 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.671 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.671 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Jaeger query container image", 2024-03-04T12:35:58.671 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.671 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.671 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.671 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_keepalived": { 2024-03-04T12:35:58.671 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_keepalived", 2024-03-04T12:35:58.671 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.672 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.672 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.672 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-04T12:35:58.672 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.672 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.672 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.672 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Keepalived container image", 2024-03-04T12:35:58.672 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.673 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.673 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.673 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.673 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_loki": { 2024-03-04T12:35:58.673 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_loki", 2024-03-04T12:35:58.673 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.673 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.673 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.673 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-04T12:35:58.674 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.674 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.674 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.674 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Loki container image", 2024-03-04T12:35:58.674 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.674 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_node_exporter": { 2024-03-04T12:35:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_node_exporter", 2024-03-04T12:35:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.675 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.676 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.676 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-04T12:35:58.676 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.676 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.676 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.676 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Prometheus container image", 2024-03-04T12:35:58.676 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.676 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.677 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.677 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.677 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_nvmeof": { 2024-03-04T12:35:58.677 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_nvmeof", 2024-03-04T12:35:58.677 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.677 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.677 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.677 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-04T12:35:58.678 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.678 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.678 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.678 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Nvme-of container image", 2024-03-04T12:35:58.678 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.678 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.678 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.679 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_prometheus": { 2024-03-04T12:35:58.679 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_prometheus", 2024-03-04T12:35:58.679 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.679 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.679 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.679 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-04T12:35:58.679 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.679 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.679 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.680 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Prometheus container image", 2024-03-04T12:35:58.680 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.680 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.680 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.680 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_promtail": { 2024-03-04T12:35:58.680 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_promtail", 2024-03-04T12:35:58.680 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.681 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.681 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.681 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-04T12:35:58.681 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.681 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.681 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.681 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Promtail container image", 2024-03-04T12:35:58.681 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.681 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.682 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.682 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.682 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_snmp_gateway": { 2024-03-04T12:35:58.682 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_snmp_gateway", 2024-03-04T12:35:58.682 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.682 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.682 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.682 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-04T12:35:58.683 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.683 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.683 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.683 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "SNMP Gateway container image", 2024-03-04T12:35:58.683 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.683 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.683 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.684 INFO:teuthology.orchestra.run.smithi060.stdout: "container_init": { 2024-03-04T12:35:58.684 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_init", 2024-03-04T12:35:58.684 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.684 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.684 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.684 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.684 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.684 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.685 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.685 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Run podman/docker with `--init`", 2024-03-04T12:35:58.685 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.685 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.685 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.685 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.685 INFO:teuthology.orchestra.run.smithi060.stdout: "daemon_cache_timeout": { 2024-03-04T12:35:58.685 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "daemon_cache_timeout", 2024-03-04T12:35:58.685 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.686 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.686 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.686 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:58.686 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.686 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.686 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.686 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-04T12:35:58.686 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.687 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.687 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.687 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.687 INFO:teuthology.orchestra.run.smithi060.stdout: "default_cephadm_command_timeout": { 2024-03-04T12:35:58.687 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default_cephadm_command_timeout", 2024-03-04T12:35:58.687 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.687 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.687 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.687 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "900", 2024-03-04T12:35:58.687 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.688 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.688 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.688 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-04T12:35:58.688 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.688 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.688 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.688 INFO:teuthology.orchestra.run.smithi060.stdout: "default_registry": { 2024-03-04T12:35:58.689 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default_registry", 2024-03-04T12:35:58.689 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.689 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.689 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.689 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "docker.io", 2024-03-04T12:35:58.689 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.689 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.689 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.689 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-04T12:35:58.690 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.690 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.690 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.690 INFO:teuthology.orchestra.run.smithi060.stdout: "device_cache_timeout": { 2024-03-04T12:35:58.690 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "device_cache_timeout", 2024-03-04T12:35:58.690 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.690 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.691 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.691 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "1800", 2024-03-04T12:35:58.691 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.691 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.691 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.691 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "seconds to cache device inventory", 2024-03-04T12:35:58.691 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.691 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.691 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.692 INFO:teuthology.orchestra.run.smithi060.stdout: "device_enhanced_scan": { 2024-03-04T12:35:58.692 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "device_enhanced_scan", 2024-03-04T12:35:58.692 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.692 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.692 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.692 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.692 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.693 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.693 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.693 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-04T12:35:58.693 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.693 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.693 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.693 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.693 INFO:teuthology.orchestra.run.smithi060.stdout: "facts_cache_timeout": { 2024-03-04T12:35:58.693 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "facts_cache_timeout", 2024-03-04T12:35:58.694 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.694 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.694 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.694 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:58.694 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.694 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.694 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.694 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "seconds to cache host facts data", 2024-03-04T12:35:58.694 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.695 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.695 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.695 INFO:teuthology.orchestra.run.smithi060.stdout: "host_check_interval": { 2024-03-04T12:35:58.695 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "host_check_interval", 2024-03-04T12:35:58.695 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.695 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.695 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.695 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:58.696 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.696 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.696 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.696 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to perform a host check", 2024-03-04T12:35:58.696 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.696 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.696 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.696 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.696 INFO:teuthology.orchestra.run.smithi060.stdout: "hw_monitoring": { 2024-03-04T12:35:58.697 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "hw_monitoring", 2024-03-04T12:35:58.697 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.697 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.697 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.697 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.697 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.697 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.697 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.697 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-04T12:35:58.698 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.698 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.698 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.698 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.698 INFO:teuthology.orchestra.run.smithi060.stdout: "inventory_list_all": { 2024-03-04T12:35:58.698 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "inventory_list_all", 2024-03-04T12:35:58.698 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.698 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.698 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.699 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.699 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.699 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.699 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.699 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-04T12:35:58.699 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.699 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.699 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.700 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.700 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.700 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.700 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.700 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.700 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.700 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.700 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.700 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.701 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.701 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.701 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.701 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.701 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.701 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.701 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.701 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.702 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.702 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.702 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.702 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.702 INFO:teuthology.orchestra.run.smithi060.stdout: "log_refresh_metadata": { 2024-03-04T12:35:58.702 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_refresh_metadata", 2024-03-04T12:35:58.702 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.702 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.702 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.702 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.703 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.703 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.703 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.703 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-04T12:35:58.703 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.703 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.703 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.704 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.704 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.704 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.704 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.704 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.704 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.704 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.704 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.704 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.705 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-04T12:35:58.705 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.705 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.705 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.705 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.705 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.705 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.705 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.706 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.706 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.706 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.706 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.706 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.706 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.706 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.706 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.706 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.707 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.707 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.707 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.707 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.707 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.707 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.707 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.707 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.708 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.708 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.708 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.708 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.708 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.708 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.708 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.708 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.709 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.709 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.709 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.709 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.709 INFO:teuthology.orchestra.run.smithi060.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-04T12:35:58.709 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-04T12:35:58.709 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.709 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.710 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.710 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.710 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.710 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.710 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.710 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-04T12:35:58.710 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.710 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.710 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.711 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.711 INFO:teuthology.orchestra.run.smithi060.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-04T12:35:58.711 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-04T12:35:58.711 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.711 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.711 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.711 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "*", 2024-03-04T12:35:58.711 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.712 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.712 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.712 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-04T12:35:58.712 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.712 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.712 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.712 INFO:teuthology.orchestra.run.smithi060.stdout: "max_count_per_host": { 2024-03-04T12:35:58.713 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_count_per_host", 2024-03-04T12:35:58.713 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.713 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.713 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.713 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.713 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.713 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.713 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.713 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "max number of daemons per service per host", 2024-03-04T12:35:58.714 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.714 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.714 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.714 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.714 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd_draining_count": { 2024-03-04T12:35:58.714 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_osd_draining_count", 2024-03-04T12:35:58.714 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.714 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.714 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.715 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.715 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.715 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.715 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.715 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-04T12:35:58.715 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.715 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.715 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.715 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.716 INFO:teuthology.orchestra.run.smithi060.stdout: "migration_current": { 2024-03-04T12:35:58.716 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "migration_current", 2024-03-04T12:35:58.716 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.716 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.716 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.716 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.716 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.717 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.717 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.717 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "internal - do not modify", 2024-03-04T12:35:58.717 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.717 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.717 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.717 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.717 INFO:teuthology.orchestra.run.smithi060.stdout: "mode": { 2024-03-04T12:35:58.717 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mode", 2024-03-04T12:35:58.718 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.718 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.718 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.718 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "root", 2024-03-04T12:35:58.718 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.718 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.718 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.718 INFO:teuthology.orchestra.run.smithi060.stdout: "cephadm-package", 2024-03-04T12:35:58.719 INFO:teuthology.orchestra.run.smithi060.stdout: "root" 2024-03-04T12:35:58.719 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.719 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "mode for remote execution of cephadm", 2024-03-04T12:35:58.719 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.719 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.719 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.719 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.719 INFO:teuthology.orchestra.run.smithi060.stdout: "oob_default_addr": { 2024-03-04T12:35:58.720 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "oob_default_addr", 2024-03-04T12:35:58.720 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.720 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.720 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.720 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "169.254.1.1", 2024-03-04T12:35:58.720 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.720 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.720 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.721 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-04T12:35:58.721 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.721 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.721 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.721 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.721 INFO:teuthology.orchestra.run.smithi060.stdout: "prometheus_alerts_path": { 2024-03-04T12:35:58.721 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "prometheus_alerts_path", 2024-03-04T12:35:58.721 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.721 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.722 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.722 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-04T12:35:58.722 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.722 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.722 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.722 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-04T12:35:58.722 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.722 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.722 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.723 INFO:teuthology.orchestra.run.smithi060.stdout: "registry_insecure": { 2024-03-04T12:35:58.723 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "registry_insecure", 2024-03-04T12:35:58.723 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.723 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.723 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.723 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.723 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.724 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.724 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.724 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-04T12:35:58.724 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.724 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.724 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.724 INFO:teuthology.orchestra.run.smithi060.stdout: "registry_password": { 2024-03-04T12:35:58.724 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "registry_password", 2024-03-04T12:35:58.725 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.725 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.725 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.725 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.725 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.725 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.725 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.725 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-04T12:35:58.726 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.726 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.726 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.726 INFO:teuthology.orchestra.run.smithi060.stdout: "registry_url": { 2024-03-04T12:35:58.726 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "registry_url", 2024-03-04T12:35:58.726 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.727 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.727 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.727 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.727 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.727 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.727 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.727 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-04T12:35:58.727 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.727 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.728 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.728 INFO:teuthology.orchestra.run.smithi060.stdout: "registry_username": { 2024-03-04T12:35:58.728 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "registry_username", 2024-03-04T12:35:58.728 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.728 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.728 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.728 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.728 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.729 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.729 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.729 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-04T12:35:58.729 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.729 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.729 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.729 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.729 INFO:teuthology.orchestra.run.smithi060.stdout: "secure_monitoring_stack": { 2024-03-04T12:35:58.730 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "secure_monitoring_stack", 2024-03-04T12:35:58.730 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.730 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.730 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.730 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.730 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.730 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.730 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.730 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-04T12:35:58.731 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.731 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.731 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.731 INFO:teuthology.orchestra.run.smithi060.stdout: "service_discovery_port": { 2024-03-04T12:35:58.731 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "service_discovery_port", 2024-03-04T12:35:58.731 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.731 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.732 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.732 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8765", 2024-03-04T12:35:58.732 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.732 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.732 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.732 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "cephadm service discovery port", 2024-03-04T12:35:58.732 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.732 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.733 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.733 INFO:teuthology.orchestra.run.smithi060.stdout: "ssh_config_file": { 2024-03-04T12:35:58.733 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ssh_config_file", 2024-03-04T12:35:58.733 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.733 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.733 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.733 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.733 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.734 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.734 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.734 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-04T12:35:58.734 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.734 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.734 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.734 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.734 INFO:teuthology.orchestra.run.smithi060.stdout: "use_agent": { 2024-03-04T12:35:58.734 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "use_agent", 2024-03-04T12:35:58.735 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.735 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.735 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.735 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.735 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.735 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.735 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.736 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-04T12:35:58.736 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.736 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.736 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.736 INFO:teuthology.orchestra.run.smithi060.stdout: "use_repo_digest": { 2024-03-04T12:35:58.736 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "use_repo_digest", 2024-03-04T12:35:58.736 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.737 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.737 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.737 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.737 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.737 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.737 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.737 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-04T12:35:58.737 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.737 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.738 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.738 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_on_failed_host_check": { 2024-03-04T12:35:58.738 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_on_failed_host_check", 2024-03-04T12:35:58.738 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.738 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.738 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.738 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.739 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.739 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.739 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.739 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "raise a health warning if the host check fails", 2024-03-04T12:35:58.739 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.739 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.739 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.739 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.739 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_on_stray_daemons": { 2024-03-04T12:35:58.740 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_on_stray_daemons", 2024-03-04T12:35:58.740 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.740 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.740 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.740 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.740 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.740 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.740 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.741 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-04T12:35:58.741 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.741 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.741 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.741 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_on_stray_hosts": { 2024-03-04T12:35:58.741 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_on_stray_hosts", 2024-03-04T12:35:58.741 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.741 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.742 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.742 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.742 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.742 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.742 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.742 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-04T12:35:58.742 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.742 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.742 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.743 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.743 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.743 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crash", 2024-03-04T12:35:58.743 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.743 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.743 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.744 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.744 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.744 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.744 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.744 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.744 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.744 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.744 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.745 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.745 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.745 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.745 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.745 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.745 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.745 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.745 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.745 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.746 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.747 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.748 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.748 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.748 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.748 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.748 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.748 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.748 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.748 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.749 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.749 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.749 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.749 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.749 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.749 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.749 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.749 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.749 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.750 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.750 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.750 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.750 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.750 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.750 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.750 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.750 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.750 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.751 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.751 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.751 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.751 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.751 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.751 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.751 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.751 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.752 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.752 INFO:teuthology.orchestra.run.smithi060.stdout: "retain_interval": { 2024-03-04T12:35:58.752 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "retain_interval", 2024-03-04T12:35:58.752 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.752 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.752 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.752 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "31536000", 2024-03-04T12:35:58.752 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.753 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.753 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.753 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-04T12:35:58.753 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.753 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.753 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.753 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_recent_interval": { 2024-03-04T12:35:58.754 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_recent_interval", 2024-03-04T12:35:58.754 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.754 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.754 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.754 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "1209600", 2024-03-04T12:35:58.754 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.754 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.754 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.755 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-04T12:35:58.755 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.755 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.755 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.755 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.755 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.755 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.755 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.755 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "dashboard", 2024-03-04T12:35:58.756 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.756 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.756 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.756 INFO:teuthology.orchestra.run.smithi060.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-04T12:35:58.756 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-04T12:35:58.756 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.756 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.756 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.757 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.757 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.757 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.757 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.757 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.757 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.757 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.757 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.757 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.758 INFO:teuthology.orchestra.run.smithi060.stdout: "ALERTMANAGER_API_HOST": { 2024-03-04T12:35:58.758 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-04T12:35:58.758 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.758 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.758 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.758 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.758 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.758 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.759 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.759 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.759 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.759 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.759 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.759 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.759 INFO:teuthology.orchestra.run.smithi060.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-04T12:35:58.759 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-04T12:35:58.760 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.760 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.760 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.760 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.760 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.760 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.760 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.760 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.761 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.761 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.761 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.761 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.761 INFO:teuthology.orchestra.run.smithi060.stdout: "AUDIT_API_ENABLED": { 2024-03-04T12:35:58.761 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "AUDIT_API_ENABLED", 2024-03-04T12:35:58.761 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.761 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.761 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.762 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.762 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.762 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.762 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.762 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.762 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.762 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.762 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.763 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.763 INFO:teuthology.orchestra.run.smithi060.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-04T12:35:58.763 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-04T12:35:58.763 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.763 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.763 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.763 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.763 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.764 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.764 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.764 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.764 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.764 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.764 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.764 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.764 INFO:teuthology.orchestra.run.smithi060.stdout: "ENABLE_BROWSABLE_API": { 2024-03-04T12:35:58.765 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-04T12:35:58.765 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.765 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.765 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.765 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.765 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.765 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.765 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.765 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.766 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.766 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.766 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.766 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.766 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-04T12:35:58.766 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-04T12:35:58.766 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.766 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.766 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.767 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.767 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.767 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.767 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.767 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.767 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.767 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.767 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.768 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-04T12:35:58.768 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-04T12:35:58.768 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.768 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.768 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.768 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.768 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.768 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.769 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.769 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.769 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.769 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.769 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.769 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-04T12:35:58.769 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-04T12:35:58.769 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.769 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.770 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.770 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.770 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.770 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.770 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.770 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.770 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.770 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.770 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.771 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.771 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-04T12:35:58.771 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-04T12:35:58.771 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.771 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.771 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.771 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.771 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.772 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.772 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.772 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.772 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.772 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.772 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.773 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-04T12:35:58.773 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-04T12:35:58.773 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.773 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.773 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.773 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.773 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.773 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.773 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.773 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.774 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.774 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.774 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.774 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.774 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-04T12:35:58.774 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-04T12:35:58.774 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.774 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.774 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.775 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.775 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.775 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.775 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.775 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.775 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.775 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.775 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.775 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.776 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-04T12:35:58.776 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-04T12:35:58.776 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.776 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.776 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.776 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.776 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.776 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.779 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.780 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.780 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.780 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.780 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.780 INFO:teuthology.orchestra.run.smithi060.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-04T12:35:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-04T12:35:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.781 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_API_PASSWORD": { 2024-03-04T12:35:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-04T12:35:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.782 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.783 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.783 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "admin", 2024-03-04T12:35:58.783 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.783 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.783 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.783 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.783 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.783 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.784 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.784 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.784 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-04T12:35:58.784 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-04T12:35:58.784 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.784 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.784 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.784 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.784 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.785 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.785 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.785 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.785 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.785 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.785 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.785 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_API_URL": { 2024-03-04T12:35:58.785 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_API_URL", 2024-03-04T12:35:58.786 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.786 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.786 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.786 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.786 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.786 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.786 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.786 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.786 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.787 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.787 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.787 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.787 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_API_USERNAME": { 2024-03-04T12:35:58.787 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-04T12:35:58.787 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.787 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.787 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.788 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "admin", 2024-03-04T12:35:58.788 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.788 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.788 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.788 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.788 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.788 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.788 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.789 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-04T12:35:58.789 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-04T12:35:58.789 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.789 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.789 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.789 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.789 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.789 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.789 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.789 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.790 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.790 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.790 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.790 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-04T12:35:58.790 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-04T12:35:58.790 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.790 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.791 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.791 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.791 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.791 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.791 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.791 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.791 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.791 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.792 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.792 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.792 INFO:teuthology.orchestra.run.smithi060.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-04T12:35:58.792 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-04T12:35:58.792 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.792 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.792 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.792 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.792 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.793 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.793 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.793 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.793 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.793 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.793 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.793 INFO:teuthology.orchestra.run.smithi060.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-04T12:35:58.793 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-04T12:35:58.793 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.794 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.794 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.794 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.794 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.794 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.794 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.794 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.794 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.795 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.795 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.795 INFO:teuthology.orchestra.run.smithi060.stdout: "PROMETHEUS_API_HOST": { 2024-03-04T12:35:58.795 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-04T12:35:58.795 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.795 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.795 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.795 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.796 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.796 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.796 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.796 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.796 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.796 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.796 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.796 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.797 INFO:teuthology.orchestra.run.smithi060.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-04T12:35:58.797 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-04T12:35:58.797 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.797 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.797 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.797 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.797 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.797 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.797 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.798 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.798 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.798 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.798 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.798 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-04T12:35:58.798 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-04T12:35:58.798 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.798 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.799 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.799 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.799 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.799 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.799 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.799 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.799 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.799 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.800 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.800 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-04T12:35:58.800 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-04T12:35:58.800 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.800 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.800 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.800 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.801 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.801 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.801 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.801 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.801 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.801 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.801 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.801 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.801 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-04T12:35:58.802 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-04T12:35:58.802 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.802 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.802 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.802 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.802 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.802 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.802 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.803 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.803 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.803 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.803 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.803 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.803 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-04T12:35:58.803 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-04T12:35:58.803 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.803 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.804 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.804 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.804 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.804 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.804 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.804 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.804 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.805 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.805 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.805 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.805 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-04T12:35:58.805 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-04T12:35:58.805 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.805 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.805 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.805 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.806 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.806 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.806 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.806 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.806 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.806 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.806 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.806 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-04T12:35:58.807 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-04T12:35:58.807 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.807 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.807 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.807 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.807 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.807 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.807 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.807 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.808 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.808 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.808 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.808 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.808 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-04T12:35:58.808 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-04T12:35:58.808 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.808 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.808 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.809 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.809 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.809 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.809 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.809 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.809 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.809 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.809 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.810 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_ENABLED": { 2024-03-04T12:35:58.810 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-04T12:35:58.810 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.810 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.810 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.810 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.810 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.810 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.810 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.811 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.811 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.811 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.811 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.811 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.811 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-04T12:35:58.811 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-04T12:35:58.811 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.811 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.812 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.812 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-04T12:35:58.812 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.812 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.812 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.812 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.812 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.812 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.813 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.813 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.813 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-04T12:35:58.813 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-04T12:35:58.813 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.813 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.813 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.813 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.813 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.814 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.814 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.814 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.814 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.814 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.814 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.814 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-04T12:35:58.814 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-04T12:35:58.815 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.815 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.815 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.815 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8", 2024-03-04T12:35:58.815 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.815 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.815 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.815 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.815 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.816 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.816 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.816 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.816 INFO:teuthology.orchestra.run.smithi060.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-04T12:35:58.816 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-04T12:35:58.816 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.816 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.816 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.816 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "45", 2024-03-04T12:35:58.817 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.817 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.817 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.817 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.817 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.817 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.817 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.817 INFO:teuthology.orchestra.run.smithi060.stdout: "RGW_API_ACCESS_KEY": { 2024-03-04T12:35:58.818 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-04T12:35:58.818 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.818 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.818 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.818 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.818 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.818 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.818 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.818 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.819 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.819 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.819 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.819 INFO:teuthology.orchestra.run.smithi060.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-04T12:35:58.819 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-04T12:35:58.819 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.819 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.820 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.820 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "admin", 2024-03-04T12:35:58.820 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.820 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.820 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.820 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.820 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.820 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.820 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.821 INFO:teuthology.orchestra.run.smithi060.stdout: "RGW_API_SECRET_KEY": { 2024-03-04T12:35:58.821 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-04T12:35:58.821 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.821 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.821 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.821 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.821 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.822 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.822 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.822 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.822 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.822 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.822 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.822 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.822 INFO:teuthology.orchestra.run.smithi060.stdout: "RGW_API_SSL_VERIFY": { 2024-03-04T12:35:58.822 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-04T12:35:58.823 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.823 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.823 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.823 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.823 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.823 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.823 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.823 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.823 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.824 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.824 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.824 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.824 INFO:teuthology.orchestra.run.smithi060.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-04T12:35:58.824 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-04T12:35:58.824 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.824 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.824 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.825 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:58.825 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.825 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.825 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.825 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.825 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.825 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.825 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.826 INFO:teuthology.orchestra.run.smithi060.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-04T12:35:58.826 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-04T12:35:58.826 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.826 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.826 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.826 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.826 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.826 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.826 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.827 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.827 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.827 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.827 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.827 INFO:teuthology.orchestra.run.smithi060.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-04T12:35:58.827 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-04T12:35:58.827 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.827 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.828 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.828 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5", 2024-03-04T12:35:58.828 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.828 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.828 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.828 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.828 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.828 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.829 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.829 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.829 INFO:teuthology.orchestra.run.smithi060.stdout: "cross_origin_url": { 2024-03-04T12:35:58.829 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cross_origin_url", 2024-03-04T12:35:58.829 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.829 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.829 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.829 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.830 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.830 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.830 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.830 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.830 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.830 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.830 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.830 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.830 INFO:teuthology.orchestra.run.smithi060.stdout: "crt_file": { 2024-03-04T12:35:58.831 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crt_file", 2024-03-04T12:35:58.831 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.831 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.831 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.831 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.831 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.831 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.831 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.832 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.832 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.832 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.832 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.832 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.832 INFO:teuthology.orchestra.run.smithi060.stdout: "debug": { 2024-03-04T12:35:58.832 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "debug", 2024-03-04T12:35:58.832 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.832 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.833 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.833 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.833 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.833 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.833 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.833 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Enable/disable debug options", 2024-03-04T12:35:58.833 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.833 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.833 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.834 INFO:teuthology.orchestra.run.smithi060.stdout: "jwt_token_ttl": { 2024-03-04T12:35:58.834 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "jwt_token_ttl", 2024-03-04T12:35:58.834 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.834 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.834 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.834 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "28800", 2024-03-04T12:35:58.834 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.834 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.835 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.835 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.835 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.835 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.835 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.835 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.835 INFO:teuthology.orchestra.run.smithi060.stdout: "key_file": { 2024-03-04T12:35:58.835 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "key_file", 2024-03-04T12:35:58.835 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.836 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.836 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.836 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.836 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.836 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.836 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.836 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.836 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.837 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.837 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.837 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.837 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.837 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.837 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.837 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.837 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.838 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.838 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.838 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.838 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.838 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.838 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.838 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.838 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.839 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.839 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.839 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.839 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.839 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.839 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.839 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.840 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.840 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.840 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.840 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.840 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.840 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.840 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.841 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.841 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.841 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.841 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.841 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.841 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.841 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.841 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.841 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.842 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.842 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.842 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.842 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.842 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.842 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.842 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.842 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.842 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.843 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.843 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.843 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.843 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.843 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.843 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.843 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.843 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.844 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.844 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.844 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.844 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.844 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.844 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.844 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.844 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.844 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.845 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.845 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.845 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.845 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.845 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.845 INFO:teuthology.orchestra.run.smithi060.stdout: "motd": { 2024-03-04T12:35:58.845 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "motd", 2024-03-04T12:35:58.845 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.846 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.846 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.846 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.846 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.846 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.846 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.846 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "The message of the day", 2024-03-04T12:35:58.846 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.846 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.846 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.847 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.847 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_resolve_ip_addr": { 2024-03-04T12:35:58.847 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "redirect_resolve_ip_addr", 2024-03-04T12:35:58.847 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.847 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.847 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.847 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.847 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.847 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.848 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.848 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.848 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.848 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.848 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.848 INFO:teuthology.orchestra.run.smithi060.stdout: "server_addr": { 2024-03-04T12:35:58.848 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_addr", 2024-03-04T12:35:58.848 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.849 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.849 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.849 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "::", 2024-03-04T12:35:58.849 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.849 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.849 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.849 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.849 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.849 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.850 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.850 INFO:teuthology.orchestra.run.smithi060.stdout: "server_port": { 2024-03-04T12:35:58.850 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_port", 2024-03-04T12:35:58.850 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.850 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.850 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.850 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8080", 2024-03-04T12:35:58.850 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.851 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.851 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.851 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.851 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.851 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.851 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.851 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.851 INFO:teuthology.orchestra.run.smithi060.stdout: "ssl": { 2024-03-04T12:35:58.851 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ssl", 2024-03-04T12:35:58.852 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.852 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.852 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.852 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.852 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.852 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.852 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.853 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.853 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.853 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.853 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.853 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.853 INFO:teuthology.orchestra.run.smithi060.stdout: "ssl_server_port": { 2024-03-04T12:35:58.853 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ssl_server_port", 2024-03-04T12:35:58.853 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.853 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.854 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.854 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8443", 2024-03-04T12:35:58.854 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.854 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.854 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.854 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.854 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.854 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.855 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.855 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.855 INFO:teuthology.orchestra.run.smithi060.stdout: "standby_behaviour": { 2024-03-04T12:35:58.855 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "standby_behaviour", 2024-03-04T12:35:58.855 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.855 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.855 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.855 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "redirect", 2024-03-04T12:35:58.855 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.856 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.856 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.856 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.856 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect" 2024-03-04T12:35:58.856 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.856 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.856 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.856 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.857 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.857 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.857 INFO:teuthology.orchestra.run.smithi060.stdout: "standby_error_status_code": { 2024-03-04T12:35:58.857 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "standby_error_status_code", 2024-03-04T12:35:58.857 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.857 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.857 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.857 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "500", 2024-03-04T12:35:58.857 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "400", 2024-03-04T12:35:58.858 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "599", 2024-03-04T12:35:58.858 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.858 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.858 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.858 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.858 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.858 INFO:teuthology.orchestra.run.smithi060.stdout: "url_prefix": { 2024-03-04T12:35:58.859 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "url_prefix", 2024-03-04T12:35:58.859 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.859 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.859 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.859 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.859 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.859 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.859 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.859 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.860 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.860 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.860 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.860 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.860 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.861 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "devicehealth", 2024-03-04T12:35:58.861 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.861 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.861 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.861 INFO:teuthology.orchestra.run.smithi060.stdout: "enable_monitoring": { 2024-03-04T12:35:58.861 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "enable_monitoring", 2024-03-04T12:35:58.861 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.861 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.862 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.862 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.862 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.862 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.862 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.862 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "monitor device health metrics", 2024-03-04T12:35:58.862 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.862 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.862 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.863 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.863 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.863 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.863 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.863 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.863 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.863 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.863 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.863 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.864 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.864 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.864 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.864 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.864 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.864 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.864 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.864 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.864 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.865 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.865 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.865 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.865 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.865 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.865 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.865 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.865 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.865 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.866 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.866 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.866 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.866 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.866 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.866 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.866 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.867 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.867 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.867 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.867 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.867 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.867 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.867 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.867 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.867 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.868 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.869 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.870 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.870 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.870 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.870 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.870 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.870 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.870 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.870 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.871 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.871 INFO:teuthology.orchestra.run.smithi060.stdout: "mark_out_threshold": { 2024-03-04T12:35:58.871 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mark_out_threshold", 2024-03-04T12:35:58.871 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.871 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.871 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.871 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "2419200", 2024-03-04T12:35:58.871 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.871 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.872 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.872 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-04T12:35:58.872 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.872 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.872 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.872 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": { 2024-03-04T12:35:58.872 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "pool_name", 2024-03-04T12:35:58.872 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.873 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.873 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.873 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "device_health_metrics", 2024-03-04T12:35:58.873 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.873 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.873 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.873 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-04T12:35:58.873 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.873 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.874 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.874 INFO:teuthology.orchestra.run.smithi060.stdout: "retention_period": { 2024-03-04T12:35:58.874 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "retention_period", 2024-03-04T12:35:58.874 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.874 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.874 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.874 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "15552000", 2024-03-04T12:35:58.874 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.875 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.875 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.875 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how long to retain device health metrics", 2024-03-04T12:35:58.875 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.875 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.875 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.875 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.875 INFO:teuthology.orchestra.run.smithi060.stdout: "scrape_frequency": { 2024-03-04T12:35:58.876 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "scrape_frequency", 2024-03-04T12:35:58.876 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.876 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.876 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.876 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "86400", 2024-03-04T12:35:58.876 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.876 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.876 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.876 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-04T12:35:58.877 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.877 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.877 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.877 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.877 INFO:teuthology.orchestra.run.smithi060.stdout: "self_heal": { 2024-03-04T12:35:58.877 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "self_heal", 2024-03-04T12:35:58.877 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.877 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.877 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.878 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:58.878 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.878 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.878 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.878 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-04T12:35:58.878 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.878 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.878 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.879 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:58.879 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:58.879 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.879 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.879 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.879 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:58.879 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.879 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.879 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.880 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to wake up and check device health", 2024-03-04T12:35:58.880 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.880 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.880 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.880 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.880 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_threshold": { 2024-03-04T12:35:58.880 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_threshold", 2024-03-04T12:35:58.880 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.880 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.881 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.881 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "7257600", 2024-03-04T12:35:58.881 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.881 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.881 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.881 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-04T12:35:58.881 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.881 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.882 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.882 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.882 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.882 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.882 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.882 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "diskprediction_local", 2024-03-04T12:35:58.882 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.882 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.882 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.883 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.883 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.883 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.883 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.883 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.883 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.883 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.883 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.883 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.883 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.884 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.884 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.884 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.884 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.884 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.884 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.884 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.884 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.886 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.887 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.888 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.889 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.890 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.891 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.891 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.891 INFO:teuthology.orchestra.run.smithi060.stdout: "predict_interval": { 2024-03-04T12:35:58.891 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "predict_interval", 2024-03-04T12:35:58.891 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.891 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.891 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.891 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "86400", 2024-03-04T12:35:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.892 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "predictor_model": { 2024-03-04T12:35:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "predictor_model", 2024-03-04T12:35:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "prophetstor", 2024-03-04T12:35:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.893 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:58.894 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.895 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "influx", 2024-03-04T12:35:58.896 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": false, 2024-03-04T12:35:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "influxdb python module not found", 2024-03-04T12:35:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "batch_size": { 2024-03-04T12:35:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "batch_size", 2024-03-04T12:35:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.897 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5000", 2024-03-04T12:35:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-04T12:35:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.898 INFO:teuthology.orchestra.run.smithi060.stdout: "database": { 2024-03-04T12:35:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "database", 2024-03-04T12:35:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "ceph", 2024-03-04T12:35:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.899 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-04T12:35:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "hostname": { 2024-03-04T12:35:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "hostname", 2024-03-04T12:35:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.900 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "InfluxDB server hostname", 2024-03-04T12:35:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.901 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "30", 2024-03-04T12:35:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "5", 2024-03-04T12:35:58.902 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-04T12:35:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.903 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.904 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.905 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.906 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.907 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.908 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.909 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.910 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.911 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "password": { 2024-03-04T12:35:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "password", 2024-03-04T12:35:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.912 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "password of InfluxDB server user", 2024-03-04T12:35:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.913 INFO:teuthology.orchestra.run.smithi060.stdout: "port": { 2024-03-04T12:35:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "port", 2024-03-04T12:35:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8086", 2024-03-04T12:35:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.914 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "InfluxDB server port", 2024-03-04T12:35:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "ssl": { 2024-03-04T12:35:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ssl", 2024-03-04T12:35:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.915 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "false", 2024-03-04T12:35:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-04T12:35:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.916 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "threads": { 2024-03-04T12:35:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "threads", 2024-03-04T12:35:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5", 2024-03-04T12:35:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1", 2024-03-04T12:35:58.917 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "32", 2024-03-04T12:35:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-04T12:35:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.918 INFO:teuthology.orchestra.run.smithi060.stdout: "username": { 2024-03-04T12:35:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "username", 2024-03-04T12:35:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.919 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "username of InfluxDB server user", 2024-03-04T12:35:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "verify_ssl": { 2024-03-04T12:35:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "verify_ssl", 2024-03-04T12:35:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.920 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "true", 2024-03-04T12:35:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-04T12:35:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.921 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "insights", 2024-03-04T12:35:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.922 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.923 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.924 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.925 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.926 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.927 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.928 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.929 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.930 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "iostat", 2024-03-04T12:35:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.931 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.932 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.933 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.934 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.935 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.936 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.937 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.938 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.939 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.940 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "k8sevents", 2024-03-04T12:35:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_event_retention_days": { 2024-03-04T12:35:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ceph_event_retention_days", 2024-03-04T12:35:58.941 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "7", 2024-03-04T12:35:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-04T12:35:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.942 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "config_check_secs": { 2024-03-04T12:35:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "config_check_secs", 2024-03-04T12:35:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:58.943 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "10", 2024-03-04T12:35:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-04T12:35:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.944 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.945 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.946 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.947 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.948 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.949 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.950 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.951 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.953 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.954 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "localpool", 2024-03-04T12:35:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "failure_domain": { 2024-03-04T12:35:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "failure_domain", 2024-03-04T12:35:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.955 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "host", 2024-03-04T12:35:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "failure domain for any created local pool", 2024-03-04T12:35:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-04T12:35:58.956 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.957 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.958 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.959 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.960 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.961 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.963 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.964 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": { 2024-03-04T12:35:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "min_size", 2024-03-04T12:35:58.965 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.966 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "default min_size for any created local pool", 2024-03-04T12:35:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-04T12:35:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "num_rep": { 2024-03-04T12:35:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "num_rep", 2024-03-04T12:35:58.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "3", 2024-03-04T12:35:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "default replica count for any created local pool", 2024-03-04T12:35:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.968 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": { 2024-03-04T12:35:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "pg_num", 2024-03-04T12:35:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.969 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "128", 2024-03-04T12:35:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "default pg_num for any created local pool", 2024-03-04T12:35:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.970 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: "prefix": { 2024-03-04T12:35:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "prefix", 2024-03-04T12:35:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.971 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.972 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.972 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.972 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "name prefix for any created local pool", 2024-03-04T12:35:58.972 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.972 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.972 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.972 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.972 INFO:teuthology.orchestra.run.smithi060.stdout: "subtree": { 2024-03-04T12:35:58.972 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "subtree", 2024-03-04T12:35:58.973 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.973 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.973 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.973 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "rack", 2024-03-04T12:35:58.973 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.973 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.973 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.973 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-04T12:35:58.974 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-04T12:35:58.974 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.974 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.974 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.974 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.974 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mds_autoscaler", 2024-03-04T12:35:58.974 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.975 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.975 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.975 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.975 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.975 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.975 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.975 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.975 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.976 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.976 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.976 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.976 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.976 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.976 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.976 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.976 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.976 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.977 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.977 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.977 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.977 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.977 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.977 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.978 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.978 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.978 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.978 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.978 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.978 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.978 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.978 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.979 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.979 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.979 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.979 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.979 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.979 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.980 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.980 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.980 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.980 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.980 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.983 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.983 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.984 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.984 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.984 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.984 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.984 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.984 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.984 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.984 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.984 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.985 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.985 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.985 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.985 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.985 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.985 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.985 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.985 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.986 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.986 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.986 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.986 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.986 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.986 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.986 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.986 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.987 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.987 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.987 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.987 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.987 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.987 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mirroring", 2024-03-04T12:35:58.987 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.987 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.987 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.988 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.988 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.988 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.988 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.988 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.988 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.988 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.988 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.989 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.989 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.989 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.989 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.989 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.989 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.989 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.989 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.990 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.990 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.990 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.990 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.990 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:58.990 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:58.990 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.991 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.991 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.991 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.991 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.991 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.991 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.991 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.991 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.992 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.992 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.992 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:58.992 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:58.992 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.992 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.992 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.993 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:58.993 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.993 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.993 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.993 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.993 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.993 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.993 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.993 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.994 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.994 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.994 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.994 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.994 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.994 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.994 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:58.994 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:58.995 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:58.995 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.995 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.995 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:58.995 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.995 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.995 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:58.995 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.996 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.996 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.996 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.996 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:58.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.996 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:58.996 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "nfs", 2024-03-04T12:35:58.997 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:58.997 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:58.997 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:58.997 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:58.997 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:58.997 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:58.997 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:58.997 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:58.997 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:58.998 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:58.998 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:58.998 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:58.998 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:58.998 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:58.998 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:58.998 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:58.998 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:58.999 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:58.999 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:58.999 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:58.999 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:58.999 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:58.999 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:58.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:58.999 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.000 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.000 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.000 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.000 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.000 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.000 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.000 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.000 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.000 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.001 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.001 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.001 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.001 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.001 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.001 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.002 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.002 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.002 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.002 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.002 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.002 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.002 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.002 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.003 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.003 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.003 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.003 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.003 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.003 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.003 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.003 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.004 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.004 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.004 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.004 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.004 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.004 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.004 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.005 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.005 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.005 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.005 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.005 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.005 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.005 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.005 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.006 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.006 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.006 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "orchestrator", 2024-03-04T12:35:59.006 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.006 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.006 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.006 INFO:teuthology.orchestra.run.smithi060.stdout: "fail_fs": { 2024-03-04T12:35:59.006 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "fail_fs", 2024-03-04T12:35:59.007 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.007 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.007 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.007 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.007 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.007 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.007 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.008 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-04T12:35:59.008 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.008 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.008 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.008 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.008 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.008 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.009 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.009 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.009 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.009 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.009 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.009 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.009 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.009 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.009 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.010 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.010 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.010 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.010 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.010 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.010 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.010 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.011 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.011 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.011 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.011 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.011 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.011 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.011 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.011 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.011 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.012 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.012 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.012 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.012 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.012 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.012 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.012 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.013 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.013 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.013 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.013 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.013 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.013 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.013 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.013 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.013 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.014 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.014 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.014 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.014 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.014 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.014 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.014 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.014 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.015 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.015 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.015 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.015 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.015 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.015 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.015 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.015 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.016 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.016 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.016 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.016 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.016 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.016 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.016 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.016 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.016 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.017 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.017 INFO:teuthology.orchestra.run.smithi060.stdout: "orchestrator": { 2024-03-04T12:35:59.017 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "orchestrator", 2024-03-04T12:35:59.017 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.017 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.017 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.017 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.017 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.018 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.018 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.018 INFO:teuthology.orchestra.run.smithi060.stdout: "cephadm", 2024-03-04T12:35:59.018 INFO:teuthology.orchestra.run.smithi060.stdout: "rook", 2024-03-04T12:35:59.018 INFO:teuthology.orchestra.run.smithi060.stdout: "test_orchestrator" 2024-03-04T12:35:59.018 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.018 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Orchestrator backend", 2024-03-04T12:35:59.018 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.018 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.019 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.019 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.019 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.019 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.019 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "osd_perf_query", 2024-03-04T12:35:59.019 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.019 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.019 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.020 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.020 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.020 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.020 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.020 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.020 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.020 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.020 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.021 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.021 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.021 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.021 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.021 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.021 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.021 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.021 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.021 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.022 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.022 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.022 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.022 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.022 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.022 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.022 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.022 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.023 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.023 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.023 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.023 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.023 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.023 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.023 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.023 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.023 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.024 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.024 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.024 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.024 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.024 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.024 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.024 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.024 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.024 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.025 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.025 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.025 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.025 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.025 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.025 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.025 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.025 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.025 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.026 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.026 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.026 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.026 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.026 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.026 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.026 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.026 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.027 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.027 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.027 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.027 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.027 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.027 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.027 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.027 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.028 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.028 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "osd_support", 2024-03-04T12:35:59.028 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.028 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.028 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.028 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.029 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.029 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.029 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.029 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.029 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.029 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.029 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.029 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.029 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.030 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.030 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.030 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.030 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.030 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.030 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.030 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.030 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.031 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.031 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.031 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.031 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.031 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.031 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.032 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.032 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.032 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.032 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.032 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.032 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.032 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.032 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.033 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.033 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.033 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.033 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.033 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.033 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.034 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.034 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.034 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.034 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.034 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.034 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.034 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.034 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.035 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.035 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.035 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.035 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.035 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.035 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.035 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.036 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.036 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.036 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.036 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.036 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.036 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.036 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.036 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.037 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.037 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.037 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.037 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.037 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.038 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.038 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "pg_autoscaler", 2024-03-04T12:35:59.038 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.038 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.038 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.038 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.039 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.039 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.039 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.039 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.039 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.039 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.039 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.040 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.040 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.040 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.040 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.040 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.040 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.040 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.041 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.041 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.041 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.041 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.041 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.041 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.041 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.042 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.042 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.042 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.042 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.042 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.042 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.042 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.043 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.043 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.043 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.043 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.043 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.043 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.043 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.043 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.043 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.044 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.044 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.044 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.044 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.044 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.044 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.044 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.044 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.045 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.045 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.045 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.045 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.045 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.045 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.045 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.045 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.045 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.046 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.046 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.046 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.046 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.046 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.046 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.046 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.046 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.046 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.047 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.047 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.047 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.047 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.047 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.047 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:59.048 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:59.048 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.048 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.048 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.048 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:59.048 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.048 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.048 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.048 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.049 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.049 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.049 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.049 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.049 INFO:teuthology.orchestra.run.smithi060.stdout: "threshold": { 2024-03-04T12:35:59.049 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "threshold", 2024-03-04T12:35:59.049 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:59.050 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.050 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.050 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "3.0", 2024-03-04T12:35:59.050 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1.0", 2024-03-04T12:35:59.050 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.050 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.050 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "scaling threshold", 2024-03-04T12:35:59.050 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-04T12:35:59.051 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.051 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.051 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.051 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.051 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.051 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "progress", 2024-03-04T12:35:59.051 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.052 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.052 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.052 INFO:teuthology.orchestra.run.smithi060.stdout: "allow_pg_recovery_event": { 2024-03-04T12:35:59.052 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "allow_pg_recovery_event", 2024-03-04T12:35:59.052 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.052 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.052 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.053 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.053 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.053 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.053 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.053 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-04T12:35:59.053 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.053 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.053 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.054 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.054 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": { 2024-03-04T12:35:59.054 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "enabled", 2024-03-04T12:35:59.054 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.054 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.054 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.054 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.055 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.055 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.055 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.055 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.055 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.055 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.055 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.056 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.056 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.056 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.056 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.056 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.056 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.056 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.056 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.056 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.057 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.057 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.057 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.057 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.057 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.057 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.057 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.057 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.058 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.058 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.058 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.058 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.058 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.058 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.058 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.058 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.058 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.059 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.059 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.059 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.059 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.059 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.059 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.059 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.059 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.060 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.060 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.060 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.060 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.060 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.060 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.060 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.060 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.061 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.061 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.061 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.061 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.061 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.061 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.061 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.061 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.061 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.062 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.062 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.062 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.062 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.062 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.062 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.062 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.062 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.063 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.063 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.063 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.063 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.063 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.063 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.063 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.063 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.063 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.064 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.064 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.064 INFO:teuthology.orchestra.run.smithi060.stdout: "max_completed_events": { 2024-03-04T12:35:59.064 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_completed_events", 2024-03-04T12:35:59.064 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.064 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.064 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.064 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "50", 2024-03-04T12:35:59.064 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.065 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.065 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.065 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "number of past completed events to remember", 2024-03-04T12:35:59.065 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.065 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.065 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.065 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:59.065 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:59.066 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.066 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.066 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.066 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5", 2024-03-04T12:35:59.066 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.066 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.066 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.066 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how long the module is going to sleep", 2024-03-04T12:35:59.067 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.067 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.067 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.067 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.067 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.067 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.067 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "prometheus", 2024-03-04T12:35:59.068 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.068 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.068 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.068 INFO:teuthology.orchestra.run.smithi060.stdout: "cache": { 2024-03-04T12:35:59.068 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cache", 2024-03-04T12:35:59.068 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.069 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.069 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.069 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.069 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.069 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.069 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.069 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.069 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.070 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.070 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.070 INFO:teuthology.orchestra.run.smithi060.stdout: "exclude_perf_counters": { 2024-03-04T12:35:59.070 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "exclude_perf_counters", 2024-03-04T12:35:59.070 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.070 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.071 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.071 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.071 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.071 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.071 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.071 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-04T12:35:59.071 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-04T12:35:59.072 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.072 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.072 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.072 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.072 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.072 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.072 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.072 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.073 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.073 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.073 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.073 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.073 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.073 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.073 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.074 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.074 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.074 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.074 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.074 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.074 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.074 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.074 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.075 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.075 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.075 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.075 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.075 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.075 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.075 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.075 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.076 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.076 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.076 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.076 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.076 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.076 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.077 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.077 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.077 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.077 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.077 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.077 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.077 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.078 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.078 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.078 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.078 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.078 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.078 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.078 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.078 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.078 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.079 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.079 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.079 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.079 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.079 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.079 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.079 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.079 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.079 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.080 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.080 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.080 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.080 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.080 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.080 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.080 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.080 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.081 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.081 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.081 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_stats_pools": { 2024-03-04T12:35:59.081 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd_stats_pools", 2024-03-04T12:35:59.081 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.081 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.081 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.081 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.082 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.082 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.082 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.082 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.082 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.082 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.082 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.083 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-04T12:35:59.083 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-04T12:35:59.083 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.083 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.083 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.083 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "300", 2024-03-04T12:35:59.084 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.084 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.084 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.084 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.084 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.084 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.084 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.085 INFO:teuthology.orchestra.run.smithi060.stdout: "scrape_interval": { 2024-03-04T12:35:59.085 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "scrape_interval", 2024-03-04T12:35:59.085 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:59.085 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.085 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.085 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "15.0", 2024-03-04T12:35:59.085 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.086 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.086 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.086 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.086 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.086 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.086 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.086 INFO:teuthology.orchestra.run.smithi060.stdout: "server_addr": { 2024-03-04T12:35:59.086 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_addr", 2024-03-04T12:35:59.087 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.087 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.087 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.087 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "::", 2024-03-04T12:35:59.087 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.087 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.088 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.088 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-04T12:35:59.088 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.088 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.088 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.088 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.088 INFO:teuthology.orchestra.run.smithi060.stdout: "server_port": { 2024-03-04T12:35:59.088 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_port", 2024-03-04T12:35:59.088 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.089 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.089 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.089 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "9283", 2024-03-04T12:35:59.089 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.089 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.089 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.089 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-04T12:35:59.090 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.090 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.090 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.090 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.090 INFO:teuthology.orchestra.run.smithi060.stdout: "stale_cache_strategy": { 2024-03-04T12:35:59.090 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "stale_cache_strategy", 2024-03-04T12:35:59.090 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.090 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.091 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.091 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "log", 2024-03-04T12:35:59.091 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.091 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.091 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.091 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.091 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.091 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.092 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.092 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.092 INFO:teuthology.orchestra.run.smithi060.stdout: "standby_behaviour": { 2024-03-04T12:35:59.092 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "standby_behaviour", 2024-03-04T12:35:59.092 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.092 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.092 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.092 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "default", 2024-03-04T12:35:59.093 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.093 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.093 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.093 INFO:teuthology.orchestra.run.smithi060.stdout: "default", 2024-03-04T12:35:59.093 INFO:teuthology.orchestra.run.smithi060.stdout: "error" 2024-03-04T12:35:59.093 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.093 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.093 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.094 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.094 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.094 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.094 INFO:teuthology.orchestra.run.smithi060.stdout: "standby_error_status_code": { 2024-03-04T12:35:59.094 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "standby_error_status_code", 2024-03-04T12:35:59.094 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.094 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.094 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.094 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "500", 2024-03-04T12:35:59.095 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "400", 2024-03-04T12:35:59.095 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "599", 2024-03-04T12:35:59.095 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.095 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.095 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.095 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.095 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.095 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.096 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.096 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.096 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.096 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd_support", 2024-03-04T12:35:59.096 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.096 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.096 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.096 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.096 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.097 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.097 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.097 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.097 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.097 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.097 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.097 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.097 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.098 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.098 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.098 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.098 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.098 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.098 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.098 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.098 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.099 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.099 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.099 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.099 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.099 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.099 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.099 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.099 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.099 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.100 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.100 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.100 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.100 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.100 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.100 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.100 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.100 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.100 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.101 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.101 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.101 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.101 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.101 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.101 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.101 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.101 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.102 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.102 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.102 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.102 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.102 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.102 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.102 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.102 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.103 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.103 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.103 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.103 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.103 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.103 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.103 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.104 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.104 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.104 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.104 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.104 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.104 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.104 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.105 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.105 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.105 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.105 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.105 INFO:teuthology.orchestra.run.smithi060.stdout: "max_concurrent_snap_create": { 2024-03-04T12:35:59.105 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_concurrent_snap_create", 2024-03-04T12:35:59.106 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.106 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.106 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.106 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:59.106 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.106 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.106 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.106 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.107 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.107 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.107 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.107 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.107 INFO:teuthology.orchestra.run.smithi060.stdout: "mirror_snapshot_schedule": { 2024-03-04T12:35:59.107 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mirror_snapshot_schedule", 2024-03-04T12:35:59.108 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.108 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.108 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.108 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.108 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.108 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.108 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.108 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.109 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.109 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.109 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.109 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.109 INFO:teuthology.orchestra.run.smithi060.stdout: "trash_purge_schedule": { 2024-03-04T12:35:59.109 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "trash_purge_schedule", 2024-03-04T12:35:59.109 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.109 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.109 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.110 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.110 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.110 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.110 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.110 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.110 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.110 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.110 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.111 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.111 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.111 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.111 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.111 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "restful", 2024-03-04T12:35:59.111 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.111 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.112 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.112 INFO:teuthology.orchestra.run.smithi060.stdout: "enable_auth": { 2024-03-04T12:35:59.112 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "enable_auth", 2024-03-04T12:35:59.112 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.112 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.112 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.112 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.113 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.113 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.113 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.113 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.113 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.113 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.113 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: "key_file": { 2024-03-04T12:35:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "key_file", 2024-03-04T12:35:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.114 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.115 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.115 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.115 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.115 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.115 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.115 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.115 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.115 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.116 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.116 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.116 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.116 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.116 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.116 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.117 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.117 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.117 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.117 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.117 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.118 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.118 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.118 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.118 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.118 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.119 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.119 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.119 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.119 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.119 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.119 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.119 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.120 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.120 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.120 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.120 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.120 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.120 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.120 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.121 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.121 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.121 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.121 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.121 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.121 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.121 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.122 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.122 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.122 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.122 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.122 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.122 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.122 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.122 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.122 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.123 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.123 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.123 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.123 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.123 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.123 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.123 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.124 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.124 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.124 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.124 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.124 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.124 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.124 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.125 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.125 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.125 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.125 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.125 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.125 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.125 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.126 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.126 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.126 INFO:teuthology.orchestra.run.smithi060.stdout: "server_addr": { 2024-03-04T12:35:59.126 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_addr", 2024-03-04T12:35:59.126 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.126 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.126 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.127 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.127 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.127 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.127 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.127 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.127 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.127 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.128 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.128 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.128 INFO:teuthology.orchestra.run.smithi060.stdout: "server_port": { 2024-03-04T12:35:59.128 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_port", 2024-03-04T12:35:59.128 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.128 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.128 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.128 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.129 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.129 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.129 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.129 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.129 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.129 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.129 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.129 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.130 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.130 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.130 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.130 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rgw", 2024-03-04T12:35:59.130 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.130 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.130 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.130 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.131 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.131 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.131 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.131 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.131 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.131 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.131 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.131 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.132 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.132 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.132 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.132 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.132 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.132 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.132 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.132 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.133 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.133 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.133 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.133 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.133 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.133 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.133 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.133 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.133 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.134 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.134 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.134 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.134 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.134 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.134 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.134 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.134 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.135 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.135 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.135 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.135 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.135 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.135 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.135 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.135 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.135 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.136 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.136 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.136 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.136 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.136 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.136 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.136 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.136 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.137 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.137 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.137 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.137 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.137 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.137 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.137 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.137 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.137 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.138 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.138 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.138 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.138 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.138 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.138 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.138 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.138 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.139 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.139 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.139 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.139 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.139 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.139 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rook", 2024-03-04T12:35:59.139 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.140 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.140 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.140 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.140 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.140 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.140 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.141 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.141 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.141 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.141 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.141 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.141 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.141 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.141 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.142 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.142 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.142 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.142 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.142 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.142 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.143 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.143 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.143 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.143 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.143 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.143 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.143 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.143 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.144 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.144 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.144 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.144 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.144 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.144 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.145 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.145 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.145 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.145 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.145 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.145 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.145 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.146 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.146 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.146 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.146 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.146 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.146 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.147 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.147 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.147 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.147 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.147 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.147 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.147 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.148 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.148 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.148 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.148 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.148 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.148 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.148 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.149 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.149 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.149 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.149 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.149 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.149 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.149 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.149 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.150 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.150 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.150 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.150 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_class": { 2024-03-04T12:35:59.150 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "storage_class", 2024-03-04T12:35:59.150 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.150 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.150 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.150 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "local", 2024-03-04T12:35:59.151 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.151 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.151 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.151 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-04T12:35:59.151 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.151 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.151 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.151 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.151 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.152 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.152 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.152 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "selftest", 2024-03-04T12:35:59.152 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.152 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.152 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.152 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.152 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.153 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.154 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.155 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.155 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.155 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.155 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.155 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.155 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.155 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.155 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.156 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.157 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.157 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.157 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.157 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.157 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.157 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.157 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.157 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.157 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.158 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.158 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.158 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.158 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.158 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.158 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.158 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.158 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.159 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.159 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.159 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.159 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.159 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.159 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.159 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.159 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.159 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.160 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.160 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.160 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.160 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.160 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.160 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.160 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.160 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.160 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.161 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.161 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.161 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.161 INFO:teuthology.orchestra.run.smithi060.stdout: "roption1": { 2024-03-04T12:35:59.161 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "roption1", 2024-03-04T12:35:59.161 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.161 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.161 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.162 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.162 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.162 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.162 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.162 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.162 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.162 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.162 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.162 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.163 INFO:teuthology.orchestra.run.smithi060.stdout: "roption2": { 2024-03-04T12:35:59.163 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "roption2", 2024-03-04T12:35:59.163 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.163 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.163 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.163 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "xyz", 2024-03-04T12:35:59.163 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.163 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.163 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.164 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.164 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.164 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.164 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.164 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.164 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption1": { 2024-03-04T12:35:59.164 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption1", 2024-03-04T12:35:59.165 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.165 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.165 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.165 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.165 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.165 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.165 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.165 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.166 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.166 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.166 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.166 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.166 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption2": { 2024-03-04T12:35:59.166 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption2", 2024-03-04T12:35:59.166 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.166 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.166 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.167 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.167 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.167 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.167 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.167 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.167 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.167 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.167 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.168 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.168 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption3": { 2024-03-04T12:35:59.168 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption3", 2024-03-04T12:35:59.168 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:59.168 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.168 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.168 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.168 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.169 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.169 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.169 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.169 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.169 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.169 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.169 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption4": { 2024-03-04T12:35:59.170 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption4", 2024-03-04T12:35:59.170 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.170 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.170 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.170 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.170 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.170 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.170 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.171 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.171 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.171 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.171 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.171 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.171 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption5": { 2024-03-04T12:35:59.171 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption5", 2024-03-04T12:35:59.171 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.171 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.172 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.172 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.172 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.172 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.172 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.172 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.172 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.172 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.173 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.173 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption6": { 2024-03-04T12:35:59.173 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption6", 2024-03-04T12:35:59.173 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.173 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.173 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.173 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.173 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.174 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.174 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.174 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.174 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.174 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.174 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.175 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption7": { 2024-03-04T12:35:59.175 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption7", 2024-03-04T12:35:59.175 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.175 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.175 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.175 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.175 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1", 2024-03-04T12:35:59.176 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "42", 2024-03-04T12:35:59.176 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.176 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.176 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.176 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.176 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.177 INFO:teuthology.orchestra.run.smithi060.stdout: "testkey": { 2024-03-04T12:35:59.177 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "testkey", 2024-03-04T12:35:59.177 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.177 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.177 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.177 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.177 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.177 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.177 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.178 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.178 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.178 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.178 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.178 INFO:teuthology.orchestra.run.smithi060.stdout: "testlkey": { 2024-03-04T12:35:59.178 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "testlkey", 2024-03-04T12:35:59.178 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.179 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.179 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.179 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.179 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.179 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.179 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.179 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.179 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.179 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.180 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.180 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.180 INFO:teuthology.orchestra.run.smithi060.stdout: "testnewline": { 2024-03-04T12:35:59.180 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "testnewline", 2024-03-04T12:35:59.180 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.180 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.180 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.180 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.181 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.181 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.181 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.181 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.181 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.181 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.181 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.181 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.182 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.182 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.182 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.182 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "snap_schedule", 2024-03-04T12:35:59.182 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.182 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.182 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.182 INFO:teuthology.orchestra.run.smithi060.stdout: "allow_m_granularity": { 2024-03-04T12:35:59.182 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "allow_m_granularity", 2024-03-04T12:35:59.183 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.183 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.183 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.183 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.183 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.183 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.183 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.183 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "allow minute scheduled snapshots", 2024-03-04T12:35:59.184 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.184 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.184 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.184 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.184 INFO:teuthology.orchestra.run.smithi060.stdout: "dump_on_update": { 2024-03-04T12:35:59.184 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "dump_on_update", 2024-03-04T12:35:59.184 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.184 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.185 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.185 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.185 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.185 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.185 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.185 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "dump database to debug log on update", 2024-03-04T12:35:59.185 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.185 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.185 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.186 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.186 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.186 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.186 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.186 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.186 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.186 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.187 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.187 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.187 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.187 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.187 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.187 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.187 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.187 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.188 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.188 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.188 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.188 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.188 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.188 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.188 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.188 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.189 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.189 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.189 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.189 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.189 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.189 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.189 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.189 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.190 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.190 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.190 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.190 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.190 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.190 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.190 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.190 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.191 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.191 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.191 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.191 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.191 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.191 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.191 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.191 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.192 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.192 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.192 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.192 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.192 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.192 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.192 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.192 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.193 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.193 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.193 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.193 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.193 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.193 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.193 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.193 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.194 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.194 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.194 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.194 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.194 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.194 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.194 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.194 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.195 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.195 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "stats", 2024-03-04T12:35:59.195 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.195 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.195 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.195 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.195 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.195 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.196 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.196 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.196 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.196 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.196 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.196 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.196 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.196 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.196 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.197 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.197 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.197 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.197 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.197 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.197 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.197 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.197 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.198 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.198 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.198 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.198 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.198 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.198 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.198 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.199 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.199 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.199 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.199 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.199 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.199 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.199 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.199 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.200 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.200 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.200 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.200 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.200 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.200 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.200 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.201 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.201 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.201 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.201 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.201 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.201 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.201 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.201 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.202 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.202 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.202 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.202 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.202 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.202 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.202 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.203 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.203 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.203 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.203 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.203 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.203 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.203 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.203 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.204 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.204 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.204 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.204 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.204 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "status", 2024-03-04T12:35:59.204 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.205 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.205 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.205 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.205 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.205 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.206 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.206 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.206 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.206 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.206 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.206 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.206 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.207 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.207 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.207 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.207 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.207 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.207 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.208 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.208 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.208 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.208 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.208 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.209 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.209 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.209 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.209 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.209 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.209 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.210 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.210 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.210 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.210 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.210 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.210 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.211 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.211 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.211 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.211 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.211 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.211 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.211 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.212 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.212 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.212 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.212 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.212 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.212 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.213 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.213 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.213 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.213 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.213 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.214 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.214 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.214 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.214 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.214 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.214 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.215 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.215 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.215 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.215 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.215 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.216 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.216 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.216 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.216 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.216 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.216 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.216 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.217 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.217 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "telegraf", 2024-03-04T12:35:59.217 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.217 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.217 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.217 INFO:teuthology.orchestra.run.smithi060.stdout: "address": { 2024-03-04T12:35:59.217 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "address", 2024-03-04T12:35:59.218 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.218 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.218 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.218 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-04T12:35:59.218 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.218 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.218 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.218 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.219 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.219 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.219 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.219 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.219 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:59.219 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:59.219 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.220 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.220 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.220 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "15", 2024-03-04T12:35:59.220 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.220 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.220 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.220 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.221 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.221 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.221 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.221 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.221 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.221 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.222 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.222 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.222 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.222 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.222 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.222 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.223 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.223 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.223 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.223 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.223 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.223 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.223 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.224 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.224 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.224 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.224 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.224 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.224 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.225 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.225 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.225 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.225 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.225 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.225 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.225 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.226 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.226 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.226 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.226 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.226 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.226 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.226 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.226 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.227 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.227 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.227 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.227 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.227 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.227 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.227 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.227 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.228 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.228 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.228 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.228 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.228 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.228 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.228 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.228 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.229 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.229 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.229 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.229 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.229 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.229 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.229 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.230 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.230 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.230 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.230 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.230 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.230 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.230 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.230 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.231 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.231 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.231 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.231 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "telemetry", 2024-03-04T12:35:59.231 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.231 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.231 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.232 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_basic": { 2024-03-04T12:35:59.232 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_basic", 2024-03-04T12:35:59.232 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.232 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.232 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.232 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.232 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.232 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.233 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.233 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-04T12:35:59.233 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.233 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.233 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.233 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.233 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_crash": { 2024-03-04T12:35:59.233 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_crash", 2024-03-04T12:35:59.234 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.234 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.234 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.234 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.234 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.234 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.234 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.234 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-04T12:35:59.234 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.235 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.235 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.235 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.235 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_device": { 2024-03-04T12:35:59.235 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_device", 2024-03-04T12:35:59.235 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.235 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.235 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.235 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.236 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.236 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.236 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.236 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-04T12:35:59.236 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.236 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.236 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.237 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_ident": { 2024-03-04T12:35:59.237 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_ident", 2024-03-04T12:35:59.237 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.237 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.237 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.237 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.237 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.237 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.238 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.238 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-04T12:35:59.238 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.238 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.238 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.238 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.238 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_perf": { 2024-03-04T12:35:59.238 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_perf", 2024-03-04T12:35:59.238 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.239 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.239 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.239 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.239 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.239 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.239 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.239 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-04T12:35:59.239 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.239 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.240 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.240 INFO:teuthology.orchestra.run.smithi060.stdout: "contact": { 2024-03-04T12:35:59.240 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "contact", 2024-03-04T12:35:59.240 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.240 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.240 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.240 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.241 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.241 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.241 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.241 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.241 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.241 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.241 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.241 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.242 INFO:teuthology.orchestra.run.smithi060.stdout: "description": { 2024-03-04T12:35:59.242 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "description", 2024-03-04T12:35:59.242 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.242 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.242 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.242 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.242 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.242 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.243 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.243 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.243 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.243 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.243 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.243 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.243 INFO:teuthology.orchestra.run.smithi060.stdout: "device_url": { 2024-03-04T12:35:59.243 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "device_url", 2024-03-04T12:35:59.243 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.244 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.244 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.244 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-04T12:35:59.244 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.244 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.244 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.244 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.244 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.245 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.245 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.245 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.245 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": { 2024-03-04T12:35:59.245 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "enabled", 2024-03-04T12:35:59.245 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.245 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.245 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.246 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.246 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.246 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.246 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.246 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.246 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.246 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.246 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.247 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.247 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:59.247 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:59.247 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.247 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.247 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.247 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "24", 2024-03-04T12:35:59.247 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "8", 2024-03-04T12:35:59.247 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.248 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.248 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.248 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.248 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.248 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.248 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.248 INFO:teuthology.orchestra.run.smithi060.stdout: "last_opt_revision": { 2024-03-04T12:35:59.248 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "last_opt_revision", 2024-03-04T12:35:59.249 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.249 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.249 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.249 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "1", 2024-03-04T12:35:59.249 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.249 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.249 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.249 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.250 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.250 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.250 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.250 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.250 INFO:teuthology.orchestra.run.smithi060.stdout: "leaderboard": { 2024-03-04T12:35:59.250 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "leaderboard", 2024-03-04T12:35:59.250 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.250 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.250 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.251 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.251 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.251 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.251 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.251 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.251 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.251 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.251 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.252 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.252 INFO:teuthology.orchestra.run.smithi060.stdout: "leaderboard_description": { 2024-03-04T12:35:59.252 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "leaderboard_description", 2024-03-04T12:35:59.252 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.252 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.252 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.252 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.252 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.253 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.253 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.253 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.253 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.253 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.253 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.253 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.253 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.253 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.254 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.254 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.254 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.254 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.254 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.254 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.254 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.254 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.254 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.255 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.255 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.255 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.255 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.255 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.255 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.255 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.255 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.255 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.256 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.256 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.256 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.256 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.256 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.256 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.256 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.256 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.256 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.257 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.257 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.257 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.257 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.257 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.257 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.257 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.257 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.258 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.258 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.258 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.258 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.258 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.258 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.258 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.258 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.258 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.259 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.259 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.259 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.259 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.259 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.259 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.259 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.259 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.259 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.260 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.260 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.260 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.260 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.260 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.260 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.260 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.260 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.261 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.261 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.261 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.261 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.261 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.261 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.261 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.261 INFO:teuthology.orchestra.run.smithi060.stdout: "organization": { 2024-03-04T12:35:59.262 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "organization", 2024-03-04T12:35:59.262 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.262 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.262 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.262 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.262 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.262 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.262 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.263 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.263 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.263 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.263 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.263 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.263 INFO:teuthology.orchestra.run.smithi060.stdout: "proxy": { 2024-03-04T12:35:59.263 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "proxy", 2024-03-04T12:35:59.263 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.264 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.264 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.264 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.264 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.264 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.264 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.264 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.265 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.265 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.265 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.265 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.265 INFO:teuthology.orchestra.run.smithi060.stdout: "url": { 2024-03-04T12:35:59.265 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "url", 2024-03-04T12:35:59.265 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.265 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.266 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.266 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-04T12:35:59.266 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.266 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.266 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.266 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.266 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.267 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.267 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.267 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.267 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.267 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.267 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.267 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test_orchestrator", 2024-03-04T12:35:59.267 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.268 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.268 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.268 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.268 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.268 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.268 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.268 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.268 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.269 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.269 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.269 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.269 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.269 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.269 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.269 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.270 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.270 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.270 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.270 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.270 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.270 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.270 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.271 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.271 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.271 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.271 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.271 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.271 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.271 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.271 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.272 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.272 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.272 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.272 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.272 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.272 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.272 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.273 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.273 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.273 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.273 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.273 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.273 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.273 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.274 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.274 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.274 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.274 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.274 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.274 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.274 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.274 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.275 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.275 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.275 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.275 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.275 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.275 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.275 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.276 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.276 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.276 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.276 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.276 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.276 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.276 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.276 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.277 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.277 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.277 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.277 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.277 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.277 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "volumes", 2024-03-04T12:35:59.277 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.277 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.278 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.278 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.278 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.278 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.278 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.278 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.278 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.278 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.278 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.279 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.279 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.279 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.279 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.279 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.279 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.279 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.279 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.280 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.280 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.280 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.280 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.280 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.280 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.280 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.280 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.281 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.281 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.281 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.281 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.281 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.281 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.281 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.281 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.282 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.282 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.282 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.282 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.282 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.282 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.282 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.283 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.283 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.283 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.283 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.283 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.283 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.283 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.283 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.284 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.284 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.284 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.284 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.284 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.284 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.285 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.285 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.285 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.285 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.285 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.285 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.285 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.285 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.285 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.286 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.287 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.287 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.287 INFO:teuthology.orchestra.run.smithi060.stdout: "max_concurrent_clones": { 2024-03-04T12:35:59.287 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_concurrent_clones", 2024-03-04T12:35:59.287 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.287 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.287 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.287 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "4", 2024-03-04T12:35:59.288 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.288 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.288 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.288 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-04T12:35:59.288 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.288 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.289 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.289 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.289 INFO:teuthology.orchestra.run.smithi060.stdout: "periodic_async_work": { 2024-03-04T12:35:59.289 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "periodic_async_work", 2024-03-04T12:35:59.289 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.289 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.289 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.289 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.290 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.290 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.290 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.290 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Periodically check for async work", 2024-03-04T12:35:59.290 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.290 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.290 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.291 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.291 INFO:teuthology.orchestra.run.smithi060.stdout: "snapshot_clone_delay": { 2024-03-04T12:35:59.291 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "snapshot_clone_delay", 2024-03-04T12:35:59.291 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.291 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.291 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.291 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:59.292 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.292 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.292 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.292 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-04T12:35:59.292 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.292 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.293 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.293 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.293 INFO:teuthology.orchestra.run.smithi060.stdout: "snapshot_clone_no_wait": { 2024-03-04T12:35:59.293 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "snapshot_clone_no_wait", 2024-03-04T12:35:59.293 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.293 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.293 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.293 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.294 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.294 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.294 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.294 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-04T12:35:59.294 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.294 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.294 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.294 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.295 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.295 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.295 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.295 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "zabbix", 2024-03-04T12:35:59.295 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.295 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.295 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.295 INFO:teuthology.orchestra.run.smithi060.stdout: "discovery_interval": { 2024-03-04T12:35:59.295 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "discovery_interval", 2024-03-04T12:35:59.296 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:59.296 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.296 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.296 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "100", 2024-03-04T12:35:59.296 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.296 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.296 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.296 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.297 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.297 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.297 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.297 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.297 INFO:teuthology.orchestra.run.smithi060.stdout: "identifier": { 2024-03-04T12:35:59.297 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "identifier", 2024-03-04T12:35:59.297 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.297 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.298 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.298 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.298 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.298 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.298 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.298 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.298 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.298 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.299 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.299 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.299 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:59.299 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:59.299 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.299 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.299 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.300 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:59.300 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.300 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.300 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.300 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.300 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.300 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.301 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.301 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.301 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.301 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.301 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.301 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.301 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.301 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.302 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.302 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.302 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.302 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.302 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.302 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.302 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.303 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.303 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.303 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.303 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.303 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.303 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.303 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.303 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.304 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.304 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.304 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.304 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.304 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.304 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.305 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.306 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.307 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.308 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.309 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "zabbix_host": { 2024-03-04T12:35:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "zabbix_host", 2024-03-04T12:35:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.310 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "zabbix_port": { 2024-03-04T12:35:59.311 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "zabbix_port", 2024-03-04T12:35:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10051", 2024-03-04T12:35:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.312 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "zabbix_sender": { 2024-03-04T12:35:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "zabbix_sender", 2024-03-04T12:35:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.313 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-04T12:35:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.314 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-04T12:35:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.315 INFO:teuthology.orchestra.run.smithi060.stdout: "modules": [ 2024-03-04T12:35:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "cephadm", 2024-03-04T12:35:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "dashboard", 2024-03-04T12:35:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "iostat", 2024-03-04T12:35:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "nfs", 2024-03-04T12:35:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "restful", 2024-03-04T12:35:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "snap_schedule" 2024-03-04T12:35:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: "available_modules": [ 2024-03-04T12:35:59.316 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "alerts", 2024-03-04T12:35:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.317 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "How frequently to reexamine health status", 2024-03-04T12:35:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.318 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.319 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.320 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.321 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.322 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.323 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.323 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.323 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.323 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.323 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.323 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.324 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.324 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.324 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.324 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.324 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.324 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.324 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.325 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.325 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.325 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.325 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.325 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.325 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.326 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.326 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.326 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.326 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.326 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.326 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.326 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.326 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.327 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.327 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.327 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.327 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.327 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.327 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.328 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.328 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.328 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.328 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.328 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.328 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.328 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.328 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.328 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.329 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.329 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.329 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.329 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_destination": { 2024-03-04T12:35:59.329 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_destination", 2024-03-04T12:35:59.329 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.329 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.329 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.330 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.330 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.330 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.330 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.330 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Email address to send alerts to", 2024-03-04T12:35:59.330 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.330 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.330 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.330 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.331 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_from_name": { 2024-03-04T12:35:59.331 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_from_name", 2024-03-04T12:35:59.331 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.331 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.331 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.331 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "Ceph", 2024-03-04T12:35:59.331 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.331 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.332 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.332 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Email From: name", 2024-03-04T12:35:59.332 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.332 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.332 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.332 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.332 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_host": { 2024-03-04T12:35:59.332 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_host", 2024-03-04T12:35:59.332 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.333 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.333 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.333 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.333 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.333 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.333 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.333 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "SMTP server", 2024-03-04T12:35:59.333 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.334 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.334 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.334 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.334 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_password": { 2024-03-04T12:35:59.334 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_password", 2024-03-04T12:35:59.334 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.334 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.334 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.334 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.335 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.335 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.335 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.335 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Password to authenticate with", 2024-03-04T12:35:59.335 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.335 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.335 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.335 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.336 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_port": { 2024-03-04T12:35:59.336 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_port", 2024-03-04T12:35:59.336 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.336 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.336 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.336 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "465", 2024-03-04T12:35:59.336 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.336 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.337 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.337 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "SMTP port", 2024-03-04T12:35:59.337 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.337 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.337 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.337 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.337 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_sender": { 2024-03-04T12:35:59.337 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_sender", 2024-03-04T12:35:59.337 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.338 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.338 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.338 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.338 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.338 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.338 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.338 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "SMTP envelope sender", 2024-03-04T12:35:59.339 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.339 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.339 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.339 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.339 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_ssl": { 2024-03-04T12:35:59.339 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_ssl", 2024-03-04T12:35:59.339 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.339 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.340 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.340 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.340 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.340 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.340 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.340 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-04T12:35:59.340 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.340 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.340 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.341 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.341 INFO:teuthology.orchestra.run.smithi060.stdout: "smtp_user": { 2024-03-04T12:35:59.341 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "smtp_user", 2024-03-04T12:35:59.341 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.341 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.341 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.341 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.342 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.342 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.342 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.342 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "User to authenticate as", 2024-03-04T12:35:59.342 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.342 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.342 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "balancer", 2024-03-04T12:35:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.343 INFO:teuthology.orchestra.run.smithi060.stdout: "active": { 2024-03-04T12:35:59.344 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "active", 2024-03-04T12:35:59.344 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.344 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.344 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.344 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.344 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.344 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.344 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.345 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "automatically balance PGs across cluster", 2024-03-04T12:35:59.345 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.345 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.345 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.345 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.345 INFO:teuthology.orchestra.run.smithi060.stdout: "begin_time": { 2024-03-04T12:35:59.345 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "begin_time", 2024-03-04T12:35:59.345 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0000", 2024-03-04T12:35:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "beginning time of day to automatically balance", 2024-03-04T12:35:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-04T12:35:59.346 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "begin_weekday": { 2024-03-04T12:35:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "begin_weekday", 2024-03-04T12:35:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:59.347 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "0", 2024-03-04T12:35:59.348 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "6", 2024-03-04T12:35:59.348 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.348 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-04T12:35:59.348 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-04T12:35:59.348 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.348 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.348 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.348 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_compat_max_iterations": { 2024-03-04T12:35:59.349 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crush_compat_max_iterations", 2024-03-04T12:35:59.349 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:59.349 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.349 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.349 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "25", 2024-03-04T12:35:59.349 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1", 2024-03-04T12:35:59.349 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "250", 2024-03-04T12:35:59.349 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.349 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-04T12:35:59.350 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.350 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.350 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.350 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.350 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_compat_metrics": { 2024-03-04T12:35:59.350 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crush_compat_metrics", 2024-03-04T12:35:59.350 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.350 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.351 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.351 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "pgs,objects,bytes", 2024-03-04T12:35:59.351 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.351 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.351 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.351 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-04T12:35:59.351 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-04T12:35:59.351 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.351 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.352 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.352 INFO:teuthology.orchestra.run.smithi060.stdout: "crush_compat_step": { 2024-03-04T12:35:59.352 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crush_compat_step", 2024-03-04T12:35:59.352 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:59.352 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.352 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.352 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0.5", 2024-03-04T12:35:59.352 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "0.001", 2024-03-04T12:35:59.353 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "0.999", 2024-03-04T12:35:59.353 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.353 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "aggressiveness of optimization", 2024-03-04T12:35:59.353 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-04T12:35:59.353 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.353 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.353 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.353 INFO:teuthology.orchestra.run.smithi060.stdout: "end_time": { 2024-03-04T12:35:59.354 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "end_time", 2024-03-04T12:35:59.354 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.354 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.354 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.354 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "2359", 2024-03-04T12:35:59.354 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.354 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.354 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.354 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "ending time of day to automatically balance", 2024-03-04T12:35:59.355 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-04T12:35:59.355 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.355 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.355 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.355 INFO:teuthology.orchestra.run.smithi060.stdout: "end_weekday": { 2024-03-04T12:35:59.355 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "end_weekday", 2024-03-04T12:35:59.355 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:59.356 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.356 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.356 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:59.356 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "0", 2024-03-04T12:35:59.356 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "6", 2024-03-04T12:35:59.356 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.357 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-04T12:35:59.357 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-04T12:35:59.357 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.357 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.357 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.357 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.357 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.357 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.358 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.358 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.358 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.358 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.358 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.358 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.358 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.359 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.359 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.359 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.359 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.359 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.359 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.359 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.359 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.360 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.360 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.360 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.360 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.360 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.360 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.360 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.361 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.361 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.361 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.361 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.361 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.361 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.361 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.362 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.362 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.362 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.362 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.362 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.362 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.363 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.363 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.363 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.363 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.363 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.363 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.363 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.364 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.364 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.364 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.364 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.364 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.364 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.364 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.365 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.365 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.365 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.365 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.365 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.365 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.365 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.366 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.366 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.366 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.366 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.366 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.366 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.366 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.366 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.367 INFO:teuthology.orchestra.run.smithi060.stdout: "min_score": { 2024-03-04T12:35:59.367 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "min_score", 2024-03-04T12:35:59.367 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:59.367 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.367 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.367 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:59.367 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.367 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.367 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.368 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-04T12:35:59.368 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.368 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.368 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.368 INFO:teuthology.orchestra.run.smithi060.stdout: "mode": { 2024-03-04T12:35:59.368 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mode", 2024-03-04T12:35:59.368 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.369 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.369 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.369 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "upmap", 2024-03-04T12:35:59.369 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.369 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.369 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.369 INFO:teuthology.orchestra.run.smithi060.stdout: "crush-compat", 2024-03-04T12:35:59.369 INFO:teuthology.orchestra.run.smithi060.stdout: "none", 2024-03-04T12:35:59.370 INFO:teuthology.orchestra.run.smithi060.stdout: "read", 2024-03-04T12:35:59.370 INFO:teuthology.orchestra.run.smithi060.stdout: "upmap", 2024-03-04T12:35:59.370 INFO:teuthology.orchestra.run.smithi060.stdout: "upmap-read" 2024-03-04T12:35:59.370 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.370 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Balancer mode", 2024-03-04T12:35:59.370 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.370 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.370 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.371 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_ids": { 2024-03-04T12:35:59.371 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "pool_ids", 2024-03-04T12:35:59.371 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.371 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.371 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.371 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.371 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.372 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.372 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.372 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-04T12:35:59.372 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.372 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.372 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.372 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:59.373 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:59.373 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.373 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.373 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.373 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:59.373 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.373 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.373 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.373 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-04T12:35:59.374 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.374 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.374 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.374 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.374 INFO:teuthology.orchestra.run.smithi060.stdout: "upmap_max_deviation": { 2024-03-04T12:35:59.374 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "upmap_max_deviation", 2024-03-04T12:35:59.374 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.375 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.375 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.375 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5", 2024-03-04T12:35:59.375 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1", 2024-03-04T12:35:59.375 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.375 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.375 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-04T12:35:59.375 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-04T12:35:59.376 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.376 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.376 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.376 INFO:teuthology.orchestra.run.smithi060.stdout: "upmap_max_optimizations": { 2024-03-04T12:35:59.376 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "upmap_max_optimizations", 2024-03-04T12:35:59.376 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:35:59.376 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.376 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.377 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:59.377 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.377 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.377 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.377 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-04T12:35:59.377 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.377 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.377 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.378 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.378 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.378 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.378 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.378 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephadm", 2024-03-04T12:35:59.378 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.378 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.378 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.379 INFO:teuthology.orchestra.run.smithi060.stdout: "agent_down_multiplier": { 2024-03-04T12:35:59.379 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "agent_down_multiplier", 2024-03-04T12:35:59.379 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:59.379 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.379 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.379 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "3.0", 2024-03-04T12:35:59.379 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.379 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.379 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.380 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-04T12:35:59.380 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.380 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.380 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.380 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.380 INFO:teuthology.orchestra.run.smithi060.stdout: "agent_refresh_rate": { 2024-03-04T12:35:59.380 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "agent_refresh_rate", 2024-03-04T12:35:59.380 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.381 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.381 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.381 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "20", 2024-03-04T12:35:59.381 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.381 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.381 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.381 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-04T12:35:59.381 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.382 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.382 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.382 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.382 INFO:teuthology.orchestra.run.smithi060.stdout: "agent_starting_port": { 2024-03-04T12:35:59.382 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "agent_starting_port", 2024-03-04T12:35:59.382 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.382 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.382 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.382 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "4721", 2024-03-04T12:35:59.383 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.383 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.383 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.383 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-04T12:35:59.383 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.383 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.383 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.383 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.384 INFO:teuthology.orchestra.run.smithi060.stdout: "allow_ptrace": { 2024-03-04T12:35:59.384 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "allow_ptrace", 2024-03-04T12:35:59.384 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.384 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.384 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.384 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.384 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.384 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.385 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.385 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-04T12:35:59.385 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-04T12:35:59.385 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.385 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.385 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.385 INFO:teuthology.orchestra.run.smithi060.stdout: "autotune_interval": { 2024-03-04T12:35:59.385 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "autotune_interval", 2024-03-04T12:35:59.385 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.386 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.386 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.386 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:59.386 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.386 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.386 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.386 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-04T12:35:59.386 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.387 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.387 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.387 INFO:teuthology.orchestra.run.smithi060.stdout: "autotune_memory_target_ratio": { 2024-03-04T12:35:59.387 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "autotune_memory_target_ratio", 2024-03-04T12:35:59.387 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:59.387 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.387 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.387 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0.7", 2024-03-04T12:35:59.388 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.388 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.388 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.388 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-04T12:35:59.388 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.388 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.388 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.389 INFO:teuthology.orchestra.run.smithi060.stdout: "cephadm_log_destination": { 2024-03-04T12:35:59.389 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cephadm_log_destination", 2024-03-04T12:35:59.389 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.389 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.389 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.389 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.389 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.389 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.390 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.390 INFO:teuthology.orchestra.run.smithi060.stdout: "file", 2024-03-04T12:35:59.390 INFO:teuthology.orchestra.run.smithi060.stdout: "file,syslog", 2024-03-04T12:35:59.390 INFO:teuthology.orchestra.run.smithi060.stdout: "syslog" 2024-03-04T12:35:59.390 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.390 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-04T12:35:59.390 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.390 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.391 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.391 INFO:teuthology.orchestra.run.smithi060.stdout: "cgroups_split": { 2024-03-04T12:35:59.391 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cgroups_split", 2024-03-04T12:35:59.391 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.391 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.391 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.392 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.392 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.392 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.392 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.392 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-04T12:35:59.392 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.392 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.393 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.393 INFO:teuthology.orchestra.run.smithi060.stdout: "config_checks_enabled": { 2024-03-04T12:35:59.393 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "config_checks_enabled", 2024-03-04T12:35:59.393 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.393 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.393 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.394 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.394 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.394 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.394 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.394 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-04T12:35:59.394 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.394 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.395 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.395 INFO:teuthology.orchestra.run.smithi060.stdout: "config_dashboard": { 2024-03-04T12:35:59.395 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "config_dashboard", 2024-03-04T12:35:59.395 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.395 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.395 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.396 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.396 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.396 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.396 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.396 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-04T12:35:59.396 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.397 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.397 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.397 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_alertmanager": { 2024-03-04T12:35:59.397 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_alertmanager", 2024-03-04T12:35:59.397 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.397 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.397 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.398 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-04T12:35:59.398 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.398 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.398 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.398 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Prometheus container image", 2024-03-04T12:35:59.398 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.399 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.399 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.399 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.399 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_base": { 2024-03-04T12:35:59.399 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_base", 2024-03-04T12:35:59.399 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.400 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.400 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.400 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-04T12:35:59.400 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.400 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.400 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.400 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Container image name, without the tag", 2024-03-04T12:35:59.401 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.401 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.401 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.401 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.401 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_elasticsearch": { 2024-03-04T12:35:59.401 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_elasticsearch", 2024-03-04T12:35:59.401 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.401 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.401 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.402 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-04T12:35:59.402 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.402 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.402 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.402 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "elasticsearch container image", 2024-03-04T12:35:59.402 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.402 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.402 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.403 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_grafana": { 2024-03-04T12:35:59.403 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_grafana", 2024-03-04T12:35:59.403 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.403 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.403 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.403 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-04T12:35:59.403 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.403 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.403 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.404 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Prometheus container image", 2024-03-04T12:35:59.404 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.404 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.404 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.404 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.404 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_haproxy": { 2024-03-04T12:35:59.404 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_haproxy", 2024-03-04T12:35:59.404 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.405 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.405 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.405 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-04T12:35:59.405 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.405 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.405 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.405 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "HAproxy container image", 2024-03-04T12:35:59.405 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.406 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.406 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.406 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_jaeger_agent": { 2024-03-04T12:35:59.406 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_jaeger_agent", 2024-03-04T12:35:59.406 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.406 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.406 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.407 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-04T12:35:59.407 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.407 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.407 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.407 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Jaeger agent container image", 2024-03-04T12:35:59.407 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.407 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.407 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.408 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_jaeger_collector": { 2024-03-04T12:35:59.408 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_jaeger_collector", 2024-03-04T12:35:59.408 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.408 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.408 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.408 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-04T12:35:59.409 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.409 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.409 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.409 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Jaeger collector container image", 2024-03-04T12:35:59.409 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.409 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.409 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.410 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_jaeger_query": { 2024-03-04T12:35:59.410 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_jaeger_query", 2024-03-04T12:35:59.410 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.410 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.410 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.410 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-04T12:35:59.410 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.410 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.411 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.411 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Jaeger query container image", 2024-03-04T12:35:59.411 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.411 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.411 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.411 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_keepalived": { 2024-03-04T12:35:59.411 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_keepalived", 2024-03-04T12:35:59.412 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.412 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.412 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.412 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-04T12:35:59.412 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.412 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.412 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.412 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Keepalived container image", 2024-03-04T12:35:59.413 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.413 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.413 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.413 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.413 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_loki": { 2024-03-04T12:35:59.413 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_loki", 2024-03-04T12:35:59.413 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.414 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.414 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.414 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-04T12:35:59.414 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.414 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.414 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.414 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Loki container image", 2024-03-04T12:35:59.414 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.414 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.415 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.415 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.415 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_node_exporter": { 2024-03-04T12:35:59.415 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_node_exporter", 2024-03-04T12:35:59.415 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.415 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.415 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.415 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-04T12:35:59.416 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.416 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.416 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.416 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Prometheus container image", 2024-03-04T12:35:59.416 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.416 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.416 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.416 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.417 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_nvmeof": { 2024-03-04T12:35:59.417 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_nvmeof", 2024-03-04T12:35:59.417 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.417 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.417 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.417 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-04T12:35:59.417 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.417 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.418 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.418 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Nvme-of container image", 2024-03-04T12:35:59.418 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.418 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.418 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.418 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.418 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_prometheus": { 2024-03-04T12:35:59.418 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_prometheus", 2024-03-04T12:35:59.419 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.419 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.419 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.419 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-04T12:35:59.419 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.419 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.419 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.419 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Prometheus container image", 2024-03-04T12:35:59.420 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.420 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.420 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.420 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.420 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_promtail": { 2024-03-04T12:35:59.420 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_promtail", 2024-03-04T12:35:59.420 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.420 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.421 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.421 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-04T12:35:59.421 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.421 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.421 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.421 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Promtail container image", 2024-03-04T12:35:59.421 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.421 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.422 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.422 INFO:teuthology.orchestra.run.smithi060.stdout: "container_image_snmp_gateway": { 2024-03-04T12:35:59.422 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_image_snmp_gateway", 2024-03-04T12:35:59.422 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.422 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.422 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.422 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-04T12:35:59.422 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.423 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.423 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.423 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "SNMP Gateway container image", 2024-03-04T12:35:59.423 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.423 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.423 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.424 INFO:teuthology.orchestra.run.smithi060.stdout: "container_init": { 2024-03-04T12:35:59.424 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "container_init", 2024-03-04T12:35:59.424 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.424 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.424 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.424 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.424 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.424 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.424 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.425 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Run podman/docker with `--init`", 2024-03-04T12:35:59.425 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.425 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.425 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.425 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.425 INFO:teuthology.orchestra.run.smithi060.stdout: "daemon_cache_timeout": { 2024-03-04T12:35:59.425 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "daemon_cache_timeout", 2024-03-04T12:35:59.425 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.426 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.426 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.426 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:59.426 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.426 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.426 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.426 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-04T12:35:59.426 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.427 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.427 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.427 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.427 INFO:teuthology.orchestra.run.smithi060.stdout: "default_cephadm_command_timeout": { 2024-03-04T12:35:59.427 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default_cephadm_command_timeout", 2024-03-04T12:35:59.427 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.428 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.428 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.428 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "900", 2024-03-04T12:35:59.428 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.428 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.428 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.429 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-04T12:35:59.429 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.429 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.429 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.429 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.429 INFO:teuthology.orchestra.run.smithi060.stdout: "default_registry": { 2024-03-04T12:35:59.429 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "default_registry", 2024-03-04T12:35:59.429 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.430 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.430 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.430 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "docker.io", 2024-03-04T12:35:59.430 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.430 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.430 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.430 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-04T12:35:59.431 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.431 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.431 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.431 INFO:teuthology.orchestra.run.smithi060.stdout: "device_cache_timeout": { 2024-03-04T12:35:59.431 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "device_cache_timeout", 2024-03-04T12:35:59.431 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.432 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.432 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.432 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "1800", 2024-03-04T12:35:59.432 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.432 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.433 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.433 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "seconds to cache device inventory", 2024-03-04T12:35:59.433 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.433 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.433 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.433 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.433 INFO:teuthology.orchestra.run.smithi060.stdout: "device_enhanced_scan": { 2024-03-04T12:35:59.433 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "device_enhanced_scan", 2024-03-04T12:35:59.434 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.434 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.434 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.434 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.434 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.434 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.435 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.435 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-04T12:35:59.435 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.435 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.435 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.435 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.436 INFO:teuthology.orchestra.run.smithi060.stdout: "facts_cache_timeout": { 2024-03-04T12:35:59.436 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "facts_cache_timeout", 2024-03-04T12:35:59.436 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.436 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.436 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.436 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:59.436 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.437 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.437 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.437 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "seconds to cache host facts data", 2024-03-04T12:35:59.437 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.437 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.437 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.437 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.437 INFO:teuthology.orchestra.run.smithi060.stdout: "host_check_interval": { 2024-03-04T12:35:59.438 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "host_check_interval", 2024-03-04T12:35:59.438 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.438 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.438 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.438 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:59.438 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.438 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.438 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.438 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to perform a host check", 2024-03-04T12:35:59.439 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.439 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.439 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.439 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.439 INFO:teuthology.orchestra.run.smithi060.stdout: "hw_monitoring": { 2024-03-04T12:35:59.439 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "hw_monitoring", 2024-03-04T12:35:59.439 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.439 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.440 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.440 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.440 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.440 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.440 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.440 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-04T12:35:59.440 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.440 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.441 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.441 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.441 INFO:teuthology.orchestra.run.smithi060.stdout: "inventory_list_all": { 2024-03-04T12:35:59.441 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "inventory_list_all", 2024-03-04T12:35:59.441 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.441 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.441 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.442 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.442 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.442 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.442 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.442 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-04T12:35:59.442 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.442 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.443 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.443 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.443 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.443 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.443 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.443 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.443 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.443 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.443 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.444 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.444 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.444 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.444 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.444 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.444 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.445 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.445 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.445 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.445 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.445 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.445 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.445 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.445 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.446 INFO:teuthology.orchestra.run.smithi060.stdout: "log_refresh_metadata": { 2024-03-04T12:35:59.446 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_refresh_metadata", 2024-03-04T12:35:59.446 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.446 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.446 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.446 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.446 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.447 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.447 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.447 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-04T12:35:59.447 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.447 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.447 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.447 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.448 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.448 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.448 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.448 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.448 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.448 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.448 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.448 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.449 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.449 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-04T12:35:59.449 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.449 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.449 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.449 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.449 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.449 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.450 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.450 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.450 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.450 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.450 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.450 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.450 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.451 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.451 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.451 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.451 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.451 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.451 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.451 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.451 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.452 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.452 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.452 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.452 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.452 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.452 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.452 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.452 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.453 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.453 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.453 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.453 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.453 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.453 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.453 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.454 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.454 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.454 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.454 INFO:teuthology.orchestra.run.smithi060.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-04T12:35:59.454 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-04T12:35:59.454 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.454 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.454 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.455 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.455 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.455 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.455 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.455 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-04T12:35:59.455 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.455 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.455 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.456 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.456 INFO:teuthology.orchestra.run.smithi060.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-04T12:35:59.456 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-04T12:35:59.456 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.456 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.456 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.456 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "*", 2024-03-04T12:35:59.457 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.457 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.457 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.457 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-04T12:35:59.457 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.457 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.457 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.457 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.457 INFO:teuthology.orchestra.run.smithi060.stdout: "max_count_per_host": { 2024-03-04T12:35:59.458 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_count_per_host", 2024-03-04T12:35:59.458 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.458 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.458 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.458 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:59.458 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.458 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.458 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.458 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "max number of daemons per service per host", 2024-03-04T12:35:59.459 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.459 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.459 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.459 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.459 INFO:teuthology.orchestra.run.smithi060.stdout: "max_osd_draining_count": { 2024-03-04T12:35:59.459 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_osd_draining_count", 2024-03-04T12:35:59.459 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.459 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.460 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.460 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:59.460 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.460 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.460 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.460 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-04T12:35:59.460 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.460 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.461 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.461 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.461 INFO:teuthology.orchestra.run.smithi060.stdout: "migration_current": { 2024-03-04T12:35:59.461 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "migration_current", 2024-03-04T12:35:59.461 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.461 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.461 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.461 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.462 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.462 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.462 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.462 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "internal - do not modify", 2024-03-04T12:35:59.462 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.462 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.462 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.462 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.463 INFO:teuthology.orchestra.run.smithi060.stdout: "mode": { 2024-03-04T12:35:59.463 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mode", 2024-03-04T12:35:59.463 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.463 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.463 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.463 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "root", 2024-03-04T12:35:59.463 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.464 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.464 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.464 INFO:teuthology.orchestra.run.smithi060.stdout: "cephadm-package", 2024-03-04T12:35:59.464 INFO:teuthology.orchestra.run.smithi060.stdout: "root" 2024-03-04T12:35:59.464 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.464 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "mode for remote execution of cephadm", 2024-03-04T12:35:59.465 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.465 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.465 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.465 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.465 INFO:teuthology.orchestra.run.smithi060.stdout: "oob_default_addr": { 2024-03-04T12:35:59.465 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "oob_default_addr", 2024-03-04T12:35:59.465 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.466 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.466 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.466 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "169.254.1.1", 2024-03-04T12:35:59.466 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.466 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.466 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.466 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-04T12:35:59.467 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.467 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.467 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.467 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.467 INFO:teuthology.orchestra.run.smithi060.stdout: "prometheus_alerts_path": { 2024-03-04T12:35:59.467 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "prometheus_alerts_path", 2024-03-04T12:35:59.467 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.468 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.468 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.468 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-04T12:35:59.468 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.468 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.468 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.468 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-04T12:35:59.469 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.469 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.469 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.469 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.469 INFO:teuthology.orchestra.run.smithi060.stdout: "registry_insecure": { 2024-03-04T12:35:59.469 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "registry_insecure", 2024-03-04T12:35:59.469 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.470 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.470 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.470 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.470 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.470 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.470 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.471 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-04T12:35:59.471 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.471 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.471 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.471 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.471 INFO:teuthology.orchestra.run.smithi060.stdout: "registry_password": { 2024-03-04T12:35:59.471 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "registry_password", 2024-03-04T12:35:59.472 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.472 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.472 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.472 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.472 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.472 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.472 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.472 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-04T12:35:59.472 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.473 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.473 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.473 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.473 INFO:teuthology.orchestra.run.smithi060.stdout: "registry_url": { 2024-03-04T12:35:59.473 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "registry_url", 2024-03-04T12:35:59.473 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.473 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.473 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.474 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.474 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.474 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.474 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.474 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-04T12:35:59.474 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.474 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.474 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.475 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.475 INFO:teuthology.orchestra.run.smithi060.stdout: "registry_username": { 2024-03-04T12:35:59.475 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "registry_username", 2024-03-04T12:35:59.475 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.475 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.475 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.475 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.476 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.476 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.476 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.476 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-04T12:35:59.476 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.476 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.476 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.477 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.477 INFO:teuthology.orchestra.run.smithi060.stdout: "secure_monitoring_stack": { 2024-03-04T12:35:59.477 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "secure_monitoring_stack", 2024-03-04T12:35:59.477 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.477 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.477 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.477 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.478 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.478 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.478 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.478 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-04T12:35:59.478 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.478 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.478 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.479 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.479 INFO:teuthology.orchestra.run.smithi060.stdout: "service_discovery_port": { 2024-03-04T12:35:59.479 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "service_discovery_port", 2024-03-04T12:35:59.479 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.479 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.479 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.479 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8765", 2024-03-04T12:35:59.480 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.480 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.480 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.480 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "cephadm service discovery port", 2024-03-04T12:35:59.480 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.480 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.480 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.480 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.481 INFO:teuthology.orchestra.run.smithi060.stdout: "ssh_config_file": { 2024-03-04T12:35:59.481 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ssh_config_file", 2024-03-04T12:35:59.481 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.481 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.481 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.481 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.481 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.482 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.482 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.482 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-04T12:35:59.482 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.482 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.482 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.482 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.482 INFO:teuthology.orchestra.run.smithi060.stdout: "use_agent": { 2024-03-04T12:35:59.483 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "use_agent", 2024-03-04T12:35:59.483 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.483 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.483 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.483 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.483 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.483 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.483 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.484 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-04T12:35:59.484 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.484 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.484 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.484 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.484 INFO:teuthology.orchestra.run.smithi060.stdout: "use_repo_digest": { 2024-03-04T12:35:59.484 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "use_repo_digest", 2024-03-04T12:35:59.484 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.485 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.485 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.485 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.485 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.485 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.485 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.485 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-04T12:35:59.485 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.485 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.486 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.486 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.486 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_on_failed_host_check": { 2024-03-04T12:35:59.486 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_on_failed_host_check", 2024-03-04T12:35:59.486 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.486 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.486 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.486 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.487 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.487 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.487 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.487 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "raise a health warning if the host check fails", 2024-03-04T12:35:59.487 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.487 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.487 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.487 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.487 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_on_stray_daemons": { 2024-03-04T12:35:59.488 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_on_stray_daemons", 2024-03-04T12:35:59.488 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.488 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.488 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.488 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.488 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.488 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.488 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-04T12:35:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_on_stray_hosts": { 2024-03-04T12:35:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_on_stray_hosts", 2024-03-04T12:35:59.489 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-04T12:35:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.490 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.491 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.491 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.491 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.491 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.491 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.491 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crash", 2024-03-04T12:35:59.491 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.491 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.492 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.492 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.492 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.492 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.492 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.492 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.492 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.492 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.493 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.493 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.493 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.493 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.493 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.493 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.493 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.493 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.493 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.494 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.494 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.494 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.494 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.494 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.494 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.494 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.494 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.495 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.495 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.495 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.495 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.495 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.495 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.495 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.495 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.496 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.496 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.496 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.496 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.496 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.497 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.497 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.497 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.497 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.497 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.497 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.497 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.497 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.498 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.498 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.498 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.498 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.498 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.498 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.499 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.499 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.499 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.499 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.499 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.499 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.499 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.500 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.500 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.500 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.500 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.500 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.500 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.500 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.500 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.501 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.501 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.501 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.501 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.501 INFO:teuthology.orchestra.run.smithi060.stdout: "retain_interval": { 2024-03-04T12:35:59.501 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "retain_interval", 2024-03-04T12:35:59.501 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.501 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.502 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.502 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "31536000", 2024-03-04T12:35:59.502 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.502 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.502 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.502 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-04T12:35:59.502 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.502 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.503 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.503 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.503 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_recent_interval": { 2024-03-04T12:35:59.503 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_recent_interval", 2024-03-04T12:35:59.503 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.504 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.504 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.504 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "1209600", 2024-03-04T12:35:59.504 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.504 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.504 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.504 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-04T12:35:59.505 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.505 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.505 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.505 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.505 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.505 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.505 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.506 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "dashboard", 2024-03-04T12:35:59.506 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.506 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.506 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.506 INFO:teuthology.orchestra.run.smithi060.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-04T12:35:59.506 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-04T12:35:59.506 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.507 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.507 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.507 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:59.507 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.507 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.507 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.507 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.508 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.508 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.508 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.508 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.508 INFO:teuthology.orchestra.run.smithi060.stdout: "ALERTMANAGER_API_HOST": { 2024-03-04T12:35:59.508 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-04T12:35:59.508 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.508 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.509 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.509 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.509 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.509 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.509 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.509 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.509 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.509 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.509 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.510 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.510 INFO:teuthology.orchestra.run.smithi060.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-04T12:35:59.510 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-04T12:35:59.510 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.510 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.510 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.510 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.510 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.511 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.511 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.511 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.511 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.511 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.511 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.511 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.511 INFO:teuthology.orchestra.run.smithi060.stdout: "AUDIT_API_ENABLED": { 2024-03-04T12:35:59.511 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "AUDIT_API_ENABLED", 2024-03-04T12:35:59.512 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.512 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.512 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.512 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.512 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.512 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.512 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.513 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.513 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.513 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.513 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.513 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.513 INFO:teuthology.orchestra.run.smithi060.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-04T12:35:59.513 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-04T12:35:59.514 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.514 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.514 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.514 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.514 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.514 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.514 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.515 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.515 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.515 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.515 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.515 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.515 INFO:teuthology.orchestra.run.smithi060.stdout: "ENABLE_BROWSABLE_API": { 2024-03-04T12:35:59.515 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-04T12:35:59.515 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.516 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.516 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.516 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.516 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.516 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.516 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.516 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.517 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.517 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.517 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.517 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.517 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-04T12:35:59.517 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-04T12:35:59.517 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.518 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.518 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.518 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.518 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.518 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.518 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.518 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.518 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.519 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.519 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.519 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.519 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-04T12:35:59.519 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-04T12:35:59.519 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.519 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.520 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.520 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.520 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.520 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.520 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.520 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.520 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.520 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.521 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.521 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.521 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-04T12:35:59.521 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-04T12:35:59.521 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.521 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.521 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.521 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.522 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.522 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.522 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.522 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.522 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.522 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.522 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.522 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.523 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-04T12:35:59.523 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-04T12:35:59.523 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.523 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.523 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.523 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.523 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.524 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.524 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.524 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.524 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.524 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.524 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.524 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.524 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-04T12:35:59.525 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-04T12:35:59.525 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.525 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.525 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.525 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.525 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.525 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.525 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.526 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.526 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.526 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.526 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.526 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.526 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-04T12:35:59.526 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-04T12:35:59.526 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.527 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.527 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.527 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.527 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.527 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.527 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.527 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.527 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.528 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.528 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.528 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.528 INFO:teuthology.orchestra.run.smithi060.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-04T12:35:59.528 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-04T12:35:59.528 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.528 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.528 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.528 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.529 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.529 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.529 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.529 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.529 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.529 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.529 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.529 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.530 INFO:teuthology.orchestra.run.smithi060.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-04T12:35:59.530 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-04T12:35:59.530 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.530 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.530 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.530 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.530 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.530 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.531 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.531 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.531 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.531 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.531 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.531 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.531 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_API_PASSWORD": { 2024-03-04T12:35:59.532 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-04T12:35:59.532 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.532 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.532 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.532 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "admin", 2024-03-04T12:35:59.532 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.532 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.533 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.533 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.533 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.533 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.533 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.533 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.534 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-04T12:35:59.534 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-04T12:35:59.534 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.534 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.534 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.534 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.534 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.534 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.535 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.535 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.535 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.535 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.535 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.535 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.536 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_API_URL": { 2024-03-04T12:35:59.536 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_API_URL", 2024-03-04T12:35:59.536 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.536 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.536 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.536 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.536 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.537 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.537 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.537 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.537 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.537 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.537 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.538 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.538 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_API_USERNAME": { 2024-03-04T12:35:59.538 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-04T12:35:59.538 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.538 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.538 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.538 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "admin", 2024-03-04T12:35:59.539 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.539 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.539 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.539 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.539 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.539 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.540 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.540 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.540 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-04T12:35:59.540 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-04T12:35:59.540 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.540 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.540 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.541 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.541 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.541 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.541 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.541 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.541 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.542 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.542 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.542 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.542 INFO:teuthology.orchestra.run.smithi060.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-04T12:35:59.542 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-04T12:35:59.542 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.542 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.542 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.543 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.543 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.543 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.543 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.543 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.543 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.543 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.544 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.544 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.544 INFO:teuthology.orchestra.run.smithi060.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-04T12:35:59.544 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-04T12:35:59.544 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.544 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.545 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.545 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.545 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.545 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.545 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.545 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.545 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.545 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.546 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.546 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.546 INFO:teuthology.orchestra.run.smithi060.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-04T12:35:59.546 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-04T12:35:59.546 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.546 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.546 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.547 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.547 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.547 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.547 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.547 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.547 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.547 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.547 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.547 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.548 INFO:teuthology.orchestra.run.smithi060.stdout: "PROMETHEUS_API_HOST": { 2024-03-04T12:35:59.548 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-04T12:35:59.548 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.548 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.548 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.548 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.549 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.549 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.549 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.549 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.549 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.549 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.549 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.550 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.550 INFO:teuthology.orchestra.run.smithi060.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-04T12:35:59.550 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-04T12:35:59.550 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.550 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.550 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.550 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.550 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.551 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.551 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.551 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.551 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.551 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.551 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.551 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.552 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-04T12:35:59.552 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-04T12:35:59.552 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.552 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.552 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.552 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.552 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.552 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.553 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.553 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.553 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.553 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.553 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.553 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.553 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-04T12:35:59.554 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-04T12:35:59.554 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.554 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.554 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.554 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.554 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.554 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.554 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.555 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.555 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.555 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.555 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.555 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.555 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-04T12:35:59.555 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-04T12:35:59.555 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.556 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.556 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.556 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.556 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.556 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.556 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.556 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.556 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.556 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.557 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.557 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.557 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-04T12:35:59.557 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-04T12:35:59.557 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.557 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.557 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.557 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.558 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.558 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.558 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.558 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.558 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.558 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.558 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.558 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-04T12:35:59.559 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-04T12:35:59.559 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.559 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.559 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.559 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.559 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.559 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.560 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.560 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.560 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.560 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.560 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.560 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.560 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-04T12:35:59.560 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-04T12:35:59.561 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.561 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.561 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.561 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.561 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.561 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.561 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.561 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.562 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.562 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.562 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.562 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.562 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-04T12:35:59.562 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-04T12:35:59.562 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.562 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.562 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.563 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.563 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.563 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.563 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.563 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.563 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.563 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.563 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.564 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_ENABLED": { 2024-03-04T12:35:59.564 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-04T12:35:59.564 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.564 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.564 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.564 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.564 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.565 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.565 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.565 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.565 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.565 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.565 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.565 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-04T12:35:59.566 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-04T12:35:59.566 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.566 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.566 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.566 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-04T12:35:59.566 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.566 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.566 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.567 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.567 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.567 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.567 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.567 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.567 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-04T12:35:59.568 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-04T12:35:59.568 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.568 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.568 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.568 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:59.568 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.568 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.568 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.569 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.569 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.569 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.569 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.569 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.569 INFO:teuthology.orchestra.run.smithi060.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-04T12:35:59.569 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-04T12:35:59.570 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.570 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.570 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.570 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8", 2024-03-04T12:35:59.570 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.570 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.571 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.571 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.571 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.571 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.571 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.571 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.571 INFO:teuthology.orchestra.run.smithi060.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-04T12:35:59.572 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-04T12:35:59.572 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.572 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.572 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.572 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "45", 2024-03-04T12:35:59.572 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.572 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.573 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.573 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.573 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.573 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.573 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.573 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.573 INFO:teuthology.orchestra.run.smithi060.stdout: "RGW_API_ACCESS_KEY": { 2024-03-04T12:35:59.574 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-04T12:35:59.574 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.574 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.574 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.574 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.574 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.574 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.574 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.575 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.575 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.575 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.575 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.575 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.575 INFO:teuthology.orchestra.run.smithi060.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-04T12:35:59.575 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-04T12:35:59.575 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.576 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.576 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.576 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "admin", 2024-03-04T12:35:59.576 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.576 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.576 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.576 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.576 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.577 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.577 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.577 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.577 INFO:teuthology.orchestra.run.smithi060.stdout: "RGW_API_SECRET_KEY": { 2024-03-04T12:35:59.577 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-04T12:35:59.577 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.578 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.578 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.578 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.578 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.578 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.578 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.578 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.579 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.579 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.579 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.579 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.579 INFO:teuthology.orchestra.run.smithi060.stdout: "RGW_API_SSL_VERIFY": { 2024-03-04T12:35:59.579 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-04T12:35:59.579 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.580 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.580 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.580 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.580 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.580 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.580 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.580 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.581 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.581 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.581 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.581 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.581 INFO:teuthology.orchestra.run.smithi060.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-04T12:35:59.581 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-04T12:35:59.581 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.582 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.582 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.582 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:35:59.582 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.582 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.582 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.582 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.582 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.582 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.583 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.583 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.583 INFO:teuthology.orchestra.run.smithi060.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-04T12:35:59.583 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-04T12:35:59.583 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.583 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.583 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.583 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:59.584 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.584 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.584 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.584 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.584 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.584 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.584 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.584 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.585 INFO:teuthology.orchestra.run.smithi060.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-04T12:35:59.585 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-04T12:35:59.585 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.585 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.585 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.585 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5", 2024-03-04T12:35:59.585 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.585 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.586 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.586 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.586 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.586 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.586 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.586 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.586 INFO:teuthology.orchestra.run.smithi060.stdout: "cross_origin_url": { 2024-03-04T12:35:59.586 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cross_origin_url", 2024-03-04T12:35:59.587 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.587 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.587 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.587 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.587 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.587 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.587 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.587 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.588 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.588 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.588 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.588 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.588 INFO:teuthology.orchestra.run.smithi060.stdout: "crt_file": { 2024-03-04T12:35:59.588 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "crt_file", 2024-03-04T12:35:59.589 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.589 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.589 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.589 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.589 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.589 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.589 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.589 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.590 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.590 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.590 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.590 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.590 INFO:teuthology.orchestra.run.smithi060.stdout: "debug": { 2024-03-04T12:35:59.590 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "debug", 2024-03-04T12:35:59.590 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.590 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.591 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.591 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.591 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.591 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.591 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.591 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Enable/disable debug options", 2024-03-04T12:35:59.592 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.592 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.592 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.592 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.592 INFO:teuthology.orchestra.run.smithi060.stdout: "jwt_token_ttl": { 2024-03-04T12:35:59.592 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "jwt_token_ttl", 2024-03-04T12:35:59.592 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.593 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.593 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.593 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "28800", 2024-03-04T12:35:59.593 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.593 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.593 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.593 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.593 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.594 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.594 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.594 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.595 INFO:teuthology.orchestra.run.smithi060.stdout: "key_file": { 2024-03-04T12:35:59.595 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "key_file", 2024-03-04T12:35:59.596 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.596 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.596 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.596 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.596 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.596 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.596 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.597 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.597 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.597 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.597 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.597 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.597 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.597 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.597 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.597 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.598 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.598 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.598 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.598 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.598 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.598 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.598 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.598 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.599 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.599 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.599 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.599 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.599 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.599 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.599 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.599 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.600 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.600 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.600 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.600 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.600 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.600 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.600 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.600 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.600 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.601 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.601 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.601 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.601 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.601 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.601 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.601 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.601 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.602 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.602 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.602 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.602 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.602 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.602 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.602 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.602 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.602 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.603 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.603 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.603 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.603 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.603 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.603 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.603 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.603 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.603 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.604 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.604 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.604 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.604 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.604 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.604 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.604 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.604 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.605 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.605 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.605 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.605 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.605 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.605 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.605 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.605 INFO:teuthology.orchestra.run.smithi060.stdout: "motd": { 2024-03-04T12:35:59.605 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "motd", 2024-03-04T12:35:59.606 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.606 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.606 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.606 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.606 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.606 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.606 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.606 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "The message of the day", 2024-03-04T12:35:59.607 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.607 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.607 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.607 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.607 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect_resolve_ip_addr": { 2024-03-04T12:35:59.607 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "redirect_resolve_ip_addr", 2024-03-04T12:35:59.607 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.607 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.607 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.608 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.608 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.608 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.608 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.608 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.608 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.608 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.608 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.608 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.609 INFO:teuthology.orchestra.run.smithi060.stdout: "server_addr": { 2024-03-04T12:35:59.609 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_addr", 2024-03-04T12:35:59.609 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.609 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.609 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.609 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "::", 2024-03-04T12:35:59.609 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.609 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.610 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.610 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.610 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.610 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.610 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.610 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.610 INFO:teuthology.orchestra.run.smithi060.stdout: "server_port": { 2024-03-04T12:35:59.610 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_port", 2024-03-04T12:35:59.610 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.611 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.611 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.611 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8080", 2024-03-04T12:35:59.611 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.611 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.611 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.611 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.611 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.612 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.612 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.612 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.612 INFO:teuthology.orchestra.run.smithi060.stdout: "ssl": { 2024-03-04T12:35:59.612 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ssl", 2024-03-04T12:35:59.612 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.612 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.612 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.613 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.613 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.613 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.613 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.613 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.613 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.613 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.613 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.613 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.614 INFO:teuthology.orchestra.run.smithi060.stdout: "ssl_server_port": { 2024-03-04T12:35:59.614 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ssl_server_port", 2024-03-04T12:35:59.614 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.614 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.614 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.614 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8443", 2024-03-04T12:35:59.614 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.614 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.615 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.615 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.615 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.615 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.615 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.615 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.615 INFO:teuthology.orchestra.run.smithi060.stdout: "standby_behaviour": { 2024-03-04T12:35:59.615 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "standby_behaviour", 2024-03-04T12:35:59.615 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.616 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.616 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.616 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "redirect", 2024-03-04T12:35:59.616 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.616 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.616 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.616 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.616 INFO:teuthology.orchestra.run.smithi060.stdout: "redirect" 2024-03-04T12:35:59.616 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.617 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.617 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.617 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.617 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.617 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.617 INFO:teuthology.orchestra.run.smithi060.stdout: "standby_error_status_code": { 2024-03-04T12:35:59.617 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "standby_error_status_code", 2024-03-04T12:35:59.617 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.618 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.618 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.618 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "500", 2024-03-04T12:35:59.618 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "400", 2024-03-04T12:35:59.618 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "599", 2024-03-04T12:35:59.618 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.618 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.618 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.618 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.619 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.619 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.619 INFO:teuthology.orchestra.run.smithi060.stdout: "url_prefix": { 2024-03-04T12:35:59.619 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "url_prefix", 2024-03-04T12:35:59.619 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.619 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.619 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.619 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.620 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.620 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.620 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.620 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.620 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.620 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.620 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.620 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.621 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.621 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.621 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.621 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "devicehealth", 2024-03-04T12:35:59.621 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.621 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.621 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.621 INFO:teuthology.orchestra.run.smithi060.stdout: "enable_monitoring": { 2024-03-04T12:35:59.622 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "enable_monitoring", 2024-03-04T12:35:59.622 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.622 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.622 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.622 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.622 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.622 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.622 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.622 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "monitor device health metrics", 2024-03-04T12:35:59.623 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.623 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.623 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.623 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.623 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.623 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.623 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.623 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.624 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.624 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.624 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.624 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.624 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.624 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.624 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.624 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.624 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.625 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.625 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.625 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.625 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.625 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.625 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.625 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.625 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.626 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.626 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.626 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.626 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.626 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.626 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.626 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.626 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.626 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.627 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.627 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.627 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.627 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.627 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.627 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.627 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.627 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.627 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.628 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.628 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.628 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.628 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.628 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.628 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.628 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.628 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.628 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.629 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.629 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.629 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.629 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.629 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.629 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.629 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.629 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.630 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.630 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.630 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.630 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.630 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.630 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.630 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.630 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.630 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.631 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.631 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.631 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.631 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.631 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.631 INFO:teuthology.orchestra.run.smithi060.stdout: "mark_out_threshold": { 2024-03-04T12:35:59.631 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mark_out_threshold", 2024-03-04T12:35:59.631 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.632 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.632 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.632 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "2419200", 2024-03-04T12:35:59.632 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.632 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.632 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.632 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-04T12:35:59.632 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.632 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.633 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.633 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.633 INFO:teuthology.orchestra.run.smithi060.stdout: "pool_name": { 2024-03-04T12:35:59.633 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "pool_name", 2024-03-04T12:35:59.633 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.633 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.633 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.633 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "device_health_metrics", 2024-03-04T12:35:59.633 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.634 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.634 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.634 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-04T12:35:59.634 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.634 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.634 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.634 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.634 INFO:teuthology.orchestra.run.smithi060.stdout: "retention_period": { 2024-03-04T12:35:59.635 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "retention_period", 2024-03-04T12:35:59.635 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.635 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.635 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.635 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "15552000", 2024-03-04T12:35:59.635 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.635 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.635 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.636 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how long to retain device health metrics", 2024-03-04T12:35:59.636 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.636 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.636 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.636 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.636 INFO:teuthology.orchestra.run.smithi060.stdout: "scrape_frequency": { 2024-03-04T12:35:59.636 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "scrape_frequency", 2024-03-04T12:35:59.637 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.637 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.637 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.637 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "86400", 2024-03-04T12:35:59.637 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.637 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.637 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.637 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-04T12:35:59.637 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.638 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.638 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.638 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.638 INFO:teuthology.orchestra.run.smithi060.stdout: "self_heal": { 2024-03-04T12:35:59.638 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "self_heal", 2024-03-04T12:35:59.638 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.638 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.638 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.639 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.639 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.639 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.639 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.639 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-04T12:35:59.639 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.639 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.639 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.639 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.640 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:59.640 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:59.640 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.640 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.640 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.640 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:59.640 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.640 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.641 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.641 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how frequently to wake up and check device health", 2024-03-04T12:35:59.641 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.641 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.641 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.641 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.641 INFO:teuthology.orchestra.run.smithi060.stdout: "warn_threshold": { 2024-03-04T12:35:59.641 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "warn_threshold", 2024-03-04T12:35:59.641 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.642 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.642 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.642 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "7257600", 2024-03-04T12:35:59.642 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.642 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.642 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.642 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-04T12:35:59.642 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.643 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.643 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.643 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.643 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.643 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.643 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.643 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "diskprediction_local", 2024-03-04T12:35:59.643 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.643 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.644 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.644 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.644 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.644 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.644 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.644 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.644 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.644 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.645 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.645 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.645 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.645 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.645 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.645 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.645 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.645 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.646 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.646 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.646 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.646 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.646 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.646 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.646 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.646 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.646 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.647 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.647 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.647 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.647 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.647 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.647 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.647 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.647 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.647 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.648 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.648 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.648 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.648 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.648 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.648 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.648 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.648 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.649 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.649 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.649 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.649 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.649 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.649 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.649 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.649 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.649 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.650 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.650 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.650 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.650 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.650 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.650 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.650 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.650 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.650 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.651 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.651 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.651 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.651 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.651 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.651 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.651 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.651 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.651 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.652 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.652 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.652 INFO:teuthology.orchestra.run.smithi060.stdout: "predict_interval": { 2024-03-04T12:35:59.652 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "predict_interval", 2024-03-04T12:35:59.652 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.652 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.652 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.652 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "86400", 2024-03-04T12:35:59.652 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.653 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.653 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.653 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.653 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.653 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.653 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.653 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.653 INFO:teuthology.orchestra.run.smithi060.stdout: "predictor_model": { 2024-03-04T12:35:59.653 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "predictor_model", 2024-03-04T12:35:59.654 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.654 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.654 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.654 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "prophetstor", 2024-03-04T12:35:59.654 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.654 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.654 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.654 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.655 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.655 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.655 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.655 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.655 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:59.655 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:59.655 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.655 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.655 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.656 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "600", 2024-03-04T12:35:59.656 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.656 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.656 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.656 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.656 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.656 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.656 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.656 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.657 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.657 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.657 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.657 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "influx", 2024-03-04T12:35:59.657 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": false, 2024-03-04T12:35:59.657 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "influxdb python module not found", 2024-03-04T12:35:59.657 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.657 INFO:teuthology.orchestra.run.smithi060.stdout: "batch_size": { 2024-03-04T12:35:59.657 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "batch_size", 2024-03-04T12:35:59.658 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.658 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.658 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.658 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5000", 2024-03-04T12:35:59.658 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.658 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.658 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.658 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-04T12:35:59.658 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.659 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.659 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.659 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.659 INFO:teuthology.orchestra.run.smithi060.stdout: "database": { 2024-03-04T12:35:59.659 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "database", 2024-03-04T12:35:59.659 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.659 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.659 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.660 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "ceph", 2024-03-04T12:35:59.660 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.660 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.660 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.660 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-04T12:35:59.660 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.660 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.660 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.661 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.661 INFO:teuthology.orchestra.run.smithi060.stdout: "hostname": { 2024-03-04T12:35:59.661 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "hostname", 2024-03-04T12:35:59.661 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.661 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.661 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.661 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.661 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.661 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.662 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.662 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "InfluxDB server hostname", 2024-03-04T12:35:59.662 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.662 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.662 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.662 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.662 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:59.662 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:59.662 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.663 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.663 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.663 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "30", 2024-03-04T12:35:59.663 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "5", 2024-03-04T12:35:59.663 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.663 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.663 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-04T12:35:59.663 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.663 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.664 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.664 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.664 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.664 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.664 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.664 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.664 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.664 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.664 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.665 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.665 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.665 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.665 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.665 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.665 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.666 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.666 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.666 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.666 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.666 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.666 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.666 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.666 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.666 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.667 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.667 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.667 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.667 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.667 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.667 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.667 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.667 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.668 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.668 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.668 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.668 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.668 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.668 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.668 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.668 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.669 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.669 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.669 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.669 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.669 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.669 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.669 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.669 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.669 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.670 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.670 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.670 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.670 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.670 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.670 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.670 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.670 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.671 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.671 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.671 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.671 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.671 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.671 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.671 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.671 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.672 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.672 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.672 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.672 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.672 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.672 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.672 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.672 INFO:teuthology.orchestra.run.smithi060.stdout: "password": { 2024-03-04T12:35:59.673 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "password", 2024-03-04T12:35:59.673 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.673 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.673 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.673 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.673 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.673 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.673 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.674 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "password of InfluxDB server user", 2024-03-04T12:35:59.674 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.674 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.674 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.674 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.674 INFO:teuthology.orchestra.run.smithi060.stdout: "port": { 2024-03-04T12:35:59.674 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "port", 2024-03-04T12:35:59.674 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.675 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.675 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.675 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "8086", 2024-03-04T12:35:59.675 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.675 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.675 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.675 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "InfluxDB server port", 2024-03-04T12:35:59.676 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.676 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.676 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.676 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.676 INFO:teuthology.orchestra.run.smithi060.stdout: "ssl": { 2024-03-04T12:35:59.676 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ssl", 2024-03-04T12:35:59.676 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.676 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.677 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.677 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "false", 2024-03-04T12:35:59.677 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.677 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.677 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.677 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-04T12:35:59.677 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.678 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.678 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.678 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.678 INFO:teuthology.orchestra.run.smithi060.stdout: "threads": { 2024-03-04T12:35:59.678 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "threads", 2024-03-04T12:35:59.678 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.678 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.678 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.679 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5", 2024-03-04T12:35:59.679 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1", 2024-03-04T12:35:59.679 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "32", 2024-03-04T12:35:59.679 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.679 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-04T12:35:59.679 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.679 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.680 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.680 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.680 INFO:teuthology.orchestra.run.smithi060.stdout: "username": { 2024-03-04T12:35:59.680 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "username", 2024-03-04T12:35:59.680 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.680 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.680 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.680 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.681 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.681 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.681 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.681 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "username of InfluxDB server user", 2024-03-04T12:35:59.681 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.681 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.681 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.681 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.681 INFO:teuthology.orchestra.run.smithi060.stdout: "verify_ssl": { 2024-03-04T12:35:59.682 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "verify_ssl", 2024-03-04T12:35:59.682 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.682 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.682 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.682 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "true", 2024-03-04T12:35:59.682 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.682 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.682 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.683 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-04T12:35:59.683 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.683 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.683 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.683 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.683 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.683 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.683 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.683 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "insights", 2024-03-04T12:35:59.684 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.684 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.684 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.684 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.684 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.684 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.684 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.684 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.685 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.685 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.685 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.685 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.685 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.685 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.685 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.685 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.685 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.686 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.686 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.686 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.686 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.686 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.686 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.686 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.686 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.686 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.687 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.687 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.687 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.687 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.687 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.687 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.687 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.687 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.687 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.688 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.688 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.688 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.688 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.688 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.688 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.688 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.688 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.689 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.689 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.689 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.689 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.689 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.689 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.689 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.689 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.689 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.690 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.690 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.690 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.690 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.690 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.690 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.690 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.690 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.690 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.691 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.691 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.691 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.691 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.691 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.691 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.691 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.691 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.692 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.692 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.692 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.692 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.692 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.692 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.692 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.692 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "iostat", 2024-03-04T12:35:59.692 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.693 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.693 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.693 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.693 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.693 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.693 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.693 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.693 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.694 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.694 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.694 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.694 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.694 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.694 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.694 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.694 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.694 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.695 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.696 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.696 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.696 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.696 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.696 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.696 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.696 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.696 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.697 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.697 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.697 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.697 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.697 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.697 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.697 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.697 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.697 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.698 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.698 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.698 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.698 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.698 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.698 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.698 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.698 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.699 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.699 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.699 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.699 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.699 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.699 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.699 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.699 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.700 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.700 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.700 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.700 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.700 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.700 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.700 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.700 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.701 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.701 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.701 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.701 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.701 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.701 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.701 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.701 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.702 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "k8sevents", 2024-03-04T12:35:59.702 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.702 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.702 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.702 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_event_retention_days": { 2024-03-04T12:35:59.702 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "ceph_event_retention_days", 2024-03-04T12:35:59.702 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.702 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.702 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.703 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "7", 2024-03-04T12:35:59.703 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.703 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.703 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.703 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-04T12:35:59.703 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.703 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.703 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.703 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.704 INFO:teuthology.orchestra.run.smithi060.stdout: "config_check_secs": { 2024-03-04T12:35:59.704 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "config_check_secs", 2024-03-04T12:35:59.704 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.704 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.704 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.704 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:59.704 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "10", 2024-03-04T12:35:59.704 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.705 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.705 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-04T12:35:59.705 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.705 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.705 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.705 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.705 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.705 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.705 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.706 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.706 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.706 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.706 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.706 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.706 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.706 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.706 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.706 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.707 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.707 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.707 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.707 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.707 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.707 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.707 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.707 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.707 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.708 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.708 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.708 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.708 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.708 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.708 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.708 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.708 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.709 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.709 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.709 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.709 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.709 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.709 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.709 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.709 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.709 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.710 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.710 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.710 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.710 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.710 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.710 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.710 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.710 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.711 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.712 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.713 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "localpool", 2024-03-04T12:35:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "failure_domain": { 2024-03-04T12:35:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "failure_domain", 2024-03-04T12:35:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.714 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "host", 2024-03-04T12:35:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "failure domain for any created local pool", 2024-03-04T12:35:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-04T12:35:59.715 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.716 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.717 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.718 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.719 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.720 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.721 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.722 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.723 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.724 INFO:teuthology.orchestra.run.smithi060.stdout: "min_size": { 2024-03-04T12:35:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "min_size", 2024-03-04T12:35:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.725 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "default min_size for any created local pool", 2024-03-04T12:35:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-04T12:35:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "num_rep": { 2024-03-04T12:35:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "num_rep", 2024-03-04T12:35:59.726 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "3", 2024-03-04T12:35:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "default replica count for any created local pool", 2024-03-04T12:35:59.727 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.728 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.728 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.728 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.728 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_num": { 2024-03-04T12:35:59.728 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "pg_num", 2024-03-04T12:35:59.728 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.728 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.728 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.729 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "128", 2024-03-04T12:35:59.729 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.729 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.729 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.729 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "default pg_num for any created local pool", 2024-03-04T12:35:59.729 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.729 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.729 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.730 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.730 INFO:teuthology.orchestra.run.smithi060.stdout: "prefix": { 2024-03-04T12:35:59.730 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "prefix", 2024-03-04T12:35:59.730 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.730 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.730 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.730 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.730 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.730 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.731 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.731 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "name prefix for any created local pool", 2024-03-04T12:35:59.731 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.731 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.731 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.731 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.731 INFO:teuthology.orchestra.run.smithi060.stdout: "subtree": { 2024-03-04T12:35:59.731 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "subtree", 2024-03-04T12:35:59.732 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.732 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.732 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.732 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "rack", 2024-03-04T12:35:59.732 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.732 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.732 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.732 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-04T12:35:59.733 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-04T12:35:59.733 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.733 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.733 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.733 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.733 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.733 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.734 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mds_autoscaler", 2024-03-04T12:35:59.734 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.734 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.734 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.734 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.734 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.734 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.734 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.734 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.735 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.735 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.735 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.735 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.735 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.735 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.735 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.735 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.736 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.736 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.736 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.736 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.736 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.736 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.736 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.736 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.737 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.737 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.737 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.737 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.737 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.737 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.737 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.737 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.737 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.738 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.738 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.738 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.738 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.738 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.738 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.738 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.739 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.739 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.739 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.739 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.739 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.739 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.739 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.739 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.739 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.740 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.740 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.740 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.740 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.740 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.740 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.740 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.740 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.741 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.741 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.741 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.741 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.741 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.741 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.741 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.742 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.742 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.742 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.742 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.742 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.742 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.742 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.742 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.743 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.743 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.743 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.743 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.743 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mirroring", 2024-03-04T12:35:59.743 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.743 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.743 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.744 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.744 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.744 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.744 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.744 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.744 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.744 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.744 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.745 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.745 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.745 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.745 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.745 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.745 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.745 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.745 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.746 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.746 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.746 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.746 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.746 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.746 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.746 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.746 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.747 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.747 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.747 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.747 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.747 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.747 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.747 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.748 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.748 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.748 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.748 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.748 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.748 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.748 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.749 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.749 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.749 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.749 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.749 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.749 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.749 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.749 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.750 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.750 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.750 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.750 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.750 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.750 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.750 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.750 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.751 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.751 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.751 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.751 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.751 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.751 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.751 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.751 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.752 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.752 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.752 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.752 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.752 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.752 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.752 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.752 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.753 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.753 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.753 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.753 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "nfs", 2024-03-04T12:35:59.753 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.753 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.753 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.753 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.754 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.754 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.754 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.754 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.754 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.754 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.754 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.754 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.755 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.755 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.755 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.755 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.755 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.755 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.755 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.756 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.756 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.756 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.756 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.756 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.756 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.756 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.756 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.757 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.757 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.757 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.757 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.757 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.757 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.757 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.757 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.758 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.758 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.758 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.758 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.758 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.758 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.758 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.758 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.759 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.759 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.759 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.759 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.759 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.759 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.759 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.759 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.759 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.760 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.760 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.760 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.760 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.760 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.760 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.760 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.760 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.761 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.761 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.761 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.761 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.761 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.761 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.761 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.761 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.762 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.762 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.762 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.762 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.762 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.762 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.762 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.762 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.763 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "orchestrator", 2024-03-04T12:35:59.763 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.763 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.763 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.763 INFO:teuthology.orchestra.run.smithi060.stdout: "fail_fs": { 2024-03-04T12:35:59.763 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "fail_fs", 2024-03-04T12:35:59.763 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.763 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.763 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.764 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.764 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.764 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.764 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.764 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-04T12:35:59.764 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.764 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.764 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.765 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.765 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.765 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.765 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.765 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.765 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.765 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.765 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.766 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.766 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.766 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.766 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.766 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.766 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.766 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.766 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.767 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.767 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.767 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.767 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.767 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.767 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.767 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.767 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.768 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.768 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.768 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.768 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.768 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.768 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.768 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.768 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.769 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.769 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.769 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.769 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.769 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.769 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.769 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.769 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.769 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.770 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.770 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.770 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.770 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.770 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.770 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.770 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.770 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.771 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.771 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.771 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.771 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.771 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.771 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.771 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.772 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.772 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.772 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.772 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.772 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.772 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.772 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.772 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.772 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.773 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.773 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.773 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.773 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.773 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.773 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.773 INFO:teuthology.orchestra.run.smithi060.stdout: "orchestrator": { 2024-03-04T12:35:59.773 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "orchestrator", 2024-03-04T12:35:59.773 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.774 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.774 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.774 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.774 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.774 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.774 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.774 INFO:teuthology.orchestra.run.smithi060.stdout: "cephadm", 2024-03-04T12:35:59.774 INFO:teuthology.orchestra.run.smithi060.stdout: "rook", 2024-03-04T12:35:59.775 INFO:teuthology.orchestra.run.smithi060.stdout: "test_orchestrator" 2024-03-04T12:35:59.775 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.775 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Orchestrator backend", 2024-03-04T12:35:59.775 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.775 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.775 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.775 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.775 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.776 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.776 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.776 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "osd_perf_query", 2024-03-04T12:35:59.776 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.776 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.776 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.776 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.776 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.777 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.777 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.777 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.777 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.777 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.777 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.777 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.777 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.778 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.778 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.778 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.778 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.778 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.778 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.778 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.778 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.779 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.779 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.779 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.779 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.779 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.779 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.779 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.779 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.779 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.780 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.780 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.780 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.780 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.780 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.780 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.780 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.780 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.781 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.781 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.781 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.781 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.781 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.781 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.781 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.781 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.782 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.782 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.782 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.782 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.782 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.782 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.782 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.782 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.782 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.783 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.783 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.783 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.783 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.783 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.783 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.783 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.783 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.784 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.784 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.784 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.784 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.784 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.784 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.784 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.785 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.785 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.785 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.785 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.785 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.785 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.785 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "osd_support", 2024-03-04T12:35:59.785 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.786 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.786 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.786 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.786 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.786 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.786 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.786 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.786 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.787 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.787 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.787 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.787 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.787 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.787 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.787 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.787 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.788 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.788 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.788 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.788 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.788 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.788 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.788 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.789 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.789 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.789 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.789 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.789 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.789 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.789 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.789 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.789 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.790 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.790 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.790 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.790 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.790 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.790 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.790 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.790 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.791 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.791 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.791 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.791 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.791 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.791 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.791 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.792 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.792 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.792 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.792 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.792 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.792 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.792 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.792 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.793 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.793 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.793 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.793 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.793 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.793 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.793 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.793 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.794 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.794 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.794 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.794 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.794 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.794 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.794 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.794 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.795 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.795 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.795 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.795 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.795 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "pg_autoscaler", 2024-03-04T12:35:59.795 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.795 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.795 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.796 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.796 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.796 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.796 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.796 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.796 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.796 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.797 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.797 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.797 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.797 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.797 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.797 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.797 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.797 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.798 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.798 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.798 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.798 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.798 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.798 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.798 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.798 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.799 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.799 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.799 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.799 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.799 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.799 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.799 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.799 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.800 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.800 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.800 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.800 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.800 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.800 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.800 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.800 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.801 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.801 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.801 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.801 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.801 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.801 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.801 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.801 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.801 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.802 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.802 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.803 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.803 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.804 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.804 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.804 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.804 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.804 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.804 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.804 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.804 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.805 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.805 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.805 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.805 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.805 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.805 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.805 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.805 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.805 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.806 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.806 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:59.806 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:59.806 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.806 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.806 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.806 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:35:59.806 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.807 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.807 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.807 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.807 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.807 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.807 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.807 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.807 INFO:teuthology.orchestra.run.smithi060.stdout: "threshold": { 2024-03-04T12:35:59.808 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "threshold", 2024-03-04T12:35:59.808 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:59.808 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.808 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.808 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "3.0", 2024-03-04T12:35:59.808 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1.0", 2024-03-04T12:35:59.808 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.809 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.809 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "scaling threshold", 2024-03-04T12:35:59.809 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-04T12:35:59.809 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.809 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.809 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.809 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.809 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.810 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.810 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "progress", 2024-03-04T12:35:59.810 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.810 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.810 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.810 INFO:teuthology.orchestra.run.smithi060.stdout: "allow_pg_recovery_event": { 2024-03-04T12:35:59.810 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "allow_pg_recovery_event", 2024-03-04T12:35:59.810 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.811 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.811 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.811 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.811 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.811 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.811 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.811 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-04T12:35:59.812 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.812 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.812 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.812 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.812 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": { 2024-03-04T12:35:59.812 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "enabled", 2024-03-04T12:35:59.812 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.813 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.813 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.813 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.813 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.813 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.813 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.813 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.814 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.814 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.814 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.814 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.814 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.814 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.814 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.814 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.815 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.815 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.815 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.815 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.815 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.815 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.815 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.815 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.816 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.816 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.816 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.816 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.816 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.816 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.817 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.817 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.817 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.817 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.817 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.817 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.817 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.817 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.817 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.818 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.818 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.818 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.818 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.818 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.818 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.818 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.819 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.819 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.819 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.819 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.819 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.819 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.819 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.819 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.820 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.821 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.821 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.821 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.821 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.821 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.821 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.821 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.821 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.821 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.822 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.822 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.822 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.822 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.822 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.822 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.822 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.822 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.823 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.823 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.823 INFO:teuthology.orchestra.run.smithi060.stdout: "max_completed_events": { 2024-03-04T12:35:59.823 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_completed_events", 2024-03-04T12:35:59.823 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.823 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.823 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.823 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "50", 2024-03-04T12:35:59.824 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.824 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.824 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.824 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "number of past completed events to remember", 2024-03-04T12:35:59.824 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.824 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.824 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.825 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.825 INFO:teuthology.orchestra.run.smithi060.stdout: "sleep_interval": { 2024-03-04T12:35:59.825 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "sleep_interval", 2024-03-04T12:35:59.825 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.825 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.825 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.825 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "5", 2024-03-04T12:35:59.826 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.826 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.826 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.826 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "how long the module is going to sleep", 2024-03-04T12:35:59.826 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.826 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.827 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.827 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.827 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.827 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.827 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.828 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "prometheus", 2024-03-04T12:35:59.828 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.828 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.828 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.828 INFO:teuthology.orchestra.run.smithi060.stdout: "cache": { 2024-03-04T12:35:59.829 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "cache", 2024-03-04T12:35:59.829 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.829 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.829 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.829 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.830 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.830 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.830 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.830 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.830 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.831 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.831 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.831 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.831 INFO:teuthology.orchestra.run.smithi060.stdout: "exclude_perf_counters": { 2024-03-04T12:35:59.831 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "exclude_perf_counters", 2024-03-04T12:35:59.832 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.832 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.832 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.832 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.832 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.833 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.833 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.833 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-04T12:35:59.833 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-04T12:35:59.834 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.834 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.834 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.834 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.835 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.835 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.835 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.835 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.835 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.836 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.836 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.836 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.836 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.836 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.836 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.837 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.837 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.837 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.837 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.837 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.837 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.837 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.837 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.837 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.838 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.838 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.838 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.838 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.838 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.838 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.838 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.839 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.839 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.839 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.839 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.839 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.839 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.839 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.840 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.840 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.840 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.840 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.840 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.840 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.841 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.841 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.841 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.841 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.841 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.841 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.841 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.841 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.842 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.842 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.842 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.842 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.842 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.842 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.842 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.843 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.843 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.843 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.843 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.843 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.843 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.844 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.844 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.844 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.844 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.844 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.844 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.844 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.845 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.845 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_stats_pools": { 2024-03-04T12:35:59.845 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd_stats_pools", 2024-03-04T12:35:59.845 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.845 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.845 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.845 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.845 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.845 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.846 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.846 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.846 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.846 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.846 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.846 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.846 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-04T12:35:59.846 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-04T12:35:59.847 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.847 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.847 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.847 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "300", 2024-03-04T12:35:59.847 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.847 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.847 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.848 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.848 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.848 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.848 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.848 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.848 INFO:teuthology.orchestra.run.smithi060.stdout: "scrape_interval": { 2024-03-04T12:35:59.848 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "scrape_interval", 2024-03-04T12:35:59.849 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:59.849 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.849 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.849 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "15.0", 2024-03-04T12:35:59.849 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.849 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.849 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.849 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.850 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.850 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.850 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.850 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.850 INFO:teuthology.orchestra.run.smithi060.stdout: "server_addr": { 2024-03-04T12:35:59.850 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_addr", 2024-03-04T12:35:59.850 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.851 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.851 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.851 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "::", 2024-03-04T12:35:59.851 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.851 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.851 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.851 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-04T12:35:59.851 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.852 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.852 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.852 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.852 INFO:teuthology.orchestra.run.smithi060.stdout: "server_port": { 2024-03-04T12:35:59.852 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_port", 2024-03-04T12:35:59.852 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.853 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.853 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.853 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "9283", 2024-03-04T12:35:59.853 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.853 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.853 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.853 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-04T12:35:59.853 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.854 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.854 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.854 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.854 INFO:teuthology.orchestra.run.smithi060.stdout: "stale_cache_strategy": { 2024-03-04T12:35:59.854 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "stale_cache_strategy", 2024-03-04T12:35:59.854 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.854 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.855 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.855 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "log", 2024-03-04T12:35:59.855 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.855 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.855 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.855 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.855 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.855 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.856 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.856 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.856 INFO:teuthology.orchestra.run.smithi060.stdout: "standby_behaviour": { 2024-03-04T12:35:59.856 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "standby_behaviour", 2024-03-04T12:35:59.856 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.856 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.856 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.857 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "default", 2024-03-04T12:35:59.857 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.857 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.857 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.857 INFO:teuthology.orchestra.run.smithi060.stdout: "default", 2024-03-04T12:35:59.857 INFO:teuthology.orchestra.run.smithi060.stdout: "error" 2024-03-04T12:35:59.857 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.857 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.858 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.858 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.858 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.858 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.858 INFO:teuthology.orchestra.run.smithi060.stdout: "standby_error_status_code": { 2024-03-04T12:35:59.858 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "standby_error_status_code", 2024-03-04T12:35:59.858 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.859 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.859 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.859 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "500", 2024-03-04T12:35:59.859 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "400", 2024-03-04T12:35:59.859 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "599", 2024-03-04T12:35:59.859 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.859 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.859 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.860 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.860 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.860 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.860 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.860 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.860 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.860 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd_support", 2024-03-04T12:35:59.860 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.861 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.861 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.861 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.861 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.861 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.861 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.861 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.861 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.862 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.862 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.862 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.862 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.862 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.862 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.862 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.862 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.863 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.863 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.863 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.863 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.863 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.863 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.863 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.864 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.864 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.864 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.864 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.864 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.864 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.864 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.864 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.864 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.865 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.865 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.865 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.865 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.865 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.865 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.865 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.865 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.865 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.866 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.866 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.866 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.866 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.866 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.866 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.866 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.867 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.867 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.867 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.867 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.867 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.867 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.867 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.867 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.868 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.868 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.868 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.868 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.868 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.868 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.868 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.868 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.869 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.869 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.869 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.869 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.869 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.869 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.869 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.869 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: "max_concurrent_snap_create": { 2024-03-04T12:35:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_concurrent_snap_create", 2024-03-04T12:35:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10", 2024-03-04T12:35:59.870 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.871 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.871 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.871 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.871 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.871 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.871 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.872 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.872 INFO:teuthology.orchestra.run.smithi060.stdout: "mirror_snapshot_schedule": { 2024-03-04T12:35:59.872 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "mirror_snapshot_schedule", 2024-03-04T12:35:59.872 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.872 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.872 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.872 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.872 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.873 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.873 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.873 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.873 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.873 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.873 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.874 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.874 INFO:teuthology.orchestra.run.smithi060.stdout: "trash_purge_schedule": { 2024-03-04T12:35:59.874 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "trash_purge_schedule", 2024-03-04T12:35:59.874 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.874 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.874 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.874 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.874 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.875 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.875 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.875 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.875 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.875 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.875 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.876 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.876 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.876 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.876 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.876 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "restful", 2024-03-04T12:35:59.876 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.876 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.877 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.877 INFO:teuthology.orchestra.run.smithi060.stdout: "enable_auth": { 2024-03-04T12:35:59.877 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "enable_auth", 2024-03-04T12:35:59.877 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.877 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.877 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.877 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.878 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.878 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.878 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.878 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.878 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.878 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.879 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.879 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.879 INFO:teuthology.orchestra.run.smithi060.stdout: "key_file": { 2024-03-04T12:35:59.879 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "key_file", 2024-03-04T12:35:59.879 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.879 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.880 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.880 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.880 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.880 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.880 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.880 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.880 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.881 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.881 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.881 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.881 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.881 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.881 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.881 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.882 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.882 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.882 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.882 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.882 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.882 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.882 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.882 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.883 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.883 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.883 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.883 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.883 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.883 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.883 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.883 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.884 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.884 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.884 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.884 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.884 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.884 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.884 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.884 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.885 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.885 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.885 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.885 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.885 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.885 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.885 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.886 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.886 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.886 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.886 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.886 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.886 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.886 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.886 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.886 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.887 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.887 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.887 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.887 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.887 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.887 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.887 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.888 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.888 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.888 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.888 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.888 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.888 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.888 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.888 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.889 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.889 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.889 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.889 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.889 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.889 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.889 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.889 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.889 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.890 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.890 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.890 INFO:teuthology.orchestra.run.smithi060.stdout: "server_addr": { 2024-03-04T12:35:59.890 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_addr", 2024-03-04T12:35:59.890 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.890 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.890 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.890 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.891 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.891 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.891 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.891 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.891 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.891 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.891 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.891 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.891 INFO:teuthology.orchestra.run.smithi060.stdout: "server_port": { 2024-03-04T12:35:59.892 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "server_port", 2024-03-04T12:35:59.892 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.892 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.892 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.892 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.892 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.892 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.892 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.893 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.893 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.893 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.893 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.893 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.893 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.893 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.893 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.894 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rgw", 2024-03-04T12:35:59.894 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.894 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.894 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.894 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.894 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.894 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.894 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.894 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.895 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.895 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.895 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.895 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.895 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.895 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.895 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.895 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.896 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.896 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.896 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.896 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.896 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.896 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.896 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.896 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.897 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.897 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.897 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.897 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.897 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.897 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.897 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.897 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.898 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.898 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.898 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.898 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.898 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.898 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.898 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.898 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.898 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.899 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.899 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.899 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.899 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.899 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.899 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.899 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.899 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.900 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.900 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.900 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.900 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.900 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.900 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.900 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.900 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.901 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.901 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.901 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.901 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.901 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.901 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.901 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.901 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.902 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.902 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.902 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.902 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.902 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.902 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.902 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.902 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.903 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.903 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.903 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.903 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rook", 2024-03-04T12:35:59.903 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.903 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.903 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.903 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.904 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.904 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.904 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.904 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.904 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.904 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.904 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.904 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.905 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.905 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.905 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.905 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.905 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.905 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.905 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.905 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.906 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.906 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.906 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.906 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.906 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.906 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.906 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.906 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.906 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.907 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.907 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.907 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.907 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.907 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.907 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.907 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.907 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.908 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.908 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.908 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.908 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.908 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.908 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.908 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.909 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.909 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.909 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.909 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.909 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.909 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.909 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.909 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.910 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.910 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.910 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.910 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.910 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.910 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.910 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.910 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.911 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.911 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.911 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.911 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.911 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.911 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.911 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.911 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.911 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.912 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.912 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.912 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.912 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.912 INFO:teuthology.orchestra.run.smithi060.stdout: "storage_class": { 2024-03-04T12:35:59.912 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "storage_class", 2024-03-04T12:35:59.912 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.912 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.913 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.913 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "local", 2024-03-04T12:35:59.913 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.913 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.913 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.913 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-04T12:35:59.913 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.913 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.914 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.914 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.914 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.914 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.914 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.914 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "selftest", 2024-03-04T12:35:59.914 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.914 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.915 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.915 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.915 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.915 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.915 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.915 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.915 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.915 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.916 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.916 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.916 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.916 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.916 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.916 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.916 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.916 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.917 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.918 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.919 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.920 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.920 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.920 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.920 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.920 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.920 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.920 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.920 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.921 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.921 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.921 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.921 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.921 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.921 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.921 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.921 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.922 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: "roption1": { 2024-03-04T12:35:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "roption1", 2024-03-04T12:35:59.923 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.924 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.924 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.924 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.924 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.924 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.924 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.924 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: "roption2": { 2024-03-04T12:35:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "roption2", 2024-03-04T12:35:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.925 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.926 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "xyz", 2024-03-04T12:35:59.926 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.926 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.926 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.926 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.926 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.926 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.926 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption1": { 2024-03-04T12:35:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption1", 2024-03-04T12:35:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.927 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.928 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.928 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.928 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.928 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.928 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.928 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.928 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption2": { 2024-03-04T12:35:59.928 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption2", 2024-03-04T12:35:59.929 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.929 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.929 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.929 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.929 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.929 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.929 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.929 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.930 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.930 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.930 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.930 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.930 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption3": { 2024-03-04T12:35:59.930 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption3", 2024-03-04T12:35:59.930 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "float", 2024-03-04T12:35:59.930 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.931 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.931 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.931 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.931 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.931 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.931 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.931 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.931 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption4": { 2024-03-04T12:35:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption4", 2024-03-04T12:35:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.932 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.933 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption5": { 2024-03-04T12:35:59.934 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption5", 2024-03-04T12:35:59.934 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.934 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.934 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.934 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.934 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.934 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.934 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption6": { 2024-03-04T12:35:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption6", 2024-03-04T12:35:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.935 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.936 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.937 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.937 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.937 INFO:teuthology.orchestra.run.smithi060.stdout: "rwoption7": { 2024-03-04T12:35:59.937 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rwoption7", 2024-03-04T12:35:59.937 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:35:59.937 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.937 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.937 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.938 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "1", 2024-03-04T12:35:59.938 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "42", 2024-03-04T12:35:59.938 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.938 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.938 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.938 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.938 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "testkey": { 2024-03-04T12:35:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "testkey", 2024-03-04T12:35:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.939 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.940 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.940 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.940 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.940 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.940 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.940 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.940 INFO:teuthology.orchestra.run.smithi060.stdout: "testlkey": { 2024-03-04T12:35:59.940 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "testlkey", 2024-03-04T12:35:59.941 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.941 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.941 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.941 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.941 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.941 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.941 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.941 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.942 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.942 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.942 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.942 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.942 INFO:teuthology.orchestra.run.smithi060.stdout: "testnewline": { 2024-03-04T12:35:59.942 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "testnewline", 2024-03-04T12:35:59.942 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.942 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.943 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.944 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.944 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.944 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.944 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.944 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "snap_schedule", 2024-03-04T12:35:59.944 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.944 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "allow_m_granularity": { 2024-03-04T12:35:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "allow_m_granularity", 2024-03-04T12:35:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.945 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "allow minute scheduled snapshots", 2024-03-04T12:35:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.946 INFO:teuthology.orchestra.run.smithi060.stdout: "dump_on_update": { 2024-03-04T12:35:59.947 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "dump_on_update", 2024-03-04T12:35:59.947 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.947 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.947 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.947 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.947 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.947 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.947 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.948 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "dump database to debug log on update", 2024-03-04T12:35:59.948 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.948 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.948 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.948 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.948 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.948 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.948 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.949 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.950 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.950 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.950 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.950 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.950 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.950 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.950 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.950 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.951 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.951 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.951 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.951 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.951 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.951 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.951 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.951 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.952 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.952 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.952 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.952 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.952 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.952 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.952 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.952 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.953 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.954 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.955 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.955 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.955 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.955 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.955 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.955 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.955 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.956 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "stats", 2024-03-04T12:35:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.957 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.958 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.958 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.958 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.958 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.958 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.958 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.958 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.958 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.959 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.960 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.961 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.961 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.961 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.961 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.961 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.961 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.961 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.961 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.962 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.962 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.962 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.962 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.962 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.962 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.962 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.962 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.963 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.964 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.964 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.964 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.964 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.964 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.964 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.964 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.964 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.965 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.965 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.965 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.965 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.965 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.965 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.965 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.965 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.966 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.966 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.966 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.966 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.966 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.966 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.966 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.966 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "status", 2024-03-04T12:35:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.967 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.968 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.969 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.969 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.969 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.969 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.969 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.969 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.969 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.969 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.970 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.971 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.971 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.971 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.971 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.971 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.971 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.971 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.971 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.972 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.973 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.974 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.974 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.974 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.974 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.974 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.974 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.974 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.974 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.975 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.976 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.976 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "telegraf", 2024-03-04T12:35:59.976 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.976 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.976 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.976 INFO:teuthology.orchestra.run.smithi060.stdout: "address": { 2024-03-04T12:35:59.976 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "address", 2024-03-04T12:35:59.976 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-04T12:35:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.977 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.978 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.978 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.978 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:35:59.978 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:35:59.978 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:35:59.978 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.978 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.978 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "15", 2024-03-04T12:35:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.979 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:35:59.980 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:35:59.980 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.980 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.980 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.980 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.980 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.980 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.980 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.981 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.982 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.982 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.982 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.982 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:35:59.982 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:35:59.982 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.982 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.982 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.983 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.983 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.983 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.983 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.983 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.983 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.983 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.983 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.984 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.984 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:35:59.984 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:35:59.984 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.984 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.984 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.984 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:35:59.984 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:35:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:35:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:35:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:35:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:35:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:35:59.985 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:35:59.986 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:35:59.986 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.986 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.986 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.986 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.986 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.986 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:35:59.986 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:35:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:35:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.987 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.988 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.988 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.988 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.988 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:35:59.988 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.988 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:35:59.988 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "telemetry", 2024-03-04T12:35:59.988 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:35:59.989 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:35:59.989 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:35:59.989 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_basic": { 2024-03-04T12:35:59.989 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_basic", 2024-03-04T12:35:59.989 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.989 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.989 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.989 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.990 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.990 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.990 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.990 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-04T12:35:59.990 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.990 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.990 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.990 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.991 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_crash": { 2024-03-04T12:35:59.991 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_crash", 2024-03-04T12:35:59.991 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.991 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.991 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.991 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.991 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.991 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.992 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.992 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-04T12:35:59.992 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.992 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.992 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.992 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.992 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_device": { 2024-03-04T12:35:59.993 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_device", 2024-03-04T12:35:59.993 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.993 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.993 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.993 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:35:59.993 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.993 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-04T12:35:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_ident": { 2024-03-04T12:35:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_ident", 2024-03-04T12:35:59.994 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.995 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.995 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.995 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.995 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.995 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.995 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.995 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-04T12:35:59.996 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.996 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.996 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.996 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.996 INFO:teuthology.orchestra.run.smithi060.stdout: "channel_perf": { 2024-03-04T12:35:59.996 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "channel_perf", 2024-03-04T12:35:59.996 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:35:59.996 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:35:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-04T12:35:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.997 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.998 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.998 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.998 INFO:teuthology.orchestra.run.smithi060.stdout: "contact": { 2024-03-04T12:35:59.998 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "contact", 2024-03-04T12:35:59.998 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:35:59.998 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:35:59.998 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:35:59.998 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:35:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:35:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:35:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:35:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:35:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:35:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:35:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:35:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:35:59.999 INFO:teuthology.orchestra.run.smithi060.stdout: "description": { 2024-03-04T12:36:00.000 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "description", 2024-03-04T12:36:00.000 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.000 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.000 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.000 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:36:00.000 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.000 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.000 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.001 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.001 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.001 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.001 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.001 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.001 INFO:teuthology.orchestra.run.smithi060.stdout: "device_url": { 2024-03-04T12:36:00.001 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "device_url", 2024-03-04T12:36:00.001 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-04T12:36:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.002 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: "enabled": { 2024-03-04T12:36:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "enabled", 2024-03-04T12:36:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:36:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.003 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:36:00.004 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.004 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.004 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.004 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.004 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.004 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.004 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.004 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.005 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:36:00.005 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:36:00.005 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:36:00.005 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.005 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.005 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "24", 2024-03-04T12:36:00.005 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "8", 2024-03-04T12:36:00.005 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: "last_opt_revision": { 2024-03-04T12:36:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "last_opt_revision", 2024-03-04T12:36:00.006 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:36:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "1", 2024-03-04T12:36:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.007 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.008 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.008 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.008 INFO:teuthology.orchestra.run.smithi060.stdout: "leaderboard": { 2024-03-04T12:36:00.008 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "leaderboard", 2024-03-04T12:36:00.008 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:36:00.008 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.008 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.008 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:36:00.009 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.009 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.009 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.009 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.009 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.009 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.009 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.009 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.012 INFO:teuthology.orchestra.run.smithi060.stdout: "leaderboard_description": { 2024-03-04T12:36:00.012 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "leaderboard_description", 2024-03-04T12:36:00.012 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.012 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.012 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.012 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:36:00.012 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.012 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:36:00.013 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:36:00.014 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.014 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.014 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.014 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:36:00.014 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.014 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.014 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:36:00.014 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:36:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:36:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:36:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:36:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:36:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:36:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.015 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.016 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.016 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.016 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:36:00.016 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:36:00.016 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:36:00.016 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.016 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.016 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:36:00.017 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.017 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.017 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.017 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.017 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.017 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.017 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.018 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.018 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:36:00.018 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:36:00.018 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.018 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.018 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.018 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:36:00.018 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.019 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.019 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:36:00.019 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:36:00.019 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:36:00.019 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:36:00.019 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:36:00.019 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:36:00.019 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:36:00.020 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.020 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.020 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.020 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.020 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.020 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.020 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:36:00.020 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:36:00.021 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:36:00.021 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.021 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.021 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:36:00.021 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.021 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.021 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.022 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.022 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.022 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.022 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.022 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.022 INFO:teuthology.orchestra.run.smithi060.stdout: "organization": { 2024-03-04T12:36:00.022 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "organization", 2024-03-04T12:36:00.022 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.023 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.023 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.023 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:36:00.023 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.023 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.023 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.023 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.023 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.024 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.024 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.024 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.025 INFO:teuthology.orchestra.run.smithi060.stdout: "proxy": { 2024-03-04T12:36:00.025 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "proxy", 2024-03-04T12:36:00.025 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.025 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.025 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.025 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:36:00.025 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.025 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.026 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.026 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.026 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.026 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.026 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.026 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.026 INFO:teuthology.orchestra.run.smithi060.stdout: "url": { 2024-03-04T12:36:00.026 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "url", 2024-03-04T12:36:00.026 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-04T12:36:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.027 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "test_orchestrator", 2024-03-04T12:36:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:36:00.028 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:36:00.029 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:36:00.029 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:36:00.029 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:36:00.029 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.029 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.029 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.029 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:36:00.029 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.030 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.030 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:36:00.030 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:36:00.030 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:36:00.030 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:36:00.030 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:36:00.030 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:36:00.030 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:36:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:36:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:36:00.031 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:36:00.032 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.032 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.032 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:36:00.032 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.032 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.032 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.032 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.032 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.032 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.033 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.033 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.033 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:36:00.033 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:36:00.033 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.033 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.033 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:36:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:36:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:36:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:36:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:36:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:36:00.034 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:36:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:36:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.035 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:36:00.036 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:36:00.036 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:36:00.036 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.036 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.036 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:36:00.036 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.036 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.036 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.037 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.037 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.037 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.037 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.037 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.037 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.037 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "volumes", 2024-03-04T12:36:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:36:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:36:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:36:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:36:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:36:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.038 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.039 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.039 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:36:00.039 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.039 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.039 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:36:00.039 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:36:00.039 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:36:00.039 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:36:00.039 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:36:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:36:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:36:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.040 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:36:00.041 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:36:00.041 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:36:00.041 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.041 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.041 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:36:00.041 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.041 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.041 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.042 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.042 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.042 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.042 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.042 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.042 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:36:00.042 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:36:00.042 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:36:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:36:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:36:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:36:00.043 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:36:00.044 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:36:00.044 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:36:00.044 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:36:00.044 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.044 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.044 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.044 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.044 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.045 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.045 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:36:00.045 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:36:00.045 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:36:00.045 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.045 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.045 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:36:00.045 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: "max_concurrent_clones": { 2024-03-04T12:36:00.046 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "max_concurrent_clones", 2024-03-04T12:36:00.047 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:36:00.047 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.047 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.047 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "4", 2024-03-04T12:36:00.047 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.047 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.047 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.047 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-04T12:36:00.048 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.048 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.048 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.048 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.048 INFO:teuthology.orchestra.run.smithi060.stdout: "periodic_async_work": { 2024-03-04T12:36:00.048 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "periodic_async_work", 2024-03-04T12:36:00.048 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:36:00.048 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.048 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.049 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:36:00.049 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.049 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.049 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.049 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Periodically check for async work", 2024-03-04T12:36:00.049 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.049 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.049 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.050 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.050 INFO:teuthology.orchestra.run.smithi060.stdout: "snapshot_clone_delay": { 2024-03-04T12:36:00.050 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "snapshot_clone_delay", 2024-03-04T12:36:00.050 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:36:00.050 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.050 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.050 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "0", 2024-03-04T12:36:00.050 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.050 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.051 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.051 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-04T12:36:00.051 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.051 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.051 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.051 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.051 INFO:teuthology.orchestra.run.smithi060.stdout: "snapshot_clone_no_wait": { 2024-03-04T12:36:00.051 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "snapshot_clone_no_wait", 2024-03-04T12:36:00.052 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:36:00.052 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.052 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.052 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "True", 2024-03-04T12:36:00.052 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.052 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.052 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.052 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-04T12:36:00.052 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.053 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.053 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.053 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.053 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.053 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.053 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.053 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "zabbix", 2024-03-04T12:36:00.053 INFO:teuthology.orchestra.run.smithi060.stdout: "can_run": true, 2024-03-04T12:36:00.054 INFO:teuthology.orchestra.run.smithi060.stdout: "error_string": "", 2024-03-04T12:36:00.054 INFO:teuthology.orchestra.run.smithi060.stdout: "module_options": { 2024-03-04T12:36:00.054 INFO:teuthology.orchestra.run.smithi060.stdout: "discovery_interval": { 2024-03-04T12:36:00.054 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "discovery_interval", 2024-03-04T12:36:00.054 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "uint", 2024-03-04T12:36:00.054 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.054 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.054 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "100", 2024-03-04T12:36:00.054 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.055 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.055 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.055 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.055 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.055 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.055 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.055 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.055 INFO:teuthology.orchestra.run.smithi060.stdout: "identifier": { 2024-03-04T12:36:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "identifier", 2024-03-04T12:36:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:36:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.056 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: "interval": { 2024-03-04T12:36:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "interval", 2024-03-04T12:36:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "secs", 2024-03-04T12:36:00.057 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.058 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.058 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "60", 2024-03-04T12:36:00.058 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.058 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.058 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.058 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.058 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.058 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: "log_level": { 2024-03-04T12:36:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_level", 2024-03-04T12:36:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.059 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:36:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:36:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:36:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:36:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:36:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:36:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:36:00.060 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:36:00.061 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.061 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.061 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.061 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.061 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.061 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.061 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster": { 2024-03-04T12:36:00.061 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster", 2024-03-04T12:36:00.062 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:36:00.062 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.062 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.062 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:36:00.062 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.062 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.062 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.062 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_cluster_level": { 2024-03-04T12:36:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_cluster_level", 2024-03-04T12:36:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.063 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.064 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.064 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "info", 2024-03-04T12:36:00.064 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.064 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.064 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [ 2024-03-04T12:36:00.064 INFO:teuthology.orchestra.run.smithi060.stdout: "", 2024-03-04T12:36:00.064 INFO:teuthology.orchestra.run.smithi060.stdout: "critical", 2024-03-04T12:36:00.064 INFO:teuthology.orchestra.run.smithi060.stdout: "debug", 2024-03-04T12:36:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: "error", 2024-03-04T12:36:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: "info", 2024-03-04T12:36:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: "warning" 2024-03-04T12:36:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.065 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.066 INFO:teuthology.orchestra.run.smithi060.stdout: "log_to_file": { 2024-03-04T12:36:00.066 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "log_to_file", 2024-03-04T12:36:00.066 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "bool", 2024-03-04T12:36:00.066 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.066 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 1, 2024-03-04T12:36:00.066 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "False", 2024-03-04T12:36:00.066 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.066 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: "zabbix_host": { 2024-03-04T12:36:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "zabbix_host", 2024-03-04T12:36:00.067 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.068 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.068 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.068 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "", 2024-03-04T12:36:00.068 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.068 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.068 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.068 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.068 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: "zabbix_port": { 2024-03-04T12:36:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "zabbix_port", 2024-03-04T12:36:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "int", 2024-03-04T12:36:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.069 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "10051", 2024-03-04T12:36:00.070 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.070 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.070 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.070 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.070 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.070 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.070 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.070 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.070 INFO:teuthology.orchestra.run.smithi060.stdout: "zabbix_sender": { 2024-03-04T12:36:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "zabbix_sender", 2024-03-04T12:36:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "str", 2024-03-04T12:36:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "level": "advanced", 2024-03-04T12:36:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "flags": 0, 2024-03-04T12:36:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-04T12:36:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "min": "", 2024-03-04T12:36:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "max": "", 2024-03-04T12:36:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "enum_allowed": [], 2024-03-04T12:36:00.071 INFO:teuthology.orchestra.run.smithi060.stdout: "desc": "", 2024-03-04T12:36:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: "long_desc": "", 2024-03-04T12:36:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: "tags": [], 2024-03-04T12:36:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: "see_also": [] 2024-03-04T12:36:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: "services": { 2024-03-04T12:36:00.072 INFO:teuthology.orchestra.run.smithi060.stdout: "dashboard": "https://172.21.15.60:8443/" 2024-03-04T12:36:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: "always_on_modules": { 2024-03-04T12:36:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: "octopus": [ 2024-03-04T12:36:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: "balancer", 2024-03-04T12:36:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: "crash", 2024-03-04T12:36:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: "devicehealth", 2024-03-04T12:36:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: "orchestrator", 2024-03-04T12:36:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscaler", 2024-03-04T12:36:00.073 INFO:teuthology.orchestra.run.smithi060.stdout: "progress", 2024-03-04T12:36:00.074 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_support", 2024-03-04T12:36:00.074 INFO:teuthology.orchestra.run.smithi060.stdout: "status", 2024-03-04T12:36:00.074 INFO:teuthology.orchestra.run.smithi060.stdout: "telemetry", 2024-03-04T12:36:00.074 INFO:teuthology.orchestra.run.smithi060.stdout: "volumes" 2024-03-04T12:36:00.074 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.074 INFO:teuthology.orchestra.run.smithi060.stdout: "pacific": [ 2024-03-04T12:36:00.074 INFO:teuthology.orchestra.run.smithi060.stdout: "balancer", 2024-03-04T12:36:00.074 INFO:teuthology.orchestra.run.smithi060.stdout: "crash", 2024-03-04T12:36:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: "devicehealth", 2024-03-04T12:36:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: "orchestrator", 2024-03-04T12:36:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscaler", 2024-03-04T12:36:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: "progress", 2024-03-04T12:36:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_support", 2024-03-04T12:36:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: "status", 2024-03-04T12:36:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: "telemetry", 2024-03-04T12:36:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: "volumes" 2024-03-04T12:36:00.075 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: "quincy": [ 2024-03-04T12:36:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: "balancer", 2024-03-04T12:36:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: "crash", 2024-03-04T12:36:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: "devicehealth", 2024-03-04T12:36:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: "orchestrator", 2024-03-04T12:36:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscaler", 2024-03-04T12:36:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: "progress", 2024-03-04T12:36:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_support", 2024-03-04T12:36:00.076 INFO:teuthology.orchestra.run.smithi060.stdout: "status", 2024-03-04T12:36:00.077 INFO:teuthology.orchestra.run.smithi060.stdout: "telemetry", 2024-03-04T12:36:00.077 INFO:teuthology.orchestra.run.smithi060.stdout: "volumes" 2024-03-04T12:36:00.077 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.077 INFO:teuthology.orchestra.run.smithi060.stdout: "reef": [ 2024-03-04T12:36:00.077 INFO:teuthology.orchestra.run.smithi060.stdout: "balancer", 2024-03-04T12:36:00.077 INFO:teuthology.orchestra.run.smithi060.stdout: "crash", 2024-03-04T12:36:00.077 INFO:teuthology.orchestra.run.smithi060.stdout: "devicehealth", 2024-03-04T12:36:00.077 INFO:teuthology.orchestra.run.smithi060.stdout: "orchestrator", 2024-03-04T12:36:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscaler", 2024-03-04T12:36:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: "progress", 2024-03-04T12:36:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_support", 2024-03-04T12:36:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: "status", 2024-03-04T12:36:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: "telemetry", 2024-03-04T12:36:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: "volumes" 2024-03-04T12:36:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: ], 2024-03-04T12:36:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: "squid": [ 2024-03-04T12:36:00.078 INFO:teuthology.orchestra.run.smithi060.stdout: "balancer", 2024-03-04T12:36:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "crash", 2024-03-04T12:36:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "devicehealth", 2024-03-04T12:36:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "orchestrator", 2024-03-04T12:36:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "pg_autoscaler", 2024-03-04T12:36:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "progress", 2024-03-04T12:36:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "rbd_support", 2024-03-04T12:36:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "status", 2024-03-04T12:36:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "telemetry", 2024-03-04T12:36:00.079 INFO:teuthology.orchestra.run.smithi060.stdout: "volumes" 2024-03-04T12:36:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-04T12:36:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: "last_failure_osd_epoch": 94, 2024-03-04T12:36:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: "active_clients": [ 2024-03-04T12:36:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "devicehealth", 2024-03-04T12:36:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-04T12:36:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.080 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-04T12:36:00.081 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:0", 2024-03-04T12:36:00.081 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 2781269356 2024-03-04T12:36:00.081 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.081 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-04T12:36:00.081 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.081 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.081 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "libcephsqlite", 2024-03-04T12:36:00.081 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-04T12:36:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-04T12:36:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:0", 2024-03-04T12:36:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 2888212560 2024-03-04T12:36:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-04T12:36:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.082 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "rbd_support", 2024-03-04T12:36:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-04T12:36:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-04T12:36:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:0", 2024-03-04T12:36:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 909015368 2024-03-04T12:36:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-04T12:36:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.083 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.084 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "snap_schedule", 2024-03-04T12:36:00.084 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-04T12:36:00.084 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.084 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-04T12:36:00.084 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:0", 2024-03-04T12:36:00.084 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 253511382 2024-03-04T12:36:00.084 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.084 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-04T12:36:00.084 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "snap_schedule", 2024-03-04T12:36:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-04T12:36:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-04T12:36:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:0", 2024-03-04T12:36:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 2850191932 2024-03-04T12:36:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.085 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-04T12:36:00.086 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:00.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.086 INFO:teuthology.orchestra.run.smithi060.stdout: "name": "volumes", 2024-03-04T12:36:00.086 INFO:teuthology.orchestra.run.smithi060.stdout: "addrvec": [ 2024-03-04T12:36:00.086 INFO:teuthology.orchestra.run.smithi060.stdout: { 2024-03-04T12:36:00.086 INFO:teuthology.orchestra.run.smithi060.stdout: "type": "v2", 2024-03-04T12:36:00.086 INFO:teuthology.orchestra.run.smithi060.stdout: "addr": "172.21.15.60:0", 2024-03-04T12:36:00.086 INFO:teuthology.orchestra.run.smithi060.stdout: "nonce": 2321773996 2024-03-04T12:36:00.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.087 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-04T12:36:00.087 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:00.087 INFO:teuthology.orchestra.run.smithi060.stdout: ] 2024-03-04T12:36:00.087 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:36:00.089 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:58 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2834585757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:36:00.089 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:58 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2847694656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:36:00.089 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:58 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1467235893' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-04T12:36:00.089 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:35:59 smithi063 ceph-mon[36717]: pgmap v314: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:36:00.089 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:58 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2834585757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:36:00.090 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:58 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2847694656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:36:00.090 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:58 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1467235893' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-04T12:36:00.090 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:35:59 smithi086 ceph-mon[36851]: pgmap v314: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:36:00.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.857+0000 7f2737d97640 1 -- 172.21.15.60:0/1467235893 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f271c0a1fa0 msgr2=0x7f271c0a4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:00.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.857+0000 7f2737d97640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f271c0a1fa0 0x7f271c0a4460 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f272c006010 tx=0x7f272c005dd0 comp rx=0 tx=0).stop 2024-03-04T12:36:00.090 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.857+0000 7f2737d97640 1 -- 172.21.15.60:0/1467235893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730106740 msgr2=0x7f27301012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.857+0000 7f2737d97640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730106740 0x7f27301012b0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f272000d950 tx=0x7f272000de10 comp rx=0 tx=0).stop 2024-03-04T12:36:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.857+0000 7f2737d97640 1 -- 172.21.15.60:0/1467235893 shutdown_connections 2024-03-04T12:36:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.857+0000 7f2737d97640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27301017f0 0x7f27301a5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.857+0000 7f2737d97640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f271c0a1fa0 0x7f271c0a4460 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.857+0000 7f2737d97640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2730106740 0x7f27301012b0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.857+0000 7f2737d97640 1 --2- 172.21.15.60:0/1467235893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27300ffe00 0x7f2730100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.857+0000 7f2737d97640 1 -- 172.21.15.60:0/1467235893 >> 172.21.15.60:0/1467235893 conn(0x7f2730075540 msgr2=0x7f27300fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:00.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.858+0000 7f2737d97640 1 -- 172.21.15.60:0/1467235893 shutdown_connections 2024-03-04T12:36:00.092 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:35:57.858+0000 7f2737d97640 1 -- 172.21.15.60:0/1467235893 wait complete. 2024-03-04T12:36:00.099 DEBUG:tasks.check_counter:Getting stats from a 2024-03-04T12:36:00.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-04T12:36:00.100 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:58 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2834585757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:36:00.100 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:58 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2847694656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:36:00.100 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:58 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1467235893' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-04T12:36:00.100 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:35:59 smithi060 ceph-mon[32745]: pgmap v314: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:36:00.354 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:36:01.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:01 smithi060 ceph-mon[32745]: pgmap v315: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-04T12:36:01.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:01 smithi086 ceph-mon[36851]: pgmap v315: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-04T12:36:01.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:01 smithi063 ceph-mon[36717]: pgmap v315: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 0 B/s wr, 14 op/s 2024-03-04T12:36:02.359 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:36:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-0": { 2024-03-04T12:36:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 993, 2024-03-04T12:36:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 969, 2024-03-04T12:36:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 2056512, 2024-03-04T12:36:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 364184, 2024-03-04T12:36:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 5, 2024-03-04T12:36:02.360 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 1, 2024-03-04T12:36:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.133585492, 2024-03-04T12:36:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.042423443, 2024-03-04T12:36:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.060430219, 2024-03-04T12:36:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.032895219, 2024-03-04T12:36:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 968, 2024-03-04T12:36:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.045895214, 2024-03-04T12:36:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000047412 2024-03-04T12:36:02.361 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 1807872, 2024-03-04T12:36:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 128864 2024-03-04T12:36:02.362 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-1": { 2024-03-04T12:36:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 617, 2024-03-04T12:36:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 773, 2024-03-04T12:36:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 154302, 2024-03-04T12:36:02.363 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 631534, 2024-03-04T12:36:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 7, 2024-03-04T12:36:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 5, 2024-03-04T12:36:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.081370756, 2024-03-04T12:36:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.027831023, 2024-03-04T12:36:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.019062463, 2024-03-04T12:36:02.364 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.027039779, 2024-03-04T12:36:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 771, 2024-03-04T12:36:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.030955019, 2024-03-04T12:36:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000040149 2024-03-04T12:36:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.365 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-04T12:36:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 437168 2024-03-04T12:36:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-2": { 2024-03-04T12:36:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 747, 2024-03-04T12:36:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 749, 2024-03-04T12:36:02.366 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 197478, 2024-03-04T12:36:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 237547, 2024-03-04T12:36:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 4, 2024-03-04T12:36:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 3, 2024-03-04T12:36:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.073841466, 2024-03-04T12:36:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.024313948, 2024-03-04T12:36:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.020226895, 2024-03-04T12:36:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.022605533, 2024-03-04T12:36:02.367 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 747, 2024-03-04T12:36:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.029964215, 2024-03-04T12:36:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000040112 2024-03-04T12:36:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.368 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-04T12:36:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 0 2024-03-04T12:36:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: "cct": { 2024-03-04T12:36:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: "total_workers": 1, 2024-03-04T12:36:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: "unhealthy_workers": 0 2024-03-04T12:36:02.369 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: "finisher-MDSRank": { 2024-03-04T12:36:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: "queue_len": 0, 2024-03-04T12:36:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: "complete_latency": { 2024-03-04T12:36:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 3382, 2024-03-04T12:36:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.112990255, 2024-03-04T12:36:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000033409 2024-03-04T12:36:02.370 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "finisher-PurgeQueue": { 2024-03-04T12:36:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "queue_len": 0, 2024-03-04T12:36:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "complete_latency": { 2024-03-04T12:36:02.371 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-03-04T12:36:02.372 INFO:teuthology.orchestra.run.smithi060.stdout: "request": 0, 2024-03-04T12:36:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "reply": 0, 2024-03-04T12:36:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "reply_latency": { 2024-03-04T12:36:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "slow_reply": 0, 2024-03-04T12:36:02.373 INFO:teuthology.orchestra.run.smithi060.stdout: "forward": 0, 2024-03-04T12:36:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_fetch_complete": 0, 2024-03-04T12:36:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_fetch_keys": 0, 2024-03-04T12:36:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_commit": 0, 2024-03-04T12:36:02.374 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_split": 0, 2024-03-04T12:36:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_merge": 0, 2024-03-04T12:36:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes": 0, 2024-03-04T12:36:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_top": 0, 2024-03-04T12:36:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_bottom": 0, 2024-03-04T12:36:02.375 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_pin_tail": 0, 2024-03-04T12:36:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_pinned": 0, 2024-03-04T12:36:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_expired": 0, 2024-03-04T12:36:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_with_caps": 0, 2024-03-04T12:36:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": 0, 2024-03-04T12:36:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "subtrees": 1, 2024-03-04T12:36:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse": 0, 2024-03-04T12:36:02.376 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_hit": 0, 2024-03-04T12:36:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_forward": 0, 2024-03-04T12:36:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_discover": 0, 2024-03-04T12:36:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_dir_fetch": 0, 2024-03-04T12:36:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_remote_ino": 0, 2024-03-04T12:36:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_lock": 0, 2024-03-04T12:36:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "load_cent": 0, 2024-03-04T12:36:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "q": 0, 2024-03-04T12:36:02.377 INFO:teuthology.orchestra.run.smithi060.stdout: "exported": 0, 2024-03-04T12:36:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: "exported_inodes": 0, 2024-03-04T12:36:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: "imported": 0, 2024-03-04T12:36:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: "imported_inodes": 0, 2024-03-04T12:36:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_dir_fetch": 0, 2024-03-04T12:36:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_backtrace_fetch": 0, 2024-03-04T12:36:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_peer_discover": 0, 2024-03-04T12:36:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rfiles": 0, 2024-03-04T12:36:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rbytes": 0, 2024-03-04T12:36:02.378 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rsnaps": 0, 2024-03-04T12:36:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_backtrace_fetch": 0, 2024-03-04T12:36:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_set_tag": 0, 2024-03-04T12:36:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_backtrace_repaired": 0, 2024-03-04T12:36:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_inotable_repaired": 0, 2024-03-04T12:36:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dir_inodes": 0, 2024-03-04T12:36:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dir_base_inodes": 0, 2024-03-04T12:36:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dirfrag_rstats": 0, 2024-03-04T12:36:02.379 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_file_inodes": 0, 2024-03-04T12:36:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_inode_file_caps": 0, 2024-03-04T12:36:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_revoke": 0, 2024-03-04T12:36:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_grant": 0, 2024-03-04T12:36:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_trunc": 0, 2024-03-04T12:36:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-04T12:36:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-04T12:36:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_caps": 0, 2024-03-04T12:36:02.380 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_caps_dirty": 0, 2024-03-04T12:36:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_cap_release": 0, 2024-03-04T12:36:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "process_request_cap_release": 0 2024-03-04T12:36:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cache": { 2024-03-04T12:36:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_update": 0, 2024-03-04T12:36:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_update_receipt": 0, 2024-03-04T12:36:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_try_discover": 0, 2024-03-04T12:36:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_send_discover": 0, 2024-03-04T12:36:02.381 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_handle_discover": 0, 2024-03-04T12:36:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays": 0, 2024-03-04T12:36:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays_delayed": 0, 2024-03-04T12:36:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays_enqueuing": 0, 2024-03-04T12:36:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_created": 0, 2024-03-04T12:36:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_enqueued": 0, 2024-03-04T12:36:02.382 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_reintegrated": 0, 2024-03-04T12:36:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_migrated": 0, 2024-03-04T12:36:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_processing": 0, 2024-03-04T12:36:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_enqueued": 0, 2024-03-04T12:36:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_prioritized": 0, 2024-03-04T12:36:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_started": 0, 2024-03-04T12:36:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_completed": 0, 2024-03-04T12:36:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_enqueue_scrub": 0, 2024-03-04T12:36:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_exportdir": 0, 2024-03-04T12:36:02.383 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_flush": 0, 2024-03-04T12:36:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_fragmentdir": 0, 2024-03-04T12:36:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_fragstats": 0, 2024-03-04T12:36:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_inodestats": 0 2024-03-04T12:36:02.384 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_log": { 2024-03-04T12:36:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "evlrg": 0, 2024-03-04T12:36:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "evadd": 0, 2024-03-04T12:36:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "evex": 0, 2024-03-04T12:36:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "evtrm": 0, 2024-03-04T12:36:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "ev": 6, 2024-03-04T12:36:02.385 INFO:teuthology.orchestra.run.smithi060.stdout: "evexg": 0, 2024-03-04T12:36:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "evexd": 0, 2024-03-04T12:36:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "segadd": 0, 2024-03-04T12:36:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "segex": 0, 2024-03-04T12:36:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "segtrm": 0, 2024-03-04T12:36:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "seg": 2, 2024-03-04T12:36:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "segmjr": 2, 2024-03-04T12:36:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "segexg": 0, 2024-03-04T12:36:02.386 INFO:teuthology.orchestra.run.smithi060.stdout: "segexd": 0, 2024-03-04T12:36:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "expos": 4194304, 2024-03-04T12:36:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "wrpos": 4196123, 2024-03-04T12:36:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "rdpos": 4195815, 2024-03-04T12:36:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "jlat": { 2024-03-04T12:36:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.387 INFO:teuthology.orchestra.run.smithi060.stdout: "replayed": 6 2024-03-04T12:36:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_mem": { 2024-03-04T12:36:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 3, 2024-03-04T12:36:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "ino+": 3, 2024-03-04T12:36:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "ino-": 0, 2024-03-04T12:36:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "dir": 1, 2024-03-04T12:36:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "dir+": 1, 2024-03-04T12:36:02.388 INFO:teuthology.orchestra.run.smithi060.stdout: "dir-": 0, 2024-03-04T12:36:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "dn": 0, 2024-03-04T12:36:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "dn+": 0, 2024-03-04T12:36:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "dn-": 0, 2024-03-04T12:36:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "cap": 0, 2024-03-04T12:36:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "cap+": 0, 2024-03-04T12:36:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "cap-": 0, 2024-03-04T12:36:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "rss": 50220, 2024-03-04T12:36:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: "heap": 215420 2024-03-04T12:36:02.389 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_server": { 2024-03-04T12:36:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "dispatch_client_request": 0, 2024-03-04T12:36:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "dispatch_server_request": 0, 2024-03-04T12:36:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_request": 0, 2024-03-04T12:36:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_session": 0, 2024-03-04T12:36:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_peer_request": 0, 2024-03-04T12:36:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "req_create_latency": { 2024-03-04T12:36:02.390 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getattr_latency": { 2024-03-04T12:36:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.391 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getfilelock_latency": { 2024-03-04T12:36:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "req_link_latency": { 2024-03-04T12:36:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.392 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookup_latency": { 2024-03-04T12:36:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.393 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookuphash_latency": { 2024-03-04T12:36:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.394 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupino_latency": { 2024-03-04T12:36:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.395 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupname_latency": { 2024-03-04T12:36:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupparent_latency": { 2024-03-04T12:36:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.396 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupsnap_latency": { 2024-03-04T12:36:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lssnap_latency": { 2024-03-04T12:36:02.397 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mkdir_latency": { 2024-03-04T12:36:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.398 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mknod_latency": { 2024-03-04T12:36:02.399 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mksnap_latency": { 2024-03-04T12:36:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.400 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "req_open_latency": { 2024-03-04T12:36:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "req_readdir_latency": { 2024-03-04T12:36:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.401 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rename_latency": { 2024-03-04T12:36:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.402 INFO:teuthology.orchestra.run.smithi060.stdout: "req_renamesnap_latency": { 2024-03-04T12:36:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "req_snapdiff_latency": { 2024-03-04T12:36:02.403 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmdir_latency": { 2024-03-04T12:36:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.404 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.405 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmsnap_latency": { 2024-03-04T12:36:02.405 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.405 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.405 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.405 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.405 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmxattr_latency": { 2024-03-04T12:36:02.405 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.405 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setattr_latency": { 2024-03-04T12:36:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.406 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setdirlayout_latency": { 2024-03-04T12:36:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setfilelock_latency": { 2024-03-04T12:36:02.407 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setlayout_latency": { 2024-03-04T12:36:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.408 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setxattr_latency": { 2024-03-04T12:36:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.409 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "req_symlink_latency": { 2024-03-04T12:36:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "req_unlink_latency": { 2024-03-04T12:36:02.410 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "cap_revoke_eviction": 0, 2024-03-04T12:36:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "cap_acquisition_throttle": 0, 2024-03-04T12:36:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getvxattr_latency": { 2024-03-04T12:36:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.411 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_sessions": { 2024-03-04T12:36:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "session_count": 2, 2024-03-04T12:36:02.412 INFO:teuthology.orchestra.run.smithi060.stdout: "session_add": 3, 2024-03-04T12:36:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "session_remove": 1, 2024-03-04T12:36:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "sessions_open": 2, 2024-03-04T12:36:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "sessions_stale": 0, 2024-03-04T12:36:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "total_load": 0, 2024-03-04T12:36:02.413 INFO:teuthology.orchestra.run.smithi060.stdout: "average_load": 0, 2024-03-04T12:36:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "avg_session_uptime": 2987, 2024-03-04T12:36:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "mdthresh_evicted": 0 2024-03-04T12:36:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "mempool": { 2024-03-04T12:36:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_bytes": 0, 2024-03-04T12:36:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_items": 0, 2024-03-04T12:36:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_bytes": 0, 2024-03-04T12:36:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_items": 0, 2024-03-04T12:36:02.414 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_bytes": 0, 2024-03-04T12:36:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_items": 0, 2024-03-04T12:36:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-04T12:36:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_items": 0, 2024-03-04T12:36:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-04T12:36:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_items": 0, 2024-03-04T12:36:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_bytes": 0, 2024-03-04T12:36:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_items": 0, 2024-03-04T12:36:02.415 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-04T12:36:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_items": 0, 2024-03-04T12:36:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_bytes": 0, 2024-03-04T12:36:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_items": 0, 2024-03-04T12:36:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_bytes": 0, 2024-03-04T12:36:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_items": 0, 2024-03-04T12:36:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-04T12:36:02.416 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_items": 0, 2024-03-04T12:36:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-04T12:36:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_items": 0, 2024-03-04T12:36:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_bytes": 0, 2024-03-04T12:36:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_items": 0, 2024-03-04T12:36:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_bytes": 0, 2024-03-04T12:36:02.417 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_items": 0, 2024-03-04T12:36:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-04T12:36:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_items": 0, 2024-03-04T12:36:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_bytes": 0, 2024-03-04T12:36:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_items": 0, 2024-03-04T12:36:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_bytes": 0, 2024-03-04T12:36:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_items": 0, 2024-03-04T12:36:02.418 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_bytes": 0, 2024-03-04T12:36:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_items": 0, 2024-03-04T12:36:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_bytes": 0, 2024-03-04T12:36:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_items": 0, 2024-03-04T12:36:02.419 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_bytes": 189410, 2024-03-04T12:36:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_items": 52, 2024-03-04T12:36:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_bytes": 0, 2024-03-04T12:36:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_items": 0, 2024-03-04T12:36:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_bytes": 0, 2024-03-04T12:36:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_items": 0, 2024-03-04T12:36:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_bytes": 0, 2024-03-04T12:36:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_items": 0, 2024-03-04T12:36:02.420 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_bytes": 0, 2024-03-04T12:36:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_items": 0, 2024-03-04T12:36:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_bytes": 9736, 2024-03-04T12:36:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_items": 203, 2024-03-04T12:36:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_bytes": 0, 2024-03-04T12:36:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_items": 0, 2024-03-04T12:36:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_bytes": 0, 2024-03-04T12:36:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_items": 0, 2024-03-04T12:36:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_bytes": 8320, 2024-03-04T12:36:02.421 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_items": 13, 2024-03-04T12:36:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_bytes": 0, 2024-03-04T12:36:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_items": 0, 2024-03-04T12:36:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_bytes": 0, 2024-03-04T12:36:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_items": 0 2024-03-04T12:36:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter": { 2024-03-04T12:36:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "op_active": 0, 2024-03-04T12:36:02.422 INFO:teuthology.orchestra.run.smithi060.stdout: "op_laggy": 0, 2024-03-04T12:36:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "op_send": 2098, 2024-03-04T12:36:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "op_send_bytes": 21, 2024-03-04T12:36:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "op_resend": 3, 2024-03-04T12:36:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "op_reply": 2098, 2024-03-04T12:36:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "op_latency": { 2024-03-04T12:36:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 2098, 2024-03-04T12:36:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 1.272987123, 2024-03-04T12:36:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000606762 2024-03-04T12:36:02.423 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: "op_inflight": 0, 2024-03-04T12:36:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: "oplen_avg": { 2024-03-04T12:36:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 2098, 2024-03-04T12:36:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 2099 2024-03-04T12:36:02.424 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "op": 2098, 2024-03-04T12:36:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "op_r": 2098, 2024-03-04T12:36:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "op_w": 0, 2024-03-04T12:36:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "op_rmw": 0, 2024-03-04T12:36:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "op_pg": 0, 2024-03-04T12:36:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_stat": 1348, 2024-03-04T12:36:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_create": 0, 2024-03-04T12:36:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_read": 748, 2024-03-04T12:36:02.425 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_write": 0, 2024-03-04T12:36:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_writefull": 0, 2024-03-04T12:36:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_writesame": 0, 2024-03-04T12:36:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_append": 0, 2024-03-04T12:36:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_zero": 0, 2024-03-04T12:36:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_truncate": 0, 2024-03-04T12:36:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_delete": 0, 2024-03-04T12:36:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_mapext": 0, 2024-03-04T12:36:02.426 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_sparse_read": 0, 2024-03-04T12:36:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_clonerange": 0, 2024-03-04T12:36:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_getxattr": 1, 2024-03-04T12:36:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_setxattr": 0, 2024-03-04T12:36:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_cmpxattr": 0, 2024-03-04T12:36:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_rmxattr": 0, 2024-03-04T12:36:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_resetxattrs": 0, 2024-03-04T12:36:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_call": 0, 2024-03-04T12:36:02.427 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_watch": 0, 2024-03-04T12:36:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_notify": 0, 2024-03-04T12:36:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_src_cmpxattr": 0, 2024-03-04T12:36:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_pgls": 0, 2024-03-04T12:36:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_pgls_filter": 0, 2024-03-04T12:36:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_other": 0, 2024-03-04T12:36:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_active": 0, 2024-03-04T12:36:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_send": 0, 2024-03-04T12:36:02.428 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_resend": 0, 2024-03-04T12:36:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_ping": 0, 2024-03-04T12:36:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_active": 0, 2024-03-04T12:36:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_send": 0, 2024-03-04T12:36:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_resend": 0, 2024-03-04T12:36:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_active": 0, 2024-03-04T12:36:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_send": 0, 2024-03-04T12:36:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_resend": 0, 2024-03-04T12:36:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_active": 0, 2024-03-04T12:36:02.429 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_send": 0, 2024-03-04T12:36:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_resend": 0, 2024-03-04T12:36:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "command_active": 0, 2024-03-04T12:36:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "command_send": 0, 2024-03-04T12:36:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "command_resend": 0, 2024-03-04T12:36:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "map_epoch": 103, 2024-03-04T12:36:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "map_full": 0, 2024-03-04T12:36:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "map_inc": 14, 2024-03-04T12:36:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_sessions": 5, 2024-03-04T12:36:02.430 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_session_open": 5, 2024-03-04T12:36:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_session_close": 0, 2024-03-04T12:36:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_laggy": 0, 2024-03-04T12:36:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_wr": 0, 2024-03-04T12:36:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_rd": 2, 2024-03-04T12:36:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_del": 0 2024-03-04T12:36:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "oft": { 2024-03-04T12:36:02.431 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_objs": 0, 2024-03-04T12:36:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_kv_pairs": 0, 2024-03-04T12:36:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_updates": 0, 2024-03-04T12:36:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_removes": 0 2024-03-04T12:36:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "osd-slow-ops": { 2024-03-04T12:36:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "slow_ops_count": 0 2024-03-04T12:36:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.432 INFO:teuthology.orchestra.run.smithi060.stdout: "purge_queue": { 2024-03-04T12:36:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_ops": 0, 2024-03-04T12:36:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_ops_high_water": 0, 2024-03-04T12:36:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing": 0, 2024-03-04T12:36:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_high_water": 0, 2024-03-04T12:36:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executed_ops": 0, 2024-03-04T12:36:02.433 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executed": 0, 2024-03-04T12:36:02.434 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_item_in_journal": 0 2024-03-04T12:36:02.434 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.434 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-04T12:36:02.434 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-03-04T12:36:02.434 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-03-04T12:36:02.434 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-03-04T12:36:02.434 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 2353, 2024-03-04T12:36:02.434 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 2180612, 2024-03-04T12:36:02.435 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:02.435 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 2353, 2024-03-04T12:36:02.435 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-03-04T12:36:02.435 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-03-04T12:36:02.436 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 2353, 2024-03-04T12:36:02.436 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 2180612, 2024-03-04T12:36:02.436 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-03-04T12:36:02.436 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.436 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.436 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.436 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:02.436 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.436 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-objecter_bytes": { 2024-03-04T12:36:02.437 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-03-04T12:36:02.437 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-03-04T12:36:02.437 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-03-04T12:36:02.437 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-03-04T12:36:02.437 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-03-04T12:36:02.437 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:02.437 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:02.437 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 2098, 2024-03-04T12:36:02.439 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 1824, 2024-03-04T12:36:02.439 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 2098, 2024-03-04T12:36:02.439 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 1824, 2024-03-04T12:36:02.439 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-03-04T12:36:02.440 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.440 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.440 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.440 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:02.440 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.440 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-objecter_ops": { 2024-03-04T12:36:02.440 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-03-04T12:36:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 1024, 2024-03-04T12:36:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-03-04T12:36:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-03-04T12:36:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-03-04T12:36:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 2098, 2024-03-04T12:36:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 2098, 2024-03-04T12:36:02.441 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 2098, 2024-03-04T12:36:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 2098, 2024-03-04T12:36:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-03-04T12:36:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.442 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-write_buf_throttle": { 2024-03-04T12:36:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-03-04T12:36:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 3758096384, 2024-03-04T12:36:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-03-04T12:36:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-03-04T12:36:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-03-04T12:36:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:02.443 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-03-04T12:36:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-03-04T12:36:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 0, 2024-03-04T12:36:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 0, 2024-03-04T12:36:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-03-04T12:36:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:02.444 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-write_buf_throttle-0x55df26b0c960": { 2024-03-04T12:36:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-03-04T12:36:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 3758096384, 2024-03-04T12:36:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-03-04T12:36:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-03-04T12:36:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-03-04T12:36:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:02.445 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-03-04T12:36:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-03-04T12:36:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 0, 2024-03-04T12:36:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 0, 2024-03-04T12:36:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-03-04T12:36:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:02.446 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:02.447 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:02.447 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:36:03.204 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-04T12:36:03.205 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-04T12:36:03.205 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-03-04T12:36:03.205 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-03-04T12:36:03.205 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-03-04T12:36:03.205 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-03-04T12:36:03.205 DEBUG:tasks.check_counter:Getting stats from d 2024-03-04T12:36:03.205 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-04T12:36:03.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:03 smithi060 ceph-mon[32745]: pgmap v316: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T12:36:03.455 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:36:03.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:03 smithi086 ceph-mon[36851]: pgmap v316: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T12:36:03.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:03 smithi063 ceph-mon[36717]: pgmap v316: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T12:36:05.267 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:05 smithi060 ceph-mon[32745]: pgmap v317: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T12:36:05.267 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:36:05.267 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-0": { 2024-03-04T12:36:05.267 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 0, 2024-03-04T12:36:05.267 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 2, 2024-03-04T12:36:05.268 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 0, 2024-03-04T12:36:05.268 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 0, 2024-03-04T12:36:05.268 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 4, 2024-03-04T12:36:05.268 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 0, 2024-03-04T12:36:05.268 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.001155768, 2024-03-04T12:36:05.268 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.000000000, 2024-03-04T12:36:05.268 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.000142103, 2024-03-04T12:36:05.268 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-04T12:36:05.269 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:05.269 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:05.269 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:05.269 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:05.269 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:05.269 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:05.269 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:05.269 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:05.269 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:05.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:05.270 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-04T12:36:05.270 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 0 2024-03-04T12:36:05.270 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:05.270 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-1": { 2024-03-04T12:36:05.270 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 6, 2024-03-04T12:36:05.270 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 167, 2024-03-04T12:36:05.270 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 3344, 2024-03-04T12:36:05.271 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 184640, 2024-03-04T12:36:05.271 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 5, 2024-03-04T12:36:05.271 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 3, 2024-03-04T12:36:05.271 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.015418997, 2024-03-04T12:36:05.271 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.010291174, 2024-03-04T12:36:05.271 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.002594121, 2024-03-04T12:36:05.271 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-04T12:36:05.271 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:05.271 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 165, 2024-03-04T12:36:05.272 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.017913100, 2024-03-04T12:36:05.272 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000108564 2024-03-04T12:36:05.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:05.272 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:05.272 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:05.272 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:05.272 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:05.272 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:05.273 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-04T12:36:05.273 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 184640 2024-03-04T12:36:05.273 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:05.273 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-2": { 2024-03-04T12:36:05.273 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 228, 2024-03-04T12:36:05.273 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 216, 2024-03-04T12:36:05.273 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 1707376, 2024-03-04T12:36:05.273 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 129312, 2024-03-04T12:36:05.273 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 2, 2024-03-04T12:36:05.274 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 1, 2024-03-04T12:36:05.274 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.046957158, 2024-03-04T12:36:05.274 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.015041108, 2024-03-04T12:36:05.274 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.037348280, 2024-03-04T12:36:05.274 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.004533889, 2024-03-04T12:36:05.274 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:05.274 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 215, 2024-03-04T12:36:05.274 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.011238660, 2024-03-04T12:36:05.274 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000052272 2024-03-04T12:36:05.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:05.275 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:05.275 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:05.275 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:05.275 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:05.275 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:05.275 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 1707376, 2024-03-04T12:36:05.275 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 129312 2024-03-04T12:36:05.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:05.276 INFO:teuthology.orchestra.run.smithi060.stdout: "cct": { 2024-03-04T12:36:05.276 INFO:teuthology.orchestra.run.smithi060.stdout: "total_workers": 0, 2024-03-04T12:36:05.276 INFO:teuthology.orchestra.run.smithi060.stdout: "unhealthy_workers": 0 2024-03-04T12:36:05.276 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:05.276 INFO:teuthology.orchestra.run.smithi060.stdout: "mempool": { 2024-03-04T12:36:05.276 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_bytes": 0, 2024-03-04T12:36:05.276 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_items": 0, 2024-03-04T12:36:05.277 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_bytes": 0, 2024-03-04T12:36:05.277 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_items": 0, 2024-03-04T12:36:05.277 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_bytes": 0, 2024-03-04T12:36:05.277 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_items": 0, 2024-03-04T12:36:05.277 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-04T12:36:05.277 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_items": 0, 2024-03-04T12:36:05.277 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-04T12:36:05.277 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_items": 0, 2024-03-04T12:36:05.278 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_bytes": 0, 2024-03-04T12:36:05.278 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_items": 0, 2024-03-04T12:36:05.278 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-04T12:36:05.278 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_items": 0, 2024-03-04T12:36:05.278 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_bytes": 0, 2024-03-04T12:36:05.278 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_items": 0, 2024-03-04T12:36:05.278 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_bytes": 0, 2024-03-04T12:36:05.278 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_items": 0, 2024-03-04T12:36:05.278 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-04T12:36:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_items": 0, 2024-03-04T12:36:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-04T12:36:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_items": 0, 2024-03-04T12:36:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_bytes": 0, 2024-03-04T12:36:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_items": 0, 2024-03-04T12:36:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_bytes": 0, 2024-03-04T12:36:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_items": 0, 2024-03-04T12:36:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-04T12:36:05.279 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_items": 0, 2024-03-04T12:36:05.280 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_bytes": 0, 2024-03-04T12:36:05.280 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_items": 0, 2024-03-04T12:36:05.280 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_bytes": 0, 2024-03-04T12:36:05.280 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_items": 0, 2024-03-04T12:36:05.280 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_bytes": 0, 2024-03-04T12:36:05.280 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_items": 0, 2024-03-04T12:36:05.280 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_bytes": 0, 2024-03-04T12:36:05.280 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_items": 0, 2024-03-04T12:36:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_bytes": 148714, 2024-03-04T12:36:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_items": 30, 2024-03-04T12:36:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_bytes": 0, 2024-03-04T12:36:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_items": 0, 2024-03-04T12:36:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_bytes": 0, 2024-03-04T12:36:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_items": 0, 2024-03-04T12:36:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_bytes": 0, 2024-03-04T12:36:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_items": 0, 2024-03-04T12:36:05.281 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_bytes": 0, 2024-03-04T12:36:05.282 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_items": 0, 2024-03-04T12:36:05.282 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_bytes": 0, 2024-03-04T12:36:05.282 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_items": 0, 2024-03-04T12:36:05.282 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_bytes": 0, 2024-03-04T12:36:05.282 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_items": 0, 2024-03-04T12:36:05.282 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_bytes": 0, 2024-03-04T12:36:05.282 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_items": 0, 2024-03-04T12:36:05.282 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_bytes": 928, 2024-03-04T12:36:05.282 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_items": 2, 2024-03-04T12:36:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_bytes": 0, 2024-03-04T12:36:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_items": 0, 2024-03-04T12:36:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_bytes": 0, 2024-03-04T12:36:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_items": 0 2024-03-04T12:36:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-04T12:36:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-03-04T12:36:05.283 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-03-04T12:36:05.284 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-03-04T12:36:05.284 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 230, 2024-03-04T12:36:05.284 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 1675189, 2024-03-04T12:36:05.284 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:05.284 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 230, 2024-03-04T12:36:05.284 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-03-04T12:36:05.284 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-03-04T12:36:05.284 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 230, 2024-03-04T12:36:05.284 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 1675189, 2024-03-04T12:36:05.285 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-03-04T12:36:05.285 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:05.285 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:05.285 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:05.285 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:05.285 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:05.285 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:36:05.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:05 smithi086 ceph-mon[36851]: pgmap v317: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T12:36:05.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:05 smithi063 ceph-mon[36717]: pgmap v317: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T12:36:06.094 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-04T12:36:06.094 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-04T12:36:06.094 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-04T12:36:06.094 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-04T12:36:06.094 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-04T12:36:06.095 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-04T12:36:06.095 DEBUG:tasks.check_counter:Getting stats from g 2024-03-04T12:36:06.095 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-04T12:36:06.356 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:36:07.131 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:07 smithi060 ceph-mon[32745]: pgmap v318: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 44 KiB/s wr, 32 op/s 2024-03-04T12:36:07.131 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:07 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:36:07.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:07 smithi086 ceph-mon[36851]: pgmap v318: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 44 KiB/s wr, 32 op/s 2024-03-04T12:36:07.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:07 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:36:07.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:07 smithi063 ceph-mon[36717]: pgmap v318: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 44 KiB/s wr, 32 op/s 2024-03-04T12:36:07.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:07 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T12:36:08.167 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:36:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-0": { 2024-03-04T12:36:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 1005, 2024-03-04T12:36:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 979, 2024-03-04T12:36:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 2063072, 2024-03-04T12:36:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 367646, 2024-03-04T12:36:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 5, 2024-03-04T12:36:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 1, 2024-03-04T12:36:08.168 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.128178055, 2024-03-04T12:36:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.050384147, 2024-03-04T12:36:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.055232896, 2024-03-04T12:36:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.024881948, 2024-03-04T12:36:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 979, 2024-03-04T12:36:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.049398325, 2024-03-04T12:36:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000050457 2024-03-04T12:36:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.169 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 1812752, 2024-03-04T12:36:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 130736 2024-03-04T12:36:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.170 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-1": { 2024-03-04T12:36:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 1365, 2024-03-04T12:36:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 1524, 2024-03-04T12:36:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 349113, 2024-03-04T12:36:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 870387, 2024-03-04T12:36:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 7, 2024-03-04T12:36:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 5, 2024-03-04T12:36:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.101829679, 2024-03-04T12:36:08.171 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.033729527, 2024-03-04T12:36:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.025009975, 2024-03-04T12:36:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.034558893, 2024-03-04T12:36:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 1521, 2024-03-04T12:36:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.051186075, 2024-03-04T12:36:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000033652 2024-03-04T12:36:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.172 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-04T12:36:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 436896 2024-03-04T12:36:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-2": { 2024-03-04T12:36:08.173 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 4, 2024-03-04T12:36:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 6, 2024-03-04T12:36:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 1629, 2024-03-04T12:36:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 1284, 2024-03-04T12:36:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 5, 2024-03-04T12:36:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 4, 2024-03-04T12:36:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.001413931, 2024-03-04T12:36:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.000065780, 2024-03-04T12:36:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.001097307, 2024-03-04T12:36:08.174 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.000122949, 2024-03-04T12:36:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 4, 2024-03-04T12:36:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.002011066, 2024-03-04T12:36:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000502766 2024-03-04T12:36:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.175 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-04T12:36:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 0 2024-03-04T12:36:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "cct": { 2024-03-04T12:36:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "total_workers": 1, 2024-03-04T12:36:08.176 INFO:teuthology.orchestra.run.smithi060.stdout: "unhealthy_workers": 0 2024-03-04T12:36:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: "finisher-MDSRank": { 2024-03-04T12:36:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: "queue_len": 0, 2024-03-04T12:36:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: "complete_latency": { 2024-03-04T12:36:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 3597, 2024-03-04T12:36:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.089725765, 2024-03-04T12:36:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000024944 2024-03-04T12:36:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:08.177 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: "finisher-PurgeQueue": { 2024-03-04T12:36:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: "queue_len": 0, 2024-03-04T12:36:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: "complete_latency": { 2024-03-04T12:36:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:08.178 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "mds": { 2024-03-04T12:36:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "request": 0, 2024-03-04T12:36:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "reply": 0, 2024-03-04T12:36:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "reply_latency": { 2024-03-04T12:36:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.179 INFO:teuthology.orchestra.run.smithi060.stdout: "slow_reply": 0, 2024-03-04T12:36:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "forward": 0, 2024-03-04T12:36:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_fetch_complete": 0, 2024-03-04T12:36:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_fetch_keys": 0, 2024-03-04T12:36:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_commit": 0, 2024-03-04T12:36:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_split": 0, 2024-03-04T12:36:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_merge": 0, 2024-03-04T12:36:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes": 0, 2024-03-04T12:36:08.180 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_top": 0, 2024-03-04T12:36:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_bottom": 0, 2024-03-04T12:36:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_pin_tail": 0, 2024-03-04T12:36:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_pinned": 0, 2024-03-04T12:36:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_expired": 0, 2024-03-04T12:36:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "inodes_with_caps": 0, 2024-03-04T12:36:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "caps": 0, 2024-03-04T12:36:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "subtrees": 1, 2024-03-04T12:36:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse": 0, 2024-03-04T12:36:08.181 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_hit": 0, 2024-03-04T12:36:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_forward": 0, 2024-03-04T12:36:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_discover": 0, 2024-03-04T12:36:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_dir_fetch": 0, 2024-03-04T12:36:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_remote_ino": 0, 2024-03-04T12:36:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "traverse_lock": 0, 2024-03-04T12:36:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "load_cent": 0, 2024-03-04T12:36:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "q": 0, 2024-03-04T12:36:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "exported": 0, 2024-03-04T12:36:08.182 INFO:teuthology.orchestra.run.smithi060.stdout: "exported_inodes": 0, 2024-03-04T12:36:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "imported": 0, 2024-03-04T12:36:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "imported_inodes": 0, 2024-03-04T12:36:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_dir_fetch": 0, 2024-03-04T12:36:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_backtrace_fetch": 0, 2024-03-04T12:36:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "openino_peer_discover": 0, 2024-03-04T12:36:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rfiles": 0, 2024-03-04T12:36:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rbytes": 0, 2024-03-04T12:36:08.183 INFO:teuthology.orchestra.run.smithi060.stdout: "root_rsnaps": 0, 2024-03-04T12:36:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_backtrace_fetch": 0, 2024-03-04T12:36:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_set_tag": 0, 2024-03-04T12:36:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_backtrace_repaired": 0, 2024-03-04T12:36:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_inotable_repaired": 0, 2024-03-04T12:36:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dir_inodes": 0, 2024-03-04T12:36:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dir_base_inodes": 0, 2024-03-04T12:36:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_dirfrag_rstats": 0, 2024-03-04T12:36:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "scrub_file_inodes": 0, 2024-03-04T12:36:08.184 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_inode_file_caps": 0, 2024-03-04T12:36:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_revoke": 0, 2024-03-04T12:36:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_grant": 0, 2024-03-04T12:36:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_trunc": 0, 2024-03-04T12:36:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-04T12:36:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-04T12:36:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_caps": 0, 2024-03-04T12:36:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_caps_dirty": 0, 2024-03-04T12:36:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_cap_release": 0, 2024-03-04T12:36:08.185 INFO:teuthology.orchestra.run.smithi060.stdout: "process_request_cap_release": 0 2024-03-04T12:36:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_cache": { 2024-03-04T12:36:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_update": 0, 2024-03-04T12:36:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_update_receipt": 0, 2024-03-04T12:36:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_try_discover": 0, 2024-03-04T12:36:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_send_discover": 0, 2024-03-04T12:36:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "dir_handle_discover": 0, 2024-03-04T12:36:08.186 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays": 0, 2024-03-04T12:36:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays_delayed": 0, 2024-03-04T12:36:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "num_strays_enqueuing": 0, 2024-03-04T12:36:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_created": 0, 2024-03-04T12:36:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_enqueued": 0, 2024-03-04T12:36:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_reintegrated": 0, 2024-03-04T12:36:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "strays_migrated": 0, 2024-03-04T12:36:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_processing": 0, 2024-03-04T12:36:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_enqueued": 0, 2024-03-04T12:36:08.187 INFO:teuthology.orchestra.run.smithi060.stdout: "num_recovering_prioritized": 0, 2024-03-04T12:36:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_started": 0, 2024-03-04T12:36:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "recovery_completed": 0, 2024-03-04T12:36:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_enqueue_scrub": 0, 2024-03-04T12:36:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_exportdir": 0, 2024-03-04T12:36:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_flush": 0, 2024-03-04T12:36:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_fragmentdir": 0, 2024-03-04T12:36:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_fragstats": 0, 2024-03-04T12:36:08.188 INFO:teuthology.orchestra.run.smithi060.stdout: "ireq_inodestats": 0 2024-03-04T12:36:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_log": { 2024-03-04T12:36:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "evlrg": 0, 2024-03-04T12:36:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "evadd": 0, 2024-03-04T12:36:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "evex": 0, 2024-03-04T12:36:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "evtrm": 0, 2024-03-04T12:36:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "ev": 5, 2024-03-04T12:36:08.189 INFO:teuthology.orchestra.run.smithi060.stdout: "evexg": 0, 2024-03-04T12:36:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "evexd": 0, 2024-03-04T12:36:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "segadd": 0, 2024-03-04T12:36:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "segex": 0, 2024-03-04T12:36:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "segtrm": 0, 2024-03-04T12:36:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "seg": 2, 2024-03-04T12:36:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "segmjr": 2, 2024-03-04T12:36:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "segexg": 0, 2024-03-04T12:36:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "segexd": 0, 2024-03-04T12:36:08.190 INFO:teuthology.orchestra.run.smithi060.stdout: "expos": 4194304, 2024-03-04T12:36:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "wrpos": 4195815, 2024-03-04T12:36:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "rdpos": 4195674, 2024-03-04T12:36:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "jlat": { 2024-03-04T12:36:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: "replayed": 5 2024-03-04T12:36:08.191 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_mem": { 2024-03-04T12:36:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "ino": 3, 2024-03-04T12:36:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "ino+": 3, 2024-03-04T12:36:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "ino-": 0, 2024-03-04T12:36:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "dir": 1, 2024-03-04T12:36:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "dir+": 1, 2024-03-04T12:36:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "dir-": 0, 2024-03-04T12:36:08.192 INFO:teuthology.orchestra.run.smithi060.stdout: "dn": 0, 2024-03-04T12:36:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "dn+": 0, 2024-03-04T12:36:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "dn-": 0, 2024-03-04T12:36:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "cap": 0, 2024-03-04T12:36:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "cap+": 0, 2024-03-04T12:36:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "cap-": 0, 2024-03-04T12:36:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "rss": 50336, 2024-03-04T12:36:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "heap": 215420 2024-03-04T12:36:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.193 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_server": { 2024-03-04T12:36:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "dispatch_client_request": 0, 2024-03-04T12:36:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "dispatch_server_request": 0, 2024-03-04T12:36:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_request": 0, 2024-03-04T12:36:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_client_session": 0, 2024-03-04T12:36:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "handle_peer_request": 0, 2024-03-04T12:36:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "req_create_latency": { 2024-03-04T12:36:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.194 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getattr_latency": { 2024-03-04T12:36:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getfilelock_latency": { 2024-03-04T12:36:08.195 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "req_link_latency": { 2024-03-04T12:36:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.196 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookup_latency": { 2024-03-04T12:36:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookuphash_latency": { 2024-03-04T12:36:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.197 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupino_latency": { 2024-03-04T12:36:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupname_latency": { 2024-03-04T12:36:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.198 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupparent_latency": { 2024-03-04T12:36:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lookupsnap_latency": { 2024-03-04T12:36:08.199 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "req_lssnap_latency": { 2024-03-04T12:36:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.200 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mkdir_latency": { 2024-03-04T12:36:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.201 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mknod_latency": { 2024-03-04T12:36:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "req_mksnap_latency": { 2024-03-04T12:36:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "req_open_latency": { 2024-03-04T12:36:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.203 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "req_readdir_latency": { 2024-03-04T12:36:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rename_latency": { 2024-03-04T12:36:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.204 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: "req_renamesnap_latency": { 2024-03-04T12:36:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: "req_snapdiff_latency": { 2024-03-04T12:36:08.205 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.206 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.206 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.206 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmdir_latency": { 2024-03-04T12:36:08.206 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.206 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.206 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.206 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.206 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmsnap_latency": { 2024-03-04T12:36:08.207 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.207 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.207 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.207 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.207 INFO:teuthology.orchestra.run.smithi060.stdout: "req_rmxattr_latency": { 2024-03-04T12:36:08.207 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.207 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.207 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.208 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setattr_latency": { 2024-03-04T12:36:08.208 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.208 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.208 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.208 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.208 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setdirlayout_latency": { 2024-03-04T12:36:08.208 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.209 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.209 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.209 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setfilelock_latency": { 2024-03-04T12:36:08.209 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.209 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.209 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.209 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.209 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setlayout_latency": { 2024-03-04T12:36:08.210 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.210 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.210 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.210 INFO:teuthology.orchestra.run.smithi060.stdout: "req_setxattr_latency": { 2024-03-04T12:36:08.210 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.210 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.210 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.210 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.211 INFO:teuthology.orchestra.run.smithi060.stdout: "req_symlink_latency": { 2024-03-04T12:36:08.211 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.211 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.211 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.211 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.211 INFO:teuthology.orchestra.run.smithi060.stdout: "req_unlink_latency": { 2024-03-04T12:36:08.211 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.211 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.211 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.212 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.212 INFO:teuthology.orchestra.run.smithi060.stdout: "cap_revoke_eviction": 0, 2024-03-04T12:36:08.212 INFO:teuthology.orchestra.run.smithi060.stdout: "cap_acquisition_throttle": 0, 2024-03-04T12:36:08.212 INFO:teuthology.orchestra.run.smithi060.stdout: "req_getvxattr_latency": { 2024-03-04T12:36:08.212 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.212 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.212 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.212 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:08.213 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.213 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_sessions": { 2024-03-04T12:36:08.213 INFO:teuthology.orchestra.run.smithi060.stdout: "session_count": 1, 2024-03-04T12:36:08.213 INFO:teuthology.orchestra.run.smithi060.stdout: "session_add": 2, 2024-03-04T12:36:08.213 INFO:teuthology.orchestra.run.smithi060.stdout: "session_remove": 1, 2024-03-04T12:36:08.213 INFO:teuthology.orchestra.run.smithi060.stdout: "sessions_open": 2, 2024-03-04T12:36:08.213 INFO:teuthology.orchestra.run.smithi060.stdout: "sessions_stale": 0, 2024-03-04T12:36:08.213 INFO:teuthology.orchestra.run.smithi060.stdout: "total_load": 0, 2024-03-04T12:36:08.213 INFO:teuthology.orchestra.run.smithi060.stdout: "average_load": 0, 2024-03-04T12:36:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: "avg_session_uptime": 2994, 2024-03-04T12:36:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: "mdthresh_evicted": 0 2024-03-04T12:36:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: "mempool": { 2024-03-04T12:36:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_bytes": 0, 2024-03-04T12:36:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_items": 0, 2024-03-04T12:36:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_bytes": 0, 2024-03-04T12:36:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_items": 0, 2024-03-04T12:36:08.214 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_bytes": 0, 2024-03-04T12:36:08.215 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_items": 0, 2024-03-04T12:36:08.215 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-04T12:36:08.215 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_items": 0, 2024-03-04T12:36:08.215 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-04T12:36:08.215 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_items": 0, 2024-03-04T12:36:08.215 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_bytes": 0, 2024-03-04T12:36:08.215 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_items": 0, 2024-03-04T12:36:08.215 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-04T12:36:08.216 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_items": 0, 2024-03-04T12:36:08.216 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_bytes": 0, 2024-03-04T12:36:08.216 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_items": 0, 2024-03-04T12:36:08.216 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_bytes": 0, 2024-03-04T12:36:08.216 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_items": 0, 2024-03-04T12:36:08.216 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-04T12:36:08.216 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_items": 0, 2024-03-04T12:36:08.216 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-04T12:36:08.216 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_items": 0, 2024-03-04T12:36:08.217 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_bytes": 0, 2024-03-04T12:36:08.217 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_items": 0, 2024-03-04T12:36:08.217 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_bytes": 0, 2024-03-04T12:36:08.217 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_items": 0, 2024-03-04T12:36:08.217 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-04T12:36:08.217 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_items": 0, 2024-03-04T12:36:08.217 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_bytes": 0, 2024-03-04T12:36:08.217 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_items": 0, 2024-03-04T12:36:08.218 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_bytes": 0, 2024-03-04T12:36:08.218 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_items": 0, 2024-03-04T12:36:08.218 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_bytes": 0, 2024-03-04T12:36:08.218 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_items": 0, 2024-03-04T12:36:08.218 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_bytes": 0, 2024-03-04T12:36:08.218 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_items": 0, 2024-03-04T12:36:08.218 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_bytes": 197473, 2024-03-04T12:36:08.218 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_items": 55, 2024-03-04T12:36:08.218 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_bytes": 0, 2024-03-04T12:36:08.219 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_items": 0, 2024-03-04T12:36:08.219 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_bytes": 0, 2024-03-04T12:36:08.219 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_items": 0, 2024-03-04T12:36:08.219 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_bytes": 0, 2024-03-04T12:36:08.219 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_items": 0, 2024-03-04T12:36:08.219 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_bytes": 0, 2024-03-04T12:36:08.219 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_items": 0, 2024-03-04T12:36:08.219 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_bytes": 9736, 2024-03-04T12:36:08.219 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_items": 203, 2024-03-04T12:36:08.220 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_bytes": 0, 2024-03-04T12:36:08.220 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_items": 0, 2024-03-04T12:36:08.220 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_bytes": 0, 2024-03-04T12:36:08.220 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_items": 0, 2024-03-04T12:36:08.220 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_bytes": 8320, 2024-03-04T12:36:08.220 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_items": 13, 2024-03-04T12:36:08.220 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_bytes": 0, 2024-03-04T12:36:08.220 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_items": 0, 2024-03-04T12:36:08.221 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_bytes": 0, 2024-03-04T12:36:08.221 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_items": 0 2024-03-04T12:36:08.221 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.221 INFO:teuthology.orchestra.run.smithi060.stdout: "objecter": { 2024-03-04T12:36:08.221 INFO:teuthology.orchestra.run.smithi060.stdout: "op_active": 0, 2024-03-04T12:36:08.221 INFO:teuthology.orchestra.run.smithi060.stdout: "op_laggy": 0, 2024-03-04T12:36:08.221 INFO:teuthology.orchestra.run.smithi060.stdout: "op_send": 2112, 2024-03-04T12:36:08.221 INFO:teuthology.orchestra.run.smithi060.stdout: "op_send_bytes": 21, 2024-03-04T12:36:08.221 INFO:teuthology.orchestra.run.smithi060.stdout: "op_resend": 3, 2024-03-04T12:36:08.222 INFO:teuthology.orchestra.run.smithi060.stdout: "op_reply": 2112, 2024-03-04T12:36:08.222 INFO:teuthology.orchestra.run.smithi060.stdout: "op_latency": { 2024-03-04T12:36:08.222 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 2112, 2024-03-04T12:36:08.222 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 1.534984383, 2024-03-04T12:36:08.222 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000726791 2024-03-04T12:36:08.222 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.222 INFO:teuthology.orchestra.run.smithi060.stdout: "op_inflight": 0, 2024-03-04T12:36:08.222 INFO:teuthology.orchestra.run.smithi060.stdout: "oplen_avg": { 2024-03-04T12:36:08.222 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 2112, 2024-03-04T12:36:08.223 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 2113 2024-03-04T12:36:08.223 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.223 INFO:teuthology.orchestra.run.smithi060.stdout: "op": 2112, 2024-03-04T12:36:08.223 INFO:teuthology.orchestra.run.smithi060.stdout: "op_r": 2112, 2024-03-04T12:36:08.223 INFO:teuthology.orchestra.run.smithi060.stdout: "op_w": 0, 2024-03-04T12:36:08.223 INFO:teuthology.orchestra.run.smithi060.stdout: "op_rmw": 0, 2024-03-04T12:36:08.223 INFO:teuthology.orchestra.run.smithi060.stdout: "op_pg": 0, 2024-03-04T12:36:08.223 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_stat": 1358, 2024-03-04T12:36:08.223 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_create": 0, 2024-03-04T12:36:08.224 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_read": 752, 2024-03-04T12:36:08.224 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_write": 0, 2024-03-04T12:36:08.224 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_writefull": 0, 2024-03-04T12:36:08.224 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_writesame": 0, 2024-03-04T12:36:08.224 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_append": 0, 2024-03-04T12:36:08.224 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_zero": 0, 2024-03-04T12:36:08.224 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_truncate": 0, 2024-03-04T12:36:08.224 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_delete": 0, 2024-03-04T12:36:08.224 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_mapext": 0, 2024-03-04T12:36:08.225 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_sparse_read": 0, 2024-03-04T12:36:08.225 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_clonerange": 0, 2024-03-04T12:36:08.225 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_getxattr": 1, 2024-03-04T12:36:08.225 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_setxattr": 0, 2024-03-04T12:36:08.225 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_cmpxattr": 0, 2024-03-04T12:36:08.225 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_rmxattr": 0, 2024-03-04T12:36:08.225 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_resetxattrs": 0, 2024-03-04T12:36:08.225 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_call": 0, 2024-03-04T12:36:08.225 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_watch": 0, 2024-03-04T12:36:08.226 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_notify": 0, 2024-03-04T12:36:08.226 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_src_cmpxattr": 0, 2024-03-04T12:36:08.226 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_pgls": 0, 2024-03-04T12:36:08.226 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_pgls_filter": 0, 2024-03-04T12:36:08.226 INFO:teuthology.orchestra.run.smithi060.stdout: "osdop_other": 0, 2024-03-04T12:36:08.226 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_active": 0, 2024-03-04T12:36:08.226 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_send": 0, 2024-03-04T12:36:08.226 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_resend": 0, 2024-03-04T12:36:08.227 INFO:teuthology.orchestra.run.smithi060.stdout: "linger_ping": 0, 2024-03-04T12:36:08.227 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_active": 0, 2024-03-04T12:36:08.227 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_send": 0, 2024-03-04T12:36:08.227 INFO:teuthology.orchestra.run.smithi060.stdout: "poolop_resend": 0, 2024-03-04T12:36:08.227 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_active": 0, 2024-03-04T12:36:08.227 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_send": 0, 2024-03-04T12:36:08.227 INFO:teuthology.orchestra.run.smithi060.stdout: "poolstat_resend": 0, 2024-03-04T12:36:08.227 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_active": 0, 2024-03-04T12:36:08.228 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_send": 0, 2024-03-04T12:36:08.228 INFO:teuthology.orchestra.run.smithi060.stdout: "statfs_resend": 0, 2024-03-04T12:36:08.228 INFO:teuthology.orchestra.run.smithi060.stdout: "command_active": 0, 2024-03-04T12:36:08.228 INFO:teuthology.orchestra.run.smithi060.stdout: "command_send": 0, 2024-03-04T12:36:08.228 INFO:teuthology.orchestra.run.smithi060.stdout: "command_resend": 0, 2024-03-04T12:36:08.228 INFO:teuthology.orchestra.run.smithi060.stdout: "map_epoch": 104, 2024-03-04T12:36:08.228 INFO:teuthology.orchestra.run.smithi060.stdout: "map_full": 0, 2024-03-04T12:36:08.228 INFO:teuthology.orchestra.run.smithi060.stdout: "map_inc": 15, 2024-03-04T12:36:08.228 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_sessions": 6, 2024-03-04T12:36:08.229 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_session_open": 6, 2024-03-04T12:36:08.229 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_session_close": 0, 2024-03-04T12:36:08.229 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_laggy": 0, 2024-03-04T12:36:08.229 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_wr": 0, 2024-03-04T12:36:08.229 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_rd": 2, 2024-03-04T12:36:08.229 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_del": 0 2024-03-04T12:36:08.229 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.229 INFO:teuthology.orchestra.run.smithi060.stdout: "oft": { 2024-03-04T12:36:08.230 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_objs": 0, 2024-03-04T12:36:08.230 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_kv_pairs": 0, 2024-03-04T12:36:08.230 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_updates": 0, 2024-03-04T12:36:08.230 INFO:teuthology.orchestra.run.smithi060.stdout: "omap_total_removes": 0 2024-03-04T12:36:08.230 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.230 INFO:teuthology.orchestra.run.smithi060.stdout: "osd-slow-ops": { 2024-03-04T12:36:08.230 INFO:teuthology.orchestra.run.smithi060.stdout: "slow_ops_count": 0 2024-03-04T12:36:08.230 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.230 INFO:teuthology.orchestra.run.smithi060.stdout: "purge_queue": { 2024-03-04T12:36:08.231 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_ops": 0, 2024-03-04T12:36:08.231 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_ops_high_water": 0, 2024-03-04T12:36:08.231 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing": 0, 2024-03-04T12:36:08.231 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executing_high_water": 0, 2024-03-04T12:36:08.231 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executed_ops": 0, 2024-03-04T12:36:08.231 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_executed": 0, 2024-03-04T12:36:08.231 INFO:teuthology.orchestra.run.smithi060.stdout: "pq_item_in_journal": 0 2024-03-04T12:36:08.231 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.231 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-04T12:36:08.232 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-03-04T12:36:08.232 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-03-04T12:36:08.232 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-03-04T12:36:08.232 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 2370, 2024-03-04T12:36:08.232 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 2184450, 2024-03-04T12:36:08.232 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:08.232 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 2370, 2024-03-04T12:36:08.232 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-03-04T12:36:08.233 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-03-04T12:36:08.233 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 2370, 2024-03-04T12:36:08.233 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 2184450, 2024-03-04T12:36:08.233 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-03-04T12:36:08.233 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.233 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.233 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.233 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:08.234 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.234 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-objecter_bytes": { 2024-03-04T12:36:08.234 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-03-04T12:36:08.234 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-03-04T12:36:08.234 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-03-04T12:36:08.234 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-03-04T12:36:08.234 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-03-04T12:36:08.234 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:08.235 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:08.235 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 2112, 2024-03-04T12:36:08.235 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 1516, 2024-03-04T12:36:08.235 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 2112, 2024-03-04T12:36:08.235 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 1516, 2024-03-04T12:36:08.235 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-03-04T12:36:08.235 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.235 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.235 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.236 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:08.236 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.236 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-objecter_ops": { 2024-03-04T12:36:08.236 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-03-04T12:36:08.236 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 1024, 2024-03-04T12:36:08.236 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-03-04T12:36:08.236 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-03-04T12:36:08.236 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-03-04T12:36:08.237 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:08.237 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:08.237 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 2112, 2024-03-04T12:36:08.237 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 2112, 2024-03-04T12:36:08.237 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 2112, 2024-03-04T12:36:08.237 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 2112, 2024-03-04T12:36:08.237 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-03-04T12:36:08.237 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.237 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.238 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.238 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:08.238 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.238 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-write_buf_throttle": { 2024-03-04T12:36:08.238 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-03-04T12:36:08.238 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 3758096384, 2024-03-04T12:36:08.238 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-03-04T12:36:08.238 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-03-04T12:36:08.238 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-03-04T12:36:08.239 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:08.239 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:08.239 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-03-04T12:36:08.239 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-03-04T12:36:08.239 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 0, 2024-03-04T12:36:08.239 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 0, 2024-03-04T12:36:08.239 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-03-04T12:36:08.239 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.240 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.240 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.240 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:08.240 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:08.240 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-write_buf_throttle-0x55b77f10e960": { 2024-03-04T12:36:08.240 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-03-04T12:36:08.240 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 3758096384, 2024-03-04T12:36:08.240 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-03-04T12:36:08.241 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 0, 2024-03-04T12:36:08.241 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 0, 2024-03-04T12:36:08.241 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:08.241 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:08.241 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-03-04T12:36:08.241 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-03-04T12:36:08.241 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 0, 2024-03-04T12:36:08.241 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 0, 2024-03-04T12:36:08.241 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-03-04T12:36:08.242 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:08.242 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:08.242 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:08.242 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:08.242 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:08.242 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:36:08.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:08 smithi060 ceph-mon[32745]: osdmap e104: 12 total, 12 up, 12 in 2024-03-04T12:36:08.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:08 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:08 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:08 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:08 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:08 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:08 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:08 smithi086 ceph-mon[36851]: osdmap e104: 12 total, 12 up, 12 in 2024-03-04T12:36:08.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:08 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:08 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:08 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:08 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:08 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:08 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:08 smithi063 ceph-mon[36717]: osdmap e104: 12 total, 12 up, 12 in 2024-03-04T12:36:08.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:08 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:08 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:08 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:08 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:08 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:08.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:08 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:09.002 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-04T12:36:09.002 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-04T12:36:09.003 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-03-04T12:36:09.003 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-03-04T12:36:09.003 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-03-04T12:36:09.003 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-03-04T12:36:09.003 DEBUG:tasks.check_counter:Getting stats from j 2024-03-04T12:36:09.003 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-04T12:36:09.285 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:09 smithi060 ceph-mon[32745]: pgmap v320: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 33 op/s 2024-03-04T12:36:09.285 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:09 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.286 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:09 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.286 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:09 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.286 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:09 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.286 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:09 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.286 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:09 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.286 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:09 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.287 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:09 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:09 smithi086 ceph-mon[36851]: pgmap v320: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 33 op/s 2024-03-04T12:36:09.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:09 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:09 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:09 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:09 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:09 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:09 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:09 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:09 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:09 smithi063 ceph-mon[36717]: pgmap v320: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 53 KiB/s wr, 33 op/s 2024-03-04T12:36:09.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:09 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:09 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:09 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:09 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:09 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:09 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:09 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:09 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:09.823 INFO:teuthology.orchestra.run.smithi060.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T12:36:10.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:10 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi086 to 174.1M 2024-03-04T12:36:10.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:10 smithi086 ceph-mon[36851]: Unable to set osd_memory_target on smithi086 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-04T12:36:10.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:10 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi063 to 174.1M 2024-03-04T12:36:10.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:10 smithi086 ceph-mon[36851]: Unable to set osd_memory_target on smithi063 to 182642073: error parsing value: Value '182642073' is below minimum 939524096 2024-03-04T12:36:10.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:10 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi086 to 174.1M 2024-03-04T12:36:10.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:10 smithi063 ceph-mon[36717]: Unable to set osd_memory_target on smithi086 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-04T12:36:10.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:10 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi063 to 174.1M 2024-03-04T12:36:10.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:10 smithi063 ceph-mon[36717]: Unable to set osd_memory_target on smithi063 to 182642073: error parsing value: Value '182642073' is below minimum 939524096 2024-03-04T12:36:10.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:10 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi086 to 174.1M 2024-03-04T12:36:10.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:10 smithi060 ceph-mon[32745]: Unable to set osd_memory_target on smithi086 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-04T12:36:10.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:10 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi063 to 174.1M 2024-03-04T12:36:10.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:10 smithi060 ceph-mon[32745]: Unable to set osd_memory_target on smithi063 to 182642073: error parsing value: Value '182642073' is below minimum 939524096 2024-03-04T12:36:11.304 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:11 smithi060 ceph-mon[32745]: pgmap v321: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:36:11.556 INFO:teuthology.orchestra.run.smithi060.stdout:{ 2024-03-04T12:36:11.556 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-0": { 2024-03-04T12:36:11.556 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 230, 2024-03-04T12:36:11.556 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 218, 2024-03-04T12:36:11.556 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 1707920, 2024-03-04T12:36:11.556 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 130560, 2024-03-04T12:36:11.557 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 4, 2024-03-04T12:36:11.557 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 0, 2024-03-04T12:36:11.557 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.046473659, 2024-03-04T12:36:11.557 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.014889611, 2024-03-04T12:36:11.557 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.036838242, 2024-03-04T12:36:11.557 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.004532413, 2024-03-04T12:36:11.557 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:11.557 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 217, 2024-03-04T12:36:11.558 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.011305474, 2024-03-04T12:36:11.558 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000052098 2024-03-04T12:36:11.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:11.558 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:11.558 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:11.558 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:11.558 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:11.558 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:11.558 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 1707920, 2024-03-04T12:36:11.559 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 130560 2024-03-04T12:36:11.559 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:11.559 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-1": { 2024-03-04T12:36:11.559 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 2, 2024-03-04T12:36:11.559 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 167, 2024-03-04T12:36:11.559 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 320, 2024-03-04T12:36:11.559 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 184544, 2024-03-04T12:36:11.559 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 5, 2024-03-04T12:36:11.559 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 3, 2024-03-04T12:36:11.560 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.015729473, 2024-03-04T12:36:11.560 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.011123390, 2024-03-04T12:36:11.560 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.001880510, 2024-03-04T12:36:11.560 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-04T12:36:11.560 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:11.560 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 164, 2024-03-04T12:36:11.560 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.012018494, 2024-03-04T12:36:11.560 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000073283 2024-03-04T12:36:11.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:11.561 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:11.561 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:11.561 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:11.561 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:11.561 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:11.561 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-04T12:36:11.561 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 184544 2024-03-04T12:36:11.562 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:11.562 INFO:teuthology.orchestra.run.smithi060.stdout: "AsyncMessenger::Worker-2": { 2024-03-04T12:36:11.562 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_messages": 4, 2024-03-04T12:36:11.562 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages": 4, 2024-03-04T12:36:11.562 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_bytes": 3024, 2024-03-04T12:36:11.562 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_bytes": 448, 2024-03-04T12:36:11.562 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_created_connections": 2, 2024-03-04T12:36:11.562 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_active_connections": 1, 2024-03-04T12:36:11.562 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_total_time": 0.000734277, 2024-03-04T12:36:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_send_time": 0.000036487, 2024-03-04T12:36:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_recv_time": 0.000678432, 2024-03-04T12:36:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-04T12:36:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 3, 2024-03-04T12:36:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000751671, 2024-03-04T12:36:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000250557 2024-03-04T12:36:11.563 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-04T12:36:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: "msgr_send_encrypted_bytes": 448 2024-03-04T12:36:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:11.564 INFO:teuthology.orchestra.run.smithi060.stdout: "cct": { 2024-03-04T12:36:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: "total_workers": 0, 2024-03-04T12:36:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: "unhealthy_workers": 0 2024-03-04T12:36:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: "mempool": { 2024-03-04T12:36:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_bytes": 0, 2024-03-04T12:36:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: "bloom_filter_items": 0, 2024-03-04T12:36:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_bytes": 0, 2024-03-04T12:36:11.565 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_alloc_items": 0, 2024-03-04T12:36:11.566 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_bytes": 0, 2024-03-04T12:36:11.566 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_data_items": 0, 2024-03-04T12:36:11.566 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-04T12:36:11.566 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_onode_items": 0, 2024-03-04T12:36:11.566 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-04T12:36:11.566 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_meta_items": 0, 2024-03-04T12:36:11.566 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_bytes": 0, 2024-03-04T12:36:11.566 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_other_items": 0, 2024-03-04T12:36:11.567 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-04T12:36:11.567 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_cache_buffer_items": 0, 2024-03-04T12:36:11.567 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_bytes": 0, 2024-03-04T12:36:11.567 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_extent_items": 0, 2024-03-04T12:36:11.567 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_bytes": 0, 2024-03-04T12:36:11.567 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_blob_items": 0, 2024-03-04T12:36:11.568 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-04T12:36:11.568 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_shared_blob_items": 0, 2024-03-04T12:36:11.568 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-04T12:36:11.568 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_inline_bl_items": 0, 2024-03-04T12:36:11.568 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_bytes": 0, 2024-03-04T12:36:11.568 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_fsck_items": 0, 2024-03-04T12:36:11.568 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_bytes": 0, 2024-03-04T12:36:11.568 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_txc_items": 0, 2024-03-04T12:36:11.568 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-04T12:36:11.569 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_deferred_items": 0, 2024-03-04T12:36:11.569 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_bytes": 0, 2024-03-04T12:36:11.569 INFO:teuthology.orchestra.run.smithi060.stdout: "bluestore_writing_items": 0, 2024-03-04T12:36:11.569 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_bytes": 0, 2024-03-04T12:36:11.569 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_items": 0, 2024-03-04T12:36:11.569 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_bytes": 0, 2024-03-04T12:36:11.569 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_reader_items": 0, 2024-03-04T12:36:11.569 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_bytes": 0, 2024-03-04T12:36:11.570 INFO:teuthology.orchestra.run.smithi060.stdout: "bluefs_file_writer_items": 0, 2024-03-04T12:36:11.570 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_bytes": 148714, 2024-03-04T12:36:11.570 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_anon_items": 30, 2024-03-04T12:36:11.570 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_bytes": 0, 2024-03-04T12:36:11.570 INFO:teuthology.orchestra.run.smithi060.stdout: "buffer_meta_items": 0, 2024-03-04T12:36:11.570 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_bytes": 0, 2024-03-04T12:36:11.570 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_items": 0, 2024-03-04T12:36:11.570 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_bytes": 0, 2024-03-04T12:36:11.570 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_mapbl_items": 0, 2024-03-04T12:36:11.571 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_bytes": 0, 2024-03-04T12:36:11.571 INFO:teuthology.orchestra.run.smithi060.stdout: "osd_pglog_items": 0, 2024-03-04T12:36:11.571 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_bytes": 0, 2024-03-04T12:36:11.571 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_items": 0, 2024-03-04T12:36:11.571 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_bytes": 0, 2024-03-04T12:36:11.571 INFO:teuthology.orchestra.run.smithi060.stdout: "osdmap_mapping_items": 0, 2024-03-04T12:36:11.571 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_bytes": 0, 2024-03-04T12:36:11.571 INFO:teuthology.orchestra.run.smithi060.stdout: "pgmap_items": 0, 2024-03-04T12:36:11.572 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_bytes": 928, 2024-03-04T12:36:11.572 INFO:teuthology.orchestra.run.smithi060.stdout: "mds_co_items": 2, 2024-03-04T12:36:11.572 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_bytes": 0, 2024-03-04T12:36:11.572 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_1_items": 0, 2024-03-04T12:36:11.572 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_bytes": 0, 2024-03-04T12:36:11.572 INFO:teuthology.orchestra.run.smithi060.stdout: "unittest_2_items": 0 2024-03-04T12:36:11.572 INFO:teuthology.orchestra.run.smithi060.stdout: }, 2024-03-04T12:36:11.572 INFO:teuthology.orchestra.run.smithi060.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-04T12:36:11.573 INFO:teuthology.orchestra.run.smithi060.stdout: "val": 0, 2024-03-04T12:36:11.573 INFO:teuthology.orchestra.run.smithi060.stdout: "max": 104857600, 2024-03-04T12:36:11.573 INFO:teuthology.orchestra.run.smithi060.stdout: "get_started": 0, 2024-03-04T12:36:11.573 INFO:teuthology.orchestra.run.smithi060.stdout: "get": 232, 2024-03-04T12:36:11.573 INFO:teuthology.orchestra.run.smithi060.stdout: "get_sum": 1675449, 2024-03-04T12:36:11.573 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:11.573 INFO:teuthology.orchestra.run.smithi060.stdout: "get_or_fail_success": 232, 2024-03-04T12:36:11.573 INFO:teuthology.orchestra.run.smithi060.stdout: "take": 0, 2024-03-04T12:36:11.573 INFO:teuthology.orchestra.run.smithi060.stdout: "take_sum": 0, 2024-03-04T12:36:11.574 INFO:teuthology.orchestra.run.smithi060.stdout: "put": 232, 2024-03-04T12:36:11.574 INFO:teuthology.orchestra.run.smithi060.stdout: "put_sum": 1675449, 2024-03-04T12:36:11.574 INFO:teuthology.orchestra.run.smithi060.stdout: "wait": { 2024-03-04T12:36:11.574 INFO:teuthology.orchestra.run.smithi060.stdout: "avgcount": 0, 2024-03-04T12:36:11.574 INFO:teuthology.orchestra.run.smithi060.stdout: "sum": 0.000000000, 2024-03-04T12:36:11.574 INFO:teuthology.orchestra.run.smithi060.stdout: "avgtime": 0.000000000 2024-03-04T12:36:11.574 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:11.574 INFO:teuthology.orchestra.run.smithi060.stdout: } 2024-03-04T12:36:11.574 INFO:teuthology.orchestra.run.smithi060.stdout:} 2024-03-04T12:36:11.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:11 smithi086 ceph-mon[36851]: pgmap v321: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:36:11.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:11 smithi063 ceph-mon[36717]: pgmap v321: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:36:12.272 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-04T12:36:12.272 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-04T12:36:12.272 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-04T12:36:12.272 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-04T12:36:12.272 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-04T12:36:12.273 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-04T12:36:12.273 DEBUG:tasks.check_counter:Getting stats from b 2024-03-04T12:36:12.273 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-04T12:36:12.536 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T12:36:12.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:12 smithi086 ceph-mon[36851]: pgmap v322: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:36:12.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:12 smithi063 ceph-mon[36717]: pgmap v322: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:36:12.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:12 smithi060 ceph-mon[32745]: pgmap v322: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:36:13.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:13 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:13.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:13 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:13.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:13 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:13.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:13 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:13.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:13 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:13.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:13 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:13.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:13 smithi060 ceph-mon[32745]: Adjusting osd_memory_target on smithi060 to 177.2M 2024-03-04T12:36:13.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:13 smithi060 ceph-mon[32745]: Unable to set osd_memory_target on smithi060 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-03-04T12:36:13.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:13 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:36:13.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:13 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:36:13.679 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:13 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:13.692 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:13 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:13.692 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:13 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:13.692 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:13 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:13.692 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:13 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:13.692 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:13 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:13.692 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:13 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:13.692 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:13 smithi063 ceph-mon[36717]: Adjusting osd_memory_target on smithi060 to 177.2M 2024-03-04T12:36:13.693 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:13 smithi063 ceph-mon[36717]: Unable to set osd_memory_target on smithi060 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-03-04T12:36:13.693 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:13 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:36:13.693 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:13 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:36:13.693 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:13 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:13.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:13 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:13.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:13 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:13.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:13 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:13.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:13 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:13.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:13 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:13.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:13 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-04T12:36:13.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:13 smithi086 ceph-mon[36851]: Adjusting osd_memory_target on smithi060 to 177.2M 2024-03-04T12:36:13.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:13 smithi086 ceph-mon[36851]: Unable to set osd_memory_target on smithi060 to 185896345: error parsing value: Value '185896345' is below minimum 939524096 2024-03-04T12:36:13.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:13 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T12:36:13.849 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:13 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T12:36:13.850 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:13 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' 2024-03-04T12:36:14.525 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-04T12:36:14.525 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-03-04T12:36:14.525 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 3502, 2024-03-04T12:36:14.526 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 3385, 2024-03-04T12:36:14.526 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1257709, 2024-03-04T12:36:14.526 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 6946762, 2024-03-04T12:36:14.526 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 15, 2024-03-04T12:36:14.526 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 6, 2024-03-04T12:36:14.526 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.303167565, 2024-03-04T12:36:14.526 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.121436463, 2024-03-04T12:36:14.526 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.128332363, 2024-03-04T12:36:14.526 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.055430531, 2024-03-04T12:36:14.527 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:14.527 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3383, 2024-03-04T12:36:14.527 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 1.207011347, 2024-03-04T12:36:14.527 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000356787 2024-03-04T12:36:14.527 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.527 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:14.527 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 4683, 2024-03-04T12:36:14.527 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000787200, 2024-03-04T12:36:14.527 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000168 2024-03-04T12:36:14.528 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.528 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-04T12:36:14.528 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 311312 2024-03-04T12:36:14.528 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.528 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-03-04T12:36:14.528 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 4551, 2024-03-04T12:36:14.528 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 6836, 2024-03-04T12:36:14.528 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1728796, 2024-03-04T12:36:14.529 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 3896027, 2024-03-04T12:36:14.529 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 19, 2024-03-04T12:36:14.529 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 5, 2024-03-04T12:36:14.529 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.468670058, 2024-03-04T12:36:14.529 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.153756493, 2024-03-04T12:36:14.529 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.423310690, 2024-03-04T12:36:14.529 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.110183278, 2024-03-04T12:36:14.529 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:14.530 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 6833, 2024-03-04T12:36:14.530 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.992931889, 2024-03-04T12:36:14.530 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000145314 2024-03-04T12:36:14.530 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.530 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:14.530 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 9467, 2024-03-04T12:36:14.530 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.001833463, 2024-03-04T12:36:14.530 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000193 2024-03-04T12:36:14.530 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.531 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 1521392, 2024-03-04T12:36:14.531 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 3266496 2024-03-04T12:36:14.531 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.531 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-03-04T12:36:14.531 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 2633, 2024-03-04T12:36:14.531 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 2838, 2024-03-04T12:36:14.531 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 2717139, 2024-03-04T12:36:14.531 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 1972091, 2024-03-04T12:36:14.532 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 59, 2024-03-04T12:36:14.532 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 7, 2024-03-04T12:36:14.532 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.290908829, 2024-03-04T12:36:14.532 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.085465275, 2024-03-04T12:36:14.532 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.163585272, 2024-03-04T12:36:14.532 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.060015233, 2024-03-04T12:36:14.532 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:14.532 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2838, 2024-03-04T12:36:14.532 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.462045480, 2024-03-04T12:36:14.533 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000162806 2024-03-04T12:36:14.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.533 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:14.533 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3793, 2024-03-04T12:36:14.533 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000733648, 2024-03-04T12:36:14.533 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000193 2024-03-04T12:36:14.533 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.533 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 1909104, 2024-03-04T12:36:14.534 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 148272 2024-03-04T12:36:14.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.534 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-03-04T12:36:14.534 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 1, 2024-03-04T12:36:14.534 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-03-04T12:36:14.534 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.534 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-MDSRank": { 2024-03-04T12:36:14.534 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-04T12:36:14.535 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-04T12:36:14.535 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1118, 2024-03-04T12:36:14.535 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.930060108, 2024-03-04T12:36:14.535 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000831896 2024-03-04T12:36:14.535 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:14.535 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.535 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-PurgeQueue": { 2024-03-04T12:36:14.535 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-04T12:36:14.535 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-04T12:36:14.536 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 45, 2024-03-04T12:36:14.536 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.003329741, 2024-03-04T12:36:14.536 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000073994 2024-03-04T12:36:14.536 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:14.536 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.536 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-04T12:36:14.536 INFO:teuthology.orchestra.run.smithi063.stdout: "request": 2076, 2024-03-04T12:36:14.536 INFO:teuthology.orchestra.run.smithi063.stdout: "reply": 1952, 2024-03-04T12:36:14.537 INFO:teuthology.orchestra.run.smithi063.stdout: "reply_latency": { 2024-03-04T12:36:14.537 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1952, 2024-03-04T12:36:14.537 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 62.290774249, 2024-03-04T12:36:14.537 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.031911257 2024-03-04T12:36:14.537 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.537 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_reply": 0, 2024-03-04T12:36:14.537 INFO:teuthology.orchestra.run.smithi063.stdout: "forward": 124, 2024-03-04T12:36:14.537 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_complete": 5, 2024-03-04T12:36:14.537 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_keys": 2, 2024-03-04T12:36:14.538 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_commit": 126, 2024-03-04T12:36:14.538 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_split": 3, 2024-03-04T12:36:14.538 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_merge": 0, 2024-03-04T12:36:14.538 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes": 64, 2024-03-04T12:36:14.538 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_top": 21, 2024-03-04T12:36:14.538 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_bottom": 17, 2024-03-04T12:36:14.538 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pin_tail": 26, 2024-03-04T12:36:14.539 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pinned": 36, 2024-03-04T12:36:14.539 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_expired": 1457, 2024-03-04T12:36:14.539 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_with_caps": 10, 2024-03-04T12:36:14.539 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": 10, 2024-03-04T12:36:14.539 INFO:teuthology.orchestra.run.smithi063.stdout: "subtrees": 7, 2024-03-04T12:36:14.539 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse": 2432, 2024-03-04T12:36:14.539 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_hit": 1970, 2024-03-04T12:36:14.540 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_forward": 124, 2024-03-04T12:36:14.540 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_discover": 0, 2024-03-04T12:36:14.540 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_dir_fetch": 0, 2024-03-04T12:36:14.540 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_remote_ino": 0, 2024-03-04T12:36:14.540 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_lock": 6, 2024-03-04T12:36:14.541 INFO:teuthology.orchestra.run.smithi063.stdout: "load_cent": 0, 2024-03-04T12:36:14.541 INFO:teuthology.orchestra.run.smithi063.stdout: "q": 0, 2024-03-04T12:36:14.541 INFO:teuthology.orchestra.run.smithi063.stdout: "exported": 11, 2024-03-04T12:36:14.541 INFO:teuthology.orchestra.run.smithi063.stdout: "exported_inodes": 1473, 2024-03-04T12:36:14.541 INFO:teuthology.orchestra.run.smithi063.stdout: "imported": 0, 2024-03-04T12:36:14.541 INFO:teuthology.orchestra.run.smithi063.stdout: "imported_inodes": 0, 2024-03-04T12:36:14.541 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_dir_fetch": 0, 2024-03-04T12:36:14.541 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_backtrace_fetch": 203, 2024-03-04T12:36:14.542 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_peer_discover": 0, 2024-03-04T12:36:14.542 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rfiles": 3452, 2024-03-04T12:36:14.542 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rbytes": 26508703, 2024-03-04T12:36:14.542 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rsnaps": 6, 2024-03-04T12:36:14.542 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_fetch": 203, 2024-03-04T12:36:14.542 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_set_tag": 0, 2024-03-04T12:36:14.542 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_repaired": 0, 2024-03-04T12:36:14.542 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_inotable_repaired": 0, 2024-03-04T12:36:14.542 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_inodes": 33, 2024-03-04T12:36:14.543 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_base_inodes": 4, 2024-03-04T12:36:14.543 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dirfrag_rstats": 29, 2024-03-04T12:36:14.543 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_file_inodes": 170, 2024-03-04T12:36:14.543 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_inode_file_caps": 2, 2024-03-04T12:36:14.543 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_revoke": 269, 2024-03-04T12:36:14.543 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_grant": 1216, 2024-03-04T12:36:14.543 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_trunc": 0, 2024-03-04T12:36:14.543 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-04T12:36:14.543 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-04T12:36:14.544 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps": 306, 2024-03-04T12:36:14.544 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps_dirty": 30, 2024-03-04T12:36:14.544 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_cap_release": 15, 2024-03-04T12:36:14.544 INFO:teuthology.orchestra.run.smithi063.stdout: "process_request_cap_release": 91 2024-03-04T12:36:14.544 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.544 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache": { 2024-03-04T12:36:14.544 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update": 0, 2024-03-04T12:36:14.544 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update_receipt": 0, 2024-03-04T12:36:14.545 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_try_discover": 0, 2024-03-04T12:36:14.545 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_send_discover": 0, 2024-03-04T12:36:14.545 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_handle_discover": 31, 2024-03-04T12:36:14.545 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays": 0, 2024-03-04T12:36:14.545 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_delayed": 0, 2024-03-04T12:36:14.545 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_enqueuing": 0, 2024-03-04T12:36:14.545 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_created": 21, 2024-03-04T12:36:14.545 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_enqueued": 19, 2024-03-04T12:36:14.545 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_reintegrated": 2, 2024-03-04T12:36:14.546 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_migrated": 0, 2024-03-04T12:36:14.546 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_processing": 0, 2024-03-04T12:36:14.546 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_enqueued": 0, 2024-03-04T12:36:14.546 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_prioritized": 0, 2024-03-04T12:36:14.546 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_started": 0, 2024-03-04T12:36:14.546 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_completed": 0, 2024-03-04T12:36:14.546 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_enqueue_scrub": 4, 2024-03-04T12:36:14.546 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_exportdir": 11, 2024-03-04T12:36:14.547 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_flush": 0, 2024-03-04T12:36:14.547 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragmentdir": 3, 2024-03-04T12:36:14.547 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragstats": 0, 2024-03-04T12:36:14.547 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_inodestats": 0 2024-03-04T12:36:14.547 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.547 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log": { 2024-03-04T12:36:14.547 INFO:teuthology.orchestra.run.smithi063.stdout: "evlrg": 0, 2024-03-04T12:36:14.547 INFO:teuthology.orchestra.run.smithi063.stdout: "evadd": 1698, 2024-03-04T12:36:14.547 INFO:teuthology.orchestra.run.smithi063.stdout: "evex": 1620, 2024-03-04T12:36:14.548 INFO:teuthology.orchestra.run.smithi063.stdout: "evtrm": 1620, 2024-03-04T12:36:14.548 INFO:teuthology.orchestra.run.smithi063.stdout: "ev": 78, 2024-03-04T12:36:14.548 INFO:teuthology.orchestra.run.smithi063.stdout: "evexg": 0, 2024-03-04T12:36:14.548 INFO:teuthology.orchestra.run.smithi063.stdout: "evexd": 193, 2024-03-04T12:36:14.548 INFO:teuthology.orchestra.run.smithi063.stdout: "segadd": 11, 2024-03-04T12:36:14.548 INFO:teuthology.orchestra.run.smithi063.stdout: "segex": 10, 2024-03-04T12:36:14.548 INFO:teuthology.orchestra.run.smithi063.stdout: "segtrm": 10, 2024-03-04T12:36:14.548 INFO:teuthology.orchestra.run.smithi063.stdout: "seg": 1, 2024-03-04T12:36:14.549 INFO:teuthology.orchestra.run.smithi063.stdout: "segmjr": 1, 2024-03-04T12:36:14.549 INFO:teuthology.orchestra.run.smithi063.stdout: "segexg": 0, 2024-03-04T12:36:14.549 INFO:teuthology.orchestra.run.smithi063.stdout: "segexd": 1, 2024-03-04T12:36:14.549 INFO:teuthology.orchestra.run.smithi063.stdout: "expos": 7945131, 2024-03-04T12:36:14.549 INFO:teuthology.orchestra.run.smithi063.stdout: "wrpos": 8144277, 2024-03-04T12:36:14.549 INFO:teuthology.orchestra.run.smithi063.stdout: "rdpos": 0, 2024-03-04T12:36:14.549 INFO:teuthology.orchestra.run.smithi063.stdout: "jlat": { 2024-03-04T12:36:14.549 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 388, 2024-03-04T12:36:14.550 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 1.246544499, 2024-03-04T12:36:14.550 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.003212743 2024-03-04T12:36:14.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.550 INFO:teuthology.orchestra.run.smithi063.stdout: "replayed": 0 2024-03-04T12:36:14.550 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.550 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_mem": { 2024-03-04T12:36:14.550 INFO:teuthology.orchestra.run.smithi063.stdout: "ino": 68, 2024-03-04T12:36:14.550 INFO:teuthology.orchestra.run.smithi063.stdout: "ino+": 1240, 2024-03-04T12:36:14.550 INFO:teuthology.orchestra.run.smithi063.stdout: "ino-": 1172, 2024-03-04T12:36:14.551 INFO:teuthology.orchestra.run.smithi063.stdout: "dir": 76, 2024-03-04T12:36:14.551 INFO:teuthology.orchestra.run.smithi063.stdout: "dir+": 91, 2024-03-04T12:36:14.551 INFO:teuthology.orchestra.run.smithi063.stdout: "dir-": 15, 2024-03-04T12:36:14.551 INFO:teuthology.orchestra.run.smithi063.stdout: "dn": 64, 2024-03-04T12:36:14.551 INFO:teuthology.orchestra.run.smithi063.stdout: "dn+": 1544, 2024-03-04T12:36:14.551 INFO:teuthology.orchestra.run.smithi063.stdout: "dn-": 1480, 2024-03-04T12:36:14.551 INFO:teuthology.orchestra.run.smithi063.stdout: "cap": 10, 2024-03-04T12:36:14.551 INFO:teuthology.orchestra.run.smithi063.stdout: "cap+": 1220, 2024-03-04T12:36:14.552 INFO:teuthology.orchestra.run.smithi063.stdout: "cap-": 1210, 2024-03-04T12:36:14.552 INFO:teuthology.orchestra.run.smithi063.stdout: "rss": 65676, 2024-03-04T12:36:14.552 INFO:teuthology.orchestra.run.smithi063.stdout: "heap": 215420 2024-03-04T12:36:14.552 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.552 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server": { 2024-03-04T12:36:14.552 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_client_request": 2367, 2024-03-04T12:36:14.552 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_server_request": 194, 2024-03-04T12:36:14.552 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_request": 2076, 2024-03-04T12:36:14.553 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_session": 50, 2024-03-04T12:36:14.553 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_peer_request": 400, 2024-03-04T12:36:14.553 INFO:teuthology.orchestra.run.smithi063.stdout: "req_create_latency": { 2024-03-04T12:36:14.553 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1160, 2024-03-04T12:36:14.553 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 54.410149559, 2024-03-04T12:36:14.553 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.046905301 2024-03-04T12:36:14.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.555 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getattr_latency": { 2024-03-04T12:36:14.555 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 315, 2024-03-04T12:36:14.555 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 1.324933971, 2024-03-04T12:36:14.555 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.004206139 2024-03-04T12:36:14.555 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.556 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getfilelock_latency": { 2024-03-04T12:36:14.556 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.556 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.556 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.556 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.557 INFO:teuthology.orchestra.run.smithi063.stdout: "req_link_latency": { 2024-03-04T12:36:14.557 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2, 2024-03-04T12:36:14.557 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.004534224, 2024-03-04T12:36:14.557 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.002267112 2024-03-04T12:36:14.557 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.557 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookup_latency": { 2024-03-04T12:36:14.557 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 59, 2024-03-04T12:36:14.557 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 4.830640396, 2024-03-04T12:36:14.558 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.081875260 2024-03-04T12:36:14.558 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.558 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookuphash_latency": { 2024-03-04T12:36:14.558 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.558 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.558 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.558 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.558 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupino_latency": { 2024-03-04T12:36:14.559 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1, 2024-03-04T12:36:14.559 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000203223, 2024-03-04T12:36:14.559 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000203223 2024-03-04T12:36:14.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.559 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupname_latency": { 2024-03-04T12:36:14.559 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.559 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.559 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.560 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupparent_latency": { 2024-03-04T12:36:14.560 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.560 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.560 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.560 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupsnap_latency": { 2024-03-04T12:36:14.560 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.561 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.561 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.561 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.561 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lssnap_latency": { 2024-03-04T12:36:14.561 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 11, 2024-03-04T12:36:14.561 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.001605373, 2024-03-04T12:36:14.562 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000145943 2024-03-04T12:36:14.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.563 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mkdir_latency": { 2024-03-04T12:36:14.563 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 49, 2024-03-04T12:36:14.563 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 1.395245531, 2024-03-04T12:36:14.563 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.028474398 2024-03-04T12:36:14.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.563 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mknod_latency": { 2024-03-04T12:36:14.563 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.563 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.564 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.564 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mksnap_latency": { 2024-03-04T12:36:14.564 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 11, 2024-03-04T12:36:14.564 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.068294705, 2024-03-04T12:36:14.564 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.006208609 2024-03-04T12:36:14.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.564 INFO:teuthology.orchestra.run.smithi063.stdout: "req_open_latency": { 2024-03-04T12:36:14.565 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 292, 2024-03-04T12:36:14.565 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.060969114, 2024-03-04T12:36:14.565 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000208798 2024-03-04T12:36:14.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.565 INFO:teuthology.orchestra.run.smithi063.stdout: "req_readdir_latency": { 2024-03-04T12:36:14.565 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 4, 2024-03-04T12:36:14.565 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.013523366, 2024-03-04T12:36:14.565 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.003380841 2024-03-04T12:36:14.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.566 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rename_latency": { 2024-03-04T12:36:14.566 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 19, 2024-03-04T12:36:14.566 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.092421958, 2024-03-04T12:36:14.566 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.004864313 2024-03-04T12:36:14.566 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.566 INFO:teuthology.orchestra.run.smithi063.stdout: "req_renamesnap_latency": { 2024-03-04T12:36:14.566 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.566 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.567 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.567 INFO:teuthology.orchestra.run.smithi063.stdout: "req_snapdiff_latency": { 2024-03-04T12:36:14.567 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.567 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.567 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.567 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.567 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmdir_latency": { 2024-03-04T12:36:14.567 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.568 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.568 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.568 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmsnap_latency": { 2024-03-04T12:36:14.568 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 5, 2024-03-04T12:36:14.568 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.024627982, 2024-03-04T12:36:14.568 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.004925596 2024-03-04T12:36:14.568 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.569 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmxattr_latency": { 2024-03-04T12:36:14.569 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.569 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.569 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.569 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setattr_latency": { 2024-03-04T12:36:14.569 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2, 2024-03-04T12:36:14.569 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.001752758, 2024-03-04T12:36:14.570 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000876379 2024-03-04T12:36:14.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.570 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setdirlayout_latency": { 2024-03-04T12:36:14.570 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.570 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.570 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.570 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.570 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setfilelock_latency": { 2024-03-04T12:36:14.570 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.571 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.571 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.571 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setlayout_latency": { 2024-03-04T12:36:14.571 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.571 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.571 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.571 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.572 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setxattr_latency": { 2024-03-04T12:36:14.572 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 9, 2024-03-04T12:36:14.572 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.023132703, 2024-03-04T12:36:14.572 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.002570300 2024-03-04T12:36:14.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.572 INFO:teuthology.orchestra.run.smithi063.stdout: "req_symlink_latency": { 2024-03-04T12:36:14.572 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1, 2024-03-04T12:36:14.572 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000724991, 2024-03-04T12:36:14.573 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000724991 2024-03-04T12:36:14.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.573 INFO:teuthology.orchestra.run.smithi063.stdout: "req_unlink_latency": { 2024-03-04T12:36:14.573 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 12, 2024-03-04T12:36:14.573 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.038014395, 2024-03-04T12:36:14.573 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.003167866 2024-03-04T12:36:14.573 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.573 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_revoke_eviction": 0, 2024-03-04T12:36:14.574 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_acquisition_throttle": 0, 2024-03-04T12:36:14.574 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getvxattr_latency": { 2024-03-04T12:36:14.574 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.574 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.574 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.574 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:14.574 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.574 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sessions": { 2024-03-04T12:36:14.575 INFO:teuthology.orchestra.run.smithi063.stdout: "session_count": 2, 2024-03-04T12:36:14.575 INFO:teuthology.orchestra.run.smithi063.stdout: "session_add": 7, 2024-03-04T12:36:14.575 INFO:teuthology.orchestra.run.smithi063.stdout: "session_remove": 5, 2024-03-04T12:36:14.575 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_open": 2, 2024-03-04T12:36:14.575 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_stale": 0, 2024-03-04T12:36:14.575 INFO:teuthology.orchestra.run.smithi063.stdout: "total_load": 32, 2024-03-04T12:36:14.575 INFO:teuthology.orchestra.run.smithi063.stdout: "average_load": 16, 2024-03-04T12:36:14.575 INFO:teuthology.orchestra.run.smithi063.stdout: "avg_session_uptime": 622, 2024-03-04T12:36:14.576 INFO:teuthology.orchestra.run.smithi063.stdout: "mdthresh_evicted": 0 2024-03-04T12:36:14.576 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.576 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-03-04T12:36:14.576 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-03-04T12:36:14.576 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-03-04T12:36:14.576 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-03-04T12:36:14.576 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-03-04T12:36:14.577 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-03-04T12:36:14.577 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-03-04T12:36:14.577 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-04T12:36:14.577 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-03-04T12:36:14.577 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-04T12:36:14.577 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-03-04T12:36:14.577 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-03-04T12:36:14.577 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-03-04T12:36:14.578 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-04T12:36:14.578 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-03-04T12:36:14.578 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-03-04T12:36:14.578 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-03-04T12:36:14.578 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-03-04T12:36:14.578 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-03-04T12:36:14.578 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-04T12:36:14.578 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-03-04T12:36:14.579 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-04T12:36:14.579 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-03-04T12:36:14.579 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-03-04T12:36:14.579 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-03-04T12:36:14.579 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-03-04T12:36:14.579 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-03-04T12:36:14.579 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-04T12:36:14.580 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-03-04T12:36:14.580 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-03-04T12:36:14.580 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-03-04T12:36:14.580 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-03-04T12:36:14.580 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-03-04T12:36:14.580 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-03-04T12:36:14.580 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-03-04T12:36:14.580 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-03-04T12:36:14.581 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-03-04T12:36:14.581 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 349577, 2024-03-04T12:36:14.581 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 197, 2024-03-04T12:36:14.581 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 1672, 2024-03-04T12:36:14.581 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 19, 2024-03-04T12:36:14.581 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-03-04T12:36:14.581 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-03-04T12:36:14.582 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-03-04T12:36:14.582 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-03-04T12:36:14.582 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-03-04T12:36:14.582 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-03-04T12:36:14.582 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 9976, 2024-03-04T12:36:14.582 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 209, 2024-03-04T12:36:14.582 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-03-04T12:36:14.583 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-03-04T12:36:14.583 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-03-04T12:36:14.583 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-03-04T12:36:14.583 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 337213, 2024-03-04T12:36:14.583 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 3224, 2024-03-04T12:36:14.583 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-03-04T12:36:14.583 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-03-04T12:36:14.583 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-03-04T12:36:14.584 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-03-04T12:36:14.584 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.584 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter": { 2024-03-04T12:36:14.584 INFO:teuthology.orchestra.run.smithi063.stdout: "op_active": 0, 2024-03-04T12:36:14.584 INFO:teuthology.orchestra.run.smithi063.stdout: "op_laggy": 0, 2024-03-04T12:36:14.584 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send": 2197, 2024-03-04T12:36:14.584 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send_bytes": 5208661, 2024-03-04T12:36:14.584 INFO:teuthology.orchestra.run.smithi063.stdout: "op_resend": 37, 2024-03-04T12:36:14.585 INFO:teuthology.orchestra.run.smithi063.stdout: "op_reply": 2197, 2024-03-04T12:36:14.585 INFO:teuthology.orchestra.run.smithi063.stdout: "op_latency": { 2024-03-04T12:36:14.585 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2197, 2024-03-04T12:36:14.585 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 59.698315093, 2024-03-04T12:36:14.585 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.027172651 2024-03-04T12:36:14.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.585 INFO:teuthology.orchestra.run.smithi063.stdout: "op_inflight": 0, 2024-03-04T12:36:14.585 INFO:teuthology.orchestra.run.smithi063.stdout: "oplen_avg": { 2024-03-04T12:36:14.586 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2197, 2024-03-04T12:36:14.586 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 3638 2024-03-04T12:36:14.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.586 INFO:teuthology.orchestra.run.smithi063.stdout: "op": 2197, 2024-03-04T12:36:14.586 INFO:teuthology.orchestra.run.smithi063.stdout: "op_r": 221, 2024-03-04T12:36:14.586 INFO:teuthology.orchestra.run.smithi063.stdout: "op_w": 1976, 2024-03-04T12:36:14.586 INFO:teuthology.orchestra.run.smithi063.stdout: "op_rmw": 0, 2024-03-04T12:36:14.586 INFO:teuthology.orchestra.run.smithi063.stdout: "op_pg": 0, 2024-03-04T12:36:14.587 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_stat": 45, 2024-03-04T12:36:14.587 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_create": 802, 2024-03-04T12:36:14.587 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_read": 7, 2024-03-04T12:36:14.587 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_write": 391, 2024-03-04T12:36:14.587 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writefull": 62, 2024-03-04T12:36:14.587 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writesame": 0, 2024-03-04T12:36:14.587 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_append": 0, 2024-03-04T12:36:14.587 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_zero": 0, 2024-03-04T12:36:14.587 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_truncate": 3, 2024-03-04T12:36:14.588 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_delete": 549, 2024-03-04T12:36:14.588 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_mapext": 0, 2024-03-04T12:36:14.588 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_sparse_read": 0, 2024-03-04T12:36:14.588 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_clonerange": 0, 2024-03-04T12:36:14.588 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_getxattr": 214, 2024-03-04T12:36:14.588 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_setxattr": 1233, 2024-03-04T12:36:14.588 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_cmpxattr": 0, 2024-03-04T12:36:14.588 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_rmxattr": 0, 2024-03-04T12:36:14.589 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_resetxattrs": 0, 2024-03-04T12:36:14.589 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_call": 0, 2024-03-04T12:36:14.589 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_watch": 0, 2024-03-04T12:36:14.589 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_notify": 0, 2024-03-04T12:36:14.589 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_src_cmpxattr": 0, 2024-03-04T12:36:14.589 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls": 0, 2024-03-04T12:36:14.589 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls_filter": 0, 2024-03-04T12:36:14.589 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_other": 0, 2024-03-04T12:36:14.589 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_active": 0, 2024-03-04T12:36:14.590 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_send": 0, 2024-03-04T12:36:14.590 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_resend": 0, 2024-03-04T12:36:14.590 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_ping": 0, 2024-03-04T12:36:14.590 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_active": 0, 2024-03-04T12:36:14.590 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_send": 0, 2024-03-04T12:36:14.590 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_resend": 0, 2024-03-04T12:36:14.590 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_active": 0, 2024-03-04T12:36:14.590 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_send": 0, 2024-03-04T12:36:14.590 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_resend": 0, 2024-03-04T12:36:14.591 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_active": 0, 2024-03-04T12:36:14.591 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_send": 0, 2024-03-04T12:36:14.591 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_resend": 0, 2024-03-04T12:36:14.591 INFO:teuthology.orchestra.run.smithi063.stdout: "command_active": 0, 2024-03-04T12:36:14.591 INFO:teuthology.orchestra.run.smithi063.stdout: "command_send": 0, 2024-03-04T12:36:14.591 INFO:teuthology.orchestra.run.smithi063.stdout: "command_resend": 0, 2024-03-04T12:36:14.591 INFO:teuthology.orchestra.run.smithi063.stdout: "map_epoch": 104, 2024-03-04T12:36:14.592 INFO:teuthology.orchestra.run.smithi063.stdout: "map_full": 0, 2024-03-04T12:36:14.592 INFO:teuthology.orchestra.run.smithi063.stdout: "map_inc": 20, 2024-03-04T12:36:14.592 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_sessions": 12, 2024-03-04T12:36:14.592 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_open": 12, 2024-03-04T12:36:14.592 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_close": 0, 2024-03-04T12:36:14.592 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_laggy": 0, 2024-03-04T12:36:14.592 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_wr": 295, 2024-03-04T12:36:14.592 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_rd": 14, 2024-03-04T12:36:14.593 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_del": 23 2024-03-04T12:36:14.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.593 INFO:teuthology.orchestra.run.smithi063.stdout: "oft": { 2024-03-04T12:36:14.593 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_objs": 1, 2024-03-04T12:36:14.593 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_kv_pairs": 0, 2024-03-04T12:36:14.593 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_updates": 0, 2024-03-04T12:36:14.593 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_removes": 0 2024-03-04T12:36:14.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.594 INFO:teuthology.orchestra.run.smithi063.stdout: "osd-slow-ops": { 2024-03-04T12:36:14.594 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_ops_count": 0 2024-03-04T12:36:14.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.594 INFO:teuthology.orchestra.run.smithi063.stdout: "purge_queue": { 2024-03-04T12:36:14.594 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops": 0, 2024-03-04T12:36:14.594 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops_high_water": 10, 2024-03-04T12:36:14.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing": 0, 2024-03-04T12:36:14.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_high_water": 5, 2024-03-04T12:36:14.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed_ops": 37, 2024-03-04T12:36:14.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed": 19, 2024-03-04T12:36:14.595 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_item_in_journal": 0 2024-03-04T12:36:14.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.595 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-04T12:36:14.595 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:14.596 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-04T12:36:14.596 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:14.596 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 10682, 2024-03-04T12:36:14.596 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 4529711, 2024-03-04T12:36:14.596 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:14.596 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 10682, 2024-03-04T12:36:14.597 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-04T12:36:14.597 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-04T12:36:14.597 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 10682, 2024-03-04T12:36:14.597 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 4529711, 2024-03-04T12:36:14.597 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:14.597 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.597 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.597 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.598 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:14.598 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.598 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_bytes": { 2024-03-04T12:36:14.598 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:14.598 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-04T12:36:14.598 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:14.598 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-04T12:36:14.599 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-04T12:36:14.599 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:14.599 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:14.599 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2197, 2024-03-04T12:36:14.599 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 5210636, 2024-03-04T12:36:14.599 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2197, 2024-03-04T12:36:14.599 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 5210636, 2024-03-04T12:36:14.599 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:14.600 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.600 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.600 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.600 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:14.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.600 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_ops": { 2024-03-04T12:36:14.600 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:14.600 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 1024, 2024-03-04T12:36:14.600 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:14.601 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-04T12:36:14.601 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-04T12:36:14.601 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:14.601 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:14.601 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2197, 2024-03-04T12:36:14.601 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 2197, 2024-03-04T12:36:14.601 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2197, 2024-03-04T12:36:14.601 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 2197, 2024-03-04T12:36:14.602 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:14.602 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.602 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.602 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.602 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:14.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.602 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle": { 2024-03-04T12:36:14.602 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:14.602 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-04T12:36:14.603 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:14.603 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 19, 2024-03-04T12:36:14.603 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 2063, 2024-03-04T12:36:14.603 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:14.603 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 19, 2024-03-04T12:36:14.603 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-04T12:36:14.603 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-04T12:36:14.603 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 3, 2024-03-04T12:36:14.604 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 2063, 2024-03-04T12:36:14.604 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:14.604 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.604 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.604 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.604 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:14.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:14.605 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle-0x560335030640": { 2024-03-04T12:36:14.605 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:14.606 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-04T12:36:14.606 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:14.606 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 1698, 2024-03-04T12:36:14.606 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 3949973, 2024-03-04T12:36:14.606 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:14.606 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 1698, 2024-03-04T12:36:14.606 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-04T12:36:14.606 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-04T12:36:14.607 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 388, 2024-03-04T12:36:14.607 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 3949973, 2024-03-04T12:36:14.607 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:14.607 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:14.607 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:14.607 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:14.607 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:14.607 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:14.608 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-04T12:36:15.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:15 smithi086 ceph-mon[36851]: pgmap v323: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:36:15.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:15 smithi063 ceph-mon[36717]: pgmap v323: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:36:15.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:15 smithi060 ceph-mon[32745]: pgmap v323: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-04T12:36:15.504 INFO:tasks.check_counter:Daemon mds.b mds.exported=11 2024-03-04T12:36:15.504 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-04T12:36:15.505 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-04T12:36:15.505 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-03-04T12:36:15.505 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-03-04T12:36:15.505 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=3 2024-03-04T12:36:15.505 DEBUG:tasks.check_counter:Getting stats from e 2024-03-04T12:36:15.505 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-04T12:36:15.762 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T12:36:16.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:16 smithi086 ceph-mon[36851]: osdmap e105: 12 total, 12 up, 12 in 2024-03-04T12:36:16.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:16 smithi063 ceph-mon[36717]: osdmap e105: 12 total, 12 up, 12 in 2024-03-04T12:36:16.367 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:16 smithi060 ceph-mon[32745]: osdmap e105: 12 total, 12 up, 12 in 2024-03-04T12:36:17.294 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:17 smithi063 ceph-mon[36717]: pgmap v325: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2024-03-04T12:36:17.335 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:17 smithi060 ceph-mon[32745]: pgmap v325: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2024-03-04T12:36:17.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:17 smithi086 ceph-mon[36851]: pgmap v325: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 249 B/s wr, 12 op/s 2024-03-04T12:36:17.551 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-04T12:36:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-03-04T12:36:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 771, 2024-03-04T12:36:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 742, 2024-03-04T12:36:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1960214, 2024-03-04T12:36:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 294168, 2024-03-04T12:36:17.551 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 6, 2024-03-04T12:36:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 2, 2024-03-04T12:36:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.110284651, 2024-03-04T12:36:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.029833950, 2024-03-04T12:36:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.058229859, 2024-03-04T12:36:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.027769374, 2024-03-04T12:36:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 741, 2024-03-04T12:36:17.552 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.032848043, 2024-03-04T12:36:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000044329 2024-03-04T12:36:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.553 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 1835888, 2024-03-04T12:36:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 134208 2024-03-04T12:36:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-03-04T12:36:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 701, 2024-03-04T12:36:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 865, 2024-03-04T12:36:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 7913972, 2024-03-04T12:36:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 664863, 2024-03-04T12:36:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 7, 2024-03-04T12:36:17.554 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 5, 2024-03-04T12:36:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.085594683, 2024-03-04T12:36:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.029320065, 2024-03-04T12:36:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.024514537, 2024-03-04T12:36:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.022221471, 2024-03-04T12:36:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 862, 2024-03-04T12:36:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.041401019, 2024-03-04T12:36:17.555 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000048029 2024-03-04T12:36:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-04T12:36:17.556 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 442576 2024-03-04T12:36:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-03-04T12:36:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 1051, 2024-03-04T12:36:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 1051, 2024-03-04T12:36:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 8721557, 2024-03-04T12:36:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 333452, 2024-03-04T12:36:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 5, 2024-03-04T12:36:17.557 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 4, 2024-03-04T12:36:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.126779027, 2024-03-04T12:36:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.039884163, 2024-03-04T12:36:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.035566853, 2024-03-04T12:36:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.039555580, 2024-03-04T12:36:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1050, 2024-03-04T12:36:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.055761270, 2024-03-04T12:36:17.558 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000053105 2024-03-04T12:36:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-04T12:36:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 448 2024-03-04T12:36:17.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-03-04T12:36:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 1, 2024-03-04T12:36:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-03-04T12:36:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-MDSRank": { 2024-03-04T12:36:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-04T12:36:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-04T12:36:17.560 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3754, 2024-03-04T12:36:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.129909080, 2024-03-04T12:36:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000034605 2024-03-04T12:36:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-PurgeQueue": { 2024-03-04T12:36:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-04T12:36:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-04T12:36:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.561 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.562 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.562 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:17.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.562 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-04T12:36:17.562 INFO:teuthology.orchestra.run.smithi063.stdout: "request": 0, 2024-03-04T12:36:17.562 INFO:teuthology.orchestra.run.smithi063.stdout: "reply": 0, 2024-03-04T12:36:17.562 INFO:teuthology.orchestra.run.smithi063.stdout: "reply_latency": { 2024-03-04T12:36:17.562 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_reply": 0, 2024-03-04T12:36:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "forward": 0, 2024-03-04T12:36:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_complete": 0, 2024-03-04T12:36:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_keys": 0, 2024-03-04T12:36:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_commit": 0, 2024-03-04T12:36:17.563 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_split": 0, 2024-03-04T12:36:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_merge": 0, 2024-03-04T12:36:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes": 1642, 2024-03-04T12:36:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_top": 208, 2024-03-04T12:36:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_bottom": 1434, 2024-03-04T12:36:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pin_tail": 0, 2024-03-04T12:36:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pinned": 1347, 2024-03-04T12:36:17.564 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_expired": 0, 2024-03-04T12:36:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_with_caps": 0, 2024-03-04T12:36:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": 0, 2024-03-04T12:36:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "subtrees": 4, 2024-03-04T12:36:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse": 0, 2024-03-04T12:36:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_hit": 0, 2024-03-04T12:36:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_forward": 0, 2024-03-04T12:36:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_discover": 0, 2024-03-04T12:36:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_dir_fetch": 0, 2024-03-04T12:36:17.565 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_remote_ino": 0, 2024-03-04T12:36:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_lock": 0, 2024-03-04T12:36:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "load_cent": 0, 2024-03-04T12:36:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "q": 0, 2024-03-04T12:36:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "exported": 0, 2024-03-04T12:36:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "exported_inodes": 0, 2024-03-04T12:36:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "imported": 0, 2024-03-04T12:36:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "imported_inodes": 0, 2024-03-04T12:36:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_dir_fetch": 0, 2024-03-04T12:36:17.566 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_backtrace_fetch": 0, 2024-03-04T12:36:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_peer_discover": 0, 2024-03-04T12:36:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rfiles": 0, 2024-03-04T12:36:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rbytes": 0, 2024-03-04T12:36:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rsnaps": 0, 2024-03-04T12:36:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_fetch": 0, 2024-03-04T12:36:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_set_tag": 0, 2024-03-04T12:36:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_repaired": 0, 2024-03-04T12:36:17.567 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_inotable_repaired": 0, 2024-03-04T12:36:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_inodes": 0, 2024-03-04T12:36:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_base_inodes": 0, 2024-03-04T12:36:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dirfrag_rstats": 0, 2024-03-04T12:36:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_file_inodes": 0, 2024-03-04T12:36:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_inode_file_caps": 0, 2024-03-04T12:36:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_revoke": 0, 2024-03-04T12:36:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_grant": 0, 2024-03-04T12:36:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_trunc": 0, 2024-03-04T12:36:17.568 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-04T12:36:17.569 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-04T12:36:17.569 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps": 0, 2024-03-04T12:36:17.569 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps_dirty": 0, 2024-03-04T12:36:17.569 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_cap_release": 0, 2024-03-04T12:36:17.569 INFO:teuthology.orchestra.run.smithi063.stdout: "process_request_cap_release": 0 2024-03-04T12:36:17.569 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.569 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache": { 2024-03-04T12:36:17.569 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update": 0, 2024-03-04T12:36:17.570 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update_receipt": 0, 2024-03-04T12:36:17.570 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_try_discover": 0, 2024-03-04T12:36:17.570 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_send_discover": 0, 2024-03-04T12:36:17.570 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_handle_discover": 0, 2024-03-04T12:36:17.570 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays": 0, 2024-03-04T12:36:17.570 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_delayed": 0, 2024-03-04T12:36:17.570 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_enqueuing": 0, 2024-03-04T12:36:17.570 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_created": 0, 2024-03-04T12:36:17.570 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_enqueued": 0, 2024-03-04T12:36:17.571 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_reintegrated": 0, 2024-03-04T12:36:17.571 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_migrated": 0, 2024-03-04T12:36:17.571 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_processing": 0, 2024-03-04T12:36:17.571 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_enqueued": 0, 2024-03-04T12:36:17.571 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_prioritized": 0, 2024-03-04T12:36:17.571 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_started": 0, 2024-03-04T12:36:17.571 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_completed": 0, 2024-03-04T12:36:17.571 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_enqueue_scrub": 0, 2024-03-04T12:36:17.571 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_exportdir": 0, 2024-03-04T12:36:17.572 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_flush": 0, 2024-03-04T12:36:17.572 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragmentdir": 0, 2024-03-04T12:36:17.572 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragstats": 0, 2024-03-04T12:36:17.572 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_inodestats": 0 2024-03-04T12:36:17.572 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.572 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log": { 2024-03-04T12:36:17.572 INFO:teuthology.orchestra.run.smithi063.stdout: "evlrg": 0, 2024-03-04T12:36:17.572 INFO:teuthology.orchestra.run.smithi063.stdout: "evadd": 0, 2024-03-04T12:36:17.572 INFO:teuthology.orchestra.run.smithi063.stdout: "evex": 0, 2024-03-04T12:36:17.573 INFO:teuthology.orchestra.run.smithi063.stdout: "evtrm": 0, 2024-03-04T12:36:17.573 INFO:teuthology.orchestra.run.smithi063.stdout: "ev": 7417, 2024-03-04T12:36:17.573 INFO:teuthology.orchestra.run.smithi063.stdout: "evexg": 0, 2024-03-04T12:36:17.573 INFO:teuthology.orchestra.run.smithi063.stdout: "evexd": 0, 2024-03-04T12:36:17.573 INFO:teuthology.orchestra.run.smithi063.stdout: "segadd": 0, 2024-03-04T12:36:17.573 INFO:teuthology.orchestra.run.smithi063.stdout: "segex": 0, 2024-03-04T12:36:17.573 INFO:teuthology.orchestra.run.smithi063.stdout: "segtrm": 0, 2024-03-04T12:36:17.573 INFO:teuthology.orchestra.run.smithi063.stdout: "seg": 10, 2024-03-04T12:36:17.573 INFO:teuthology.orchestra.run.smithi063.stdout: "segmjr": 2, 2024-03-04T12:36:17.574 INFO:teuthology.orchestra.run.smithi063.stdout: "segexg": 0, 2024-03-04T12:36:17.574 INFO:teuthology.orchestra.run.smithi063.stdout: "segexd": 0, 2024-03-04T12:36:17.574 INFO:teuthology.orchestra.run.smithi063.stdout: "expos": 4194304, 2024-03-04T12:36:17.574 INFO:teuthology.orchestra.run.smithi063.stdout: "wrpos": 20316236, 2024-03-04T12:36:17.574 INFO:teuthology.orchestra.run.smithi063.stdout: "rdpos": 20314018, 2024-03-04T12:36:17.574 INFO:teuthology.orchestra.run.smithi063.stdout: "jlat": { 2024-03-04T12:36:17.574 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.574 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.574 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.575 INFO:teuthology.orchestra.run.smithi063.stdout: "replayed": 7417 2024-03-04T12:36:17.575 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.575 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_mem": { 2024-03-04T12:36:17.575 INFO:teuthology.orchestra.run.smithi063.stdout: "ino": 632, 2024-03-04T12:36:17.575 INFO:teuthology.orchestra.run.smithi063.stdout: "ino+": 5019, 2024-03-04T12:36:17.575 INFO:teuthology.orchestra.run.smithi063.stdout: "ino-": 4387, 2024-03-04T12:36:17.575 INFO:teuthology.orchestra.run.smithi063.stdout: "dir": 75, 2024-03-04T12:36:17.575 INFO:teuthology.orchestra.run.smithi063.stdout: "dir+": 264, 2024-03-04T12:36:17.576 INFO:teuthology.orchestra.run.smithi063.stdout: "dir-": 189, 2024-03-04T12:36:17.576 INFO:teuthology.orchestra.run.smithi063.stdout: "dn": 1642, 2024-03-04T12:36:17.576 INFO:teuthology.orchestra.run.smithi063.stdout: "dn+": 5445, 2024-03-04T12:36:17.576 INFO:teuthology.orchestra.run.smithi063.stdout: "dn-": 3803, 2024-03-04T12:36:17.576 INFO:teuthology.orchestra.run.smithi063.stdout: "cap": 0, 2024-03-04T12:36:17.576 INFO:teuthology.orchestra.run.smithi063.stdout: "cap+": 0, 2024-03-04T12:36:17.576 INFO:teuthology.orchestra.run.smithi063.stdout: "cap-": 0, 2024-03-04T12:36:17.576 INFO:teuthology.orchestra.run.smithi063.stdout: "rss": 58336, 2024-03-04T12:36:17.577 INFO:teuthology.orchestra.run.smithi063.stdout: "heap": 215420 2024-03-04T12:36:17.577 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.577 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server": { 2024-03-04T12:36:17.577 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_client_request": 0, 2024-03-04T12:36:17.577 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_server_request": 0, 2024-03-04T12:36:17.577 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_request": 0, 2024-03-04T12:36:17.577 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_session": 0, 2024-03-04T12:36:17.577 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_peer_request": 0, 2024-03-04T12:36:17.577 INFO:teuthology.orchestra.run.smithi063.stdout: "req_create_latency": { 2024-03-04T12:36:17.578 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.578 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.578 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.578 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getattr_latency": { 2024-03-04T12:36:17.578 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.578 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.578 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.578 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.579 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getfilelock_latency": { 2024-03-04T12:36:17.579 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.579 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.579 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.579 INFO:teuthology.orchestra.run.smithi063.stdout: "req_link_latency": { 2024-03-04T12:36:17.579 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.579 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.579 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.579 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.580 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookup_latency": { 2024-03-04T12:36:17.580 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.580 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.580 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.580 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.580 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookuphash_latency": { 2024-03-04T12:36:17.580 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.580 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.581 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.581 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupino_latency": { 2024-03-04T12:36:17.581 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.581 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.581 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.581 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.581 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupname_latency": { 2024-03-04T12:36:17.581 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.582 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.582 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.582 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupparent_latency": { 2024-03-04T12:36:17.582 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.582 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.582 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.582 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.582 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupsnap_latency": { 2024-03-04T12:36:17.583 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.583 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.583 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.583 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lssnap_latency": { 2024-03-04T12:36:17.583 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.583 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.583 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.583 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.584 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mkdir_latency": { 2024-03-04T12:36:17.584 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.584 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.584 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.585 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mknod_latency": { 2024-03-04T12:36:17.585 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.585 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.585 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.585 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.585 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mksnap_latency": { 2024-03-04T12:36:17.585 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.586 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.586 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.586 INFO:teuthology.orchestra.run.smithi063.stdout: "req_open_latency": { 2024-03-04T12:36:17.586 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.586 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.586 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.586 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.587 INFO:teuthology.orchestra.run.smithi063.stdout: "req_readdir_latency": { 2024-03-04T12:36:17.587 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.587 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.587 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.587 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rename_latency": { 2024-03-04T12:36:17.587 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.587 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.587 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.588 INFO:teuthology.orchestra.run.smithi063.stdout: "req_renamesnap_latency": { 2024-03-04T12:36:17.588 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.588 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.588 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.588 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.588 INFO:teuthology.orchestra.run.smithi063.stdout: "req_snapdiff_latency": { 2024-03-04T12:36:17.588 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.589 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.589 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.589 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmdir_latency": { 2024-03-04T12:36:17.589 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.589 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.589 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.590 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmsnap_latency": { 2024-03-04T12:36:17.590 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.590 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.590 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.590 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmxattr_latency": { 2024-03-04T12:36:17.590 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.590 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.590 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.591 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setattr_latency": { 2024-03-04T12:36:17.591 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.591 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.591 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.591 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setdirlayout_latency": { 2024-03-04T12:36:17.591 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.591 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.592 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.592 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setfilelock_latency": { 2024-03-04T12:36:17.592 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.592 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.592 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.592 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setlayout_latency": { 2024-03-04T12:36:17.593 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.593 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.593 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.593 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setxattr_latency": { 2024-03-04T12:36:17.593 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.593 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.593 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "req_symlink_latency": { 2024-03-04T12:36:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "req_unlink_latency": { 2024-03-04T12:36:17.594 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_revoke_eviction": 0, 2024-03-04T12:36:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_acquisition_throttle": 0, 2024-03-04T12:36:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getvxattr_latency": { 2024-03-04T12:36:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.595 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sessions": { 2024-03-04T12:36:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "session_count": 1, 2024-03-04T12:36:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "session_add": 3, 2024-03-04T12:36:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "session_remove": 2, 2024-03-04T12:36:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_open": 1, 2024-03-04T12:36:17.596 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_stale": 0, 2024-03-04T12:36:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "total_load": 0, 2024-03-04T12:36:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "average_load": 0, 2024-03-04T12:36:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "avg_session_uptime": 2041, 2024-03-04T12:36:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "mdthresh_evicted": 0 2024-03-04T12:36:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-03-04T12:36:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-03-04T12:36:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-03-04T12:36:17.597 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-03-04T12:36:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-03-04T12:36:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-03-04T12:36:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-03-04T12:36:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-04T12:36:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-03-04T12:36:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-04T12:36:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-03-04T12:36:17.598 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-03-04T12:36:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-03-04T12:36:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-04T12:36:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-03-04T12:36:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-03-04T12:36:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-03-04T12:36:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-03-04T12:36:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-03-04T12:36:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-04T12:36:17.599 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-03-04T12:36:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-04T12:36:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-03-04T12:36:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-03-04T12:36:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-03-04T12:36:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-03-04T12:36:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-03-04T12:36:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-04T12:36:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-03-04T12:36:17.600 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-03-04T12:36:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-03-04T12:36:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-03-04T12:36:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-03-04T12:36:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-03-04T12:36:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-03-04T12:36:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-03-04T12:36:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-03-04T12:36:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 230959, 2024-03-04T12:36:17.601 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 727, 2024-03-04T12:36:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-03-04T12:36:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-03-04T12:36:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-03-04T12:36:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-03-04T12:36:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-03-04T12:36:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-03-04T12:36:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-03-04T12:36:17.602 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-03-04T12:36:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 9736, 2024-03-04T12:36:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 203, 2024-03-04T12:36:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-03-04T12:36:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-03-04T12:36:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-03-04T12:36:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-03-04T12:36:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 2730942, 2024-03-04T12:36:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 25565, 2024-03-04T12:36:17.603 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-03-04T12:36:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-03-04T12:36:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-03-04T12:36:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-03-04T12:36:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter": { 2024-03-04T12:36:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "op_active": 0, 2024-03-04T12:36:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "op_laggy": 0, 2024-03-04T12:36:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send": 2249, 2024-03-04T12:36:17.604 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send_bytes": 21, 2024-03-04T12:36:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "op_resend": 3, 2024-03-04T12:36:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "op_reply": 2249, 2024-03-04T12:36:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "op_latency": { 2024-03-04T12:36:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2249, 2024-03-04T12:36:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 1.300984995, 2024-03-04T12:36:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000578472 2024-03-04T12:36:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.605 INFO:teuthology.orchestra.run.smithi063.stdout: "op_inflight": 0, 2024-03-04T12:36:17.606 INFO:teuthology.orchestra.run.smithi063.stdout: "oplen_avg": { 2024-03-04T12:36:17.606 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2249, 2024-03-04T12:36:17.606 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 2250 2024-03-04T12:36:17.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.606 INFO:teuthology.orchestra.run.smithi063.stdout: "op": 2249, 2024-03-04T12:36:17.606 INFO:teuthology.orchestra.run.smithi063.stdout: "op_r": 2249, 2024-03-04T12:36:17.606 INFO:teuthology.orchestra.run.smithi063.stdout: "op_w": 0, 2024-03-04T12:36:17.606 INFO:teuthology.orchestra.run.smithi063.stdout: "op_rmw": 0, 2024-03-04T12:36:17.606 INFO:teuthology.orchestra.run.smithi063.stdout: "op_pg": 0, 2024-03-04T12:36:17.607 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_stat": 1437, 2024-03-04T12:36:17.607 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_create": 0, 2024-03-04T12:36:17.607 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_read": 810, 2024-03-04T12:36:17.607 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_write": 0, 2024-03-04T12:36:17.607 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writefull": 0, 2024-03-04T12:36:17.607 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writesame": 0, 2024-03-04T12:36:17.607 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_append": 0, 2024-03-04T12:36:17.607 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_zero": 0, 2024-03-04T12:36:17.607 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_truncate": 0, 2024-03-04T12:36:17.608 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_delete": 0, 2024-03-04T12:36:17.608 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_mapext": 0, 2024-03-04T12:36:17.608 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_sparse_read": 0, 2024-03-04T12:36:17.608 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_clonerange": 0, 2024-03-04T12:36:17.608 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_getxattr": 1, 2024-03-04T12:36:17.608 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_setxattr": 0, 2024-03-04T12:36:17.608 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_cmpxattr": 0, 2024-03-04T12:36:17.608 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_rmxattr": 0, 2024-03-04T12:36:17.608 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_resetxattrs": 0, 2024-03-04T12:36:17.609 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_call": 0, 2024-03-04T12:36:17.609 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_watch": 0, 2024-03-04T12:36:17.609 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_notify": 0, 2024-03-04T12:36:17.609 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_src_cmpxattr": 0, 2024-03-04T12:36:17.609 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls": 0, 2024-03-04T12:36:17.609 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls_filter": 0, 2024-03-04T12:36:17.609 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_other": 0, 2024-03-04T12:36:17.609 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_active": 0, 2024-03-04T12:36:17.610 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_send": 0, 2024-03-04T12:36:17.610 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_resend": 0, 2024-03-04T12:36:17.610 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_ping": 0, 2024-03-04T12:36:17.610 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_active": 0, 2024-03-04T12:36:17.610 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_send": 0, 2024-03-04T12:36:17.610 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_resend": 0, 2024-03-04T12:36:17.610 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_active": 0, 2024-03-04T12:36:17.610 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_send": 0, 2024-03-04T12:36:17.611 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_resend": 0, 2024-03-04T12:36:17.611 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_active": 0, 2024-03-04T12:36:17.611 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_send": 0, 2024-03-04T12:36:17.611 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_resend": 0, 2024-03-04T12:36:17.611 INFO:teuthology.orchestra.run.smithi063.stdout: "command_active": 0, 2024-03-04T12:36:17.611 INFO:teuthology.orchestra.run.smithi063.stdout: "command_send": 0, 2024-03-04T12:36:17.611 INFO:teuthology.orchestra.run.smithi063.stdout: "command_resend": 0, 2024-03-04T12:36:17.611 INFO:teuthology.orchestra.run.smithi063.stdout: "map_epoch": 105, 2024-03-04T12:36:17.611 INFO:teuthology.orchestra.run.smithi063.stdout: "map_full": 0, 2024-03-04T12:36:17.612 INFO:teuthology.orchestra.run.smithi063.stdout: "map_inc": 16, 2024-03-04T12:36:17.612 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_sessions": 7, 2024-03-04T12:36:17.612 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_open": 7, 2024-03-04T12:36:17.612 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_close": 0, 2024-03-04T12:36:17.612 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_laggy": 0, 2024-03-04T12:36:17.612 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_wr": 0, 2024-03-04T12:36:17.612 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_rd": 2, 2024-03-04T12:36:17.612 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_del": 0 2024-03-04T12:36:17.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.613 INFO:teuthology.orchestra.run.smithi063.stdout: "oft": { 2024-03-04T12:36:17.613 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_objs": 0, 2024-03-04T12:36:17.613 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_kv_pairs": 0, 2024-03-04T12:36:17.613 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_updates": 0, 2024-03-04T12:36:17.613 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_removes": 0 2024-03-04T12:36:17.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.613 INFO:teuthology.orchestra.run.smithi063.stdout: "osd-slow-ops": { 2024-03-04T12:36:17.613 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_ops_count": 0 2024-03-04T12:36:17.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.614 INFO:teuthology.orchestra.run.smithi063.stdout: "purge_queue": { 2024-03-04T12:36:17.614 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops": 0, 2024-03-04T12:36:17.614 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops_high_water": 0, 2024-03-04T12:36:17.614 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing": 0, 2024-03-04T12:36:17.614 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_high_water": 0, 2024-03-04T12:36:17.614 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed_ops": 0, 2024-03-04T12:36:17.614 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed": 0, 2024-03-04T12:36:17.614 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_item_in_journal": 0 2024-03-04T12:36:17.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.615 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-04T12:36:17.615 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:17.615 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-04T12:36:17.615 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:17.615 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 2519, 2024-03-04T12:36:17.615 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 18352350, 2024-03-04T12:36:17.615 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:17.616 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 2519, 2024-03-04T12:36:17.616 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-04T12:36:17.616 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-04T12:36:17.616 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2519, 2024-03-04T12:36:17.616 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 18352350, 2024-03-04T12:36:17.616 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:17.616 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.616 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.616 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.617 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:17.617 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.617 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_bytes": { 2024-03-04T12:36:17.617 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:17.617 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-04T12:36:17.617 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:17.617 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-04T12:36:17.617 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-04T12:36:17.618 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:17.618 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:17.618 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2249, 2024-03-04T12:36:17.618 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 16121937, 2024-03-04T12:36:17.618 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2249, 2024-03-04T12:36:17.618 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 16121937, 2024-03-04T12:36:17.618 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:17.618 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.618 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.619 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.619 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:17.619 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.619 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_ops": { 2024-03-04T12:36:17.619 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:17.619 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 1024, 2024-03-04T12:36:17.619 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:17.619 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-04T12:36:17.619 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-04T12:36:17.620 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:17.620 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:17.620 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2249, 2024-03-04T12:36:17.620 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 2249, 2024-03-04T12:36:17.620 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2249, 2024-03-04T12:36:17.620 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 2249, 2024-03-04T12:36:17.620 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:17.620 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.621 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.621 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.621 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:17.621 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.621 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle": { 2024-03-04T12:36:17.621 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:17.621 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-04T12:36:17.621 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:17.622 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-04T12:36:17.622 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-04T12:36:17.622 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:17.622 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:17.622 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-04T12:36:17.622 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-04T12:36:17.622 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-03-04T12:36:17.622 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-03-04T12:36:17.622 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:17.623 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.623 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.623 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.623 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:17.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:17.623 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle-0x558fea69c960": { 2024-03-04T12:36:17.623 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:17.623 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-04T12:36:17.624 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:17.624 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-04T12:36:17.624 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-04T12:36:17.624 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:17.624 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:17.624 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-04T12:36:17.624 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-04T12:36:17.624 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-03-04T12:36:17.625 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-03-04T12:36:17.625 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:17.625 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:17.625 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:17.625 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:17.625 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:17.625 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:17.625 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-04T12:36:18.583 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-04T12:36:18.583 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-04T12:36:18.584 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-03-04T12:36:18.584 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-04T12:36:18.584 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-04T12:36:18.584 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-04T12:36:18.584 DEBUG:tasks.check_counter:Getting stats from h 2024-03-04T12:36:18.584 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-04T12:36:18.836 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T12:36:19.250 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:19 smithi063 ceph-mon[36717]: pgmap v326: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:36:19.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:19 smithi086 ceph-mon[36851]: pgmap v326: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:36:19.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:19 smithi060 ceph-mon[32745]: pgmap v326: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-04T12:36:20.622 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-04T12:36:20.622 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-03-04T12:36:20.623 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 2568, 2024-03-04T12:36:20.623 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 2870, 2024-03-04T12:36:20.623 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1873532, 2024-03-04T12:36:20.623 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 6091017, 2024-03-04T12:36:20.623 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 10, 2024-03-04T12:36:20.623 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 4, 2024-03-04T12:36:20.623 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.219849238, 2024-03-04T12:36:20.623 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.097608147, 2024-03-04T12:36:20.624 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.095230648, 2024-03-04T12:36:20.624 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.029339219, 2024-03-04T12:36:20.624 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:20.624 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2870, 2024-03-04T12:36:20.624 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.121395485, 2024-03-04T12:36:20.624 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000042298 2024-03-04T12:36:20.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.624 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:20.625 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3318, 2024-03-04T12:36:20.625 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000624300, 2024-03-04T12:36:20.625 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000188 2024-03-04T12:36:20.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.625 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-04T12:36:20.625 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 0 2024-03-04T12:36:20.625 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.625 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-03-04T12:36:20.625 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 1397, 2024-03-04T12:36:20.626 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 1560, 2024-03-04T12:36:20.626 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 422392, 2024-03-04T12:36:20.626 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 9176906, 2024-03-04T12:36:20.626 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 17, 2024-03-04T12:36:20.626 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 8, 2024-03-04T12:36:20.626 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.126052055, 2024-03-04T12:36:20.626 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.049494419, 2024-03-04T12:36:20.626 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.028670161, 2024-03-04T12:36:20.627 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.036959197, 2024-03-04T12:36:20.627 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:20.627 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1557, 2024-03-04T12:36:20.627 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.114852663, 2024-03-04T12:36:20.627 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000073765 2024-03-04T12:36:20.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.627 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:20.627 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 15, 2024-03-04T12:36:20.627 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000002055, 2024-03-04T12:36:20.628 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000137 2024-03-04T12:36:20.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.628 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-04T12:36:20.628 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 444064 2024-03-04T12:36:20.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.628 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-03-04T12:36:20.628 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 9856, 2024-03-04T12:36:20.628 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 22885, 2024-03-04T12:36:20.628 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 5511235, 2024-03-04T12:36:20.629 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 19771569, 2024-03-04T12:36:20.629 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 16, 2024-03-04T12:36:20.629 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 9, 2024-03-04T12:36:20.629 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.868906970, 2024-03-04T12:36:20.629 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.404818186, 2024-03-04T12:36:20.629 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.958825777, 2024-03-04T12:36:20.630 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.025583881, 2024-03-04T12:36:20.630 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:20.630 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 22883, 2024-03-04T12:36:20.630 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.376897609, 2024-03-04T12:36:20.630 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000016470 2024-03-04T12:36:20.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.630 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:20.630 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 29364, 2024-03-04T12:36:20.631 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.005407115, 2024-03-04T12:36:20.631 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000184 2024-03-04T12:36:20.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.631 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 4695680, 2024-03-04T12:36:20.631 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 12043360 2024-03-04T12:36:20.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.631 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-03-04T12:36:20.631 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 1, 2024-03-04T12:36:20.632 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-03-04T12:36:20.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.632 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-MDSRank": { 2024-03-04T12:36:20.632 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-04T12:36:20.632 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-04T12:36:20.632 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1242, 2024-03-04T12:36:20.632 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 4.174652245, 2024-03-04T12:36:20.632 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.003361233 2024-03-04T12:36:20.633 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:20.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.633 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-PurgeQueue": { 2024-03-04T12:36:20.633 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-04T12:36:20.633 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-04T12:36:20.633 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1177, 2024-03-04T12:36:20.633 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.099511250, 2024-03-04T12:36:20.633 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000084546 2024-03-04T12:36:20.633 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-04T12:36:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "request": 5251, 2024-03-04T12:36:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "reply": 5232, 2024-03-04T12:36:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "reply_latency": { 2024-03-04T12:36:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 5232, 2024-03-04T12:36:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 452.521272098, 2024-03-04T12:36:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.086491068 2024-03-04T12:36:20.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_reply": 0, 2024-03-04T12:36:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "forward": 19, 2024-03-04T12:36:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_complete": 0, 2024-03-04T12:36:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_keys": 0, 2024-03-04T12:36:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_commit": 166, 2024-03-04T12:36:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_split": 19, 2024-03-04T12:36:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_merge": 0, 2024-03-04T12:36:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes": 1097, 2024-03-04T12:36:20.635 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_top": 244, 2024-03-04T12:36:20.636 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_bottom": 822, 2024-03-04T12:36:20.636 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pin_tail": 31, 2024-03-04T12:36:20.636 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pinned": 742, 2024-03-04T12:36:20.636 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_expired": 3834, 2024-03-04T12:36:20.636 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_with_caps": 598, 2024-03-04T12:36:20.636 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": 598, 2024-03-04T12:36:20.636 INFO:teuthology.orchestra.run.smithi063.stdout: "subtrees": 5, 2024-03-04T12:36:20.636 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse": 6474, 2024-03-04T12:36:20.637 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_hit": 5144, 2024-03-04T12:36:20.637 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_forward": 14, 2024-03-04T12:36:20.637 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_discover": 3, 2024-03-04T12:36:20.637 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_dir_fetch": 0, 2024-03-04T12:36:20.637 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_remote_ino": 0, 2024-03-04T12:36:20.637 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_lock": 36, 2024-03-04T12:36:20.637 INFO:teuthology.orchestra.run.smithi063.stdout: "load_cent": 0, 2024-03-04T12:36:20.637 INFO:teuthology.orchestra.run.smithi063.stdout: "q": 0, 2024-03-04T12:36:20.637 INFO:teuthology.orchestra.run.smithi063.stdout: "exported": 80, 2024-03-04T12:36:20.637 INFO:teuthology.orchestra.run.smithi063.stdout: "exported_inodes": 3785, 2024-03-04T12:36:20.638 INFO:teuthology.orchestra.run.smithi063.stdout: "imported": 2, 2024-03-04T12:36:20.638 INFO:teuthology.orchestra.run.smithi063.stdout: "imported_inodes": 1322, 2024-03-04T12:36:20.638 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_dir_fetch": 0, 2024-03-04T12:36:20.638 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_backtrace_fetch": 1028, 2024-03-04T12:36:20.638 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_peer_discover": 1, 2024-03-04T12:36:20.638 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rfiles": 771, 2024-03-04T12:36:20.639 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rbytes": 13163370, 2024-03-04T12:36:20.639 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rsnaps": 2, 2024-03-04T12:36:20.639 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_fetch": 1028, 2024-03-04T12:36:20.639 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_set_tag": 0, 2024-03-04T12:36:20.639 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_repaired": 0, 2024-03-04T12:36:20.639 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_inotable_repaired": 0, 2024-03-04T12:36:20.639 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_inodes": 40, 2024-03-04T12:36:20.639 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_base_inodes": 0, 2024-03-04T12:36:20.639 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dirfrag_rstats": 40, 2024-03-04T12:36:20.640 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_file_inodes": 988, 2024-03-04T12:36:20.640 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_inode_file_caps": 5, 2024-03-04T12:36:20.640 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_revoke": 121, 2024-03-04T12:36:20.640 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_grant": 5046, 2024-03-04T12:36:20.640 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_trunc": 8, 2024-03-04T12:36:20.640 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-04T12:36:20.640 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-04T12:36:20.640 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps": 1988, 2024-03-04T12:36:20.641 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps_dirty": 1072, 2024-03-04T12:36:20.641 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_cap_release": 264, 2024-03-04T12:36:20.641 INFO:teuthology.orchestra.run.smithi063.stdout: "process_request_cap_release": 1497 2024-03-04T12:36:20.641 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.641 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache": { 2024-03-04T12:36:20.641 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update": 4, 2024-03-04T12:36:20.641 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update_receipt": 0, 2024-03-04T12:36:20.641 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_try_discover": 0, 2024-03-04T12:36:20.641 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_send_discover": 8, 2024-03-04T12:36:20.642 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_handle_discover": 103, 2024-03-04T12:36:20.642 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays": 0, 2024-03-04T12:36:20.642 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_delayed": 0, 2024-03-04T12:36:20.642 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_enqueuing": 0, 2024-03-04T12:36:20.642 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_created": 614, 2024-03-04T12:36:20.642 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_enqueued": 614, 2024-03-04T12:36:20.642 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_reintegrated": 0, 2024-03-04T12:36:20.642 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_migrated": 0, 2024-03-04T12:36:20.642 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_processing": 0, 2024-03-04T12:36:20.643 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_enqueued": 0, 2024-03-04T12:36:20.643 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_prioritized": 0, 2024-03-04T12:36:20.643 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_started": 0, 2024-03-04T12:36:20.643 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_completed": 0, 2024-03-04T12:36:20.643 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_enqueue_scrub": 0, 2024-03-04T12:36:20.643 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_exportdir": 89, 2024-03-04T12:36:20.643 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_flush": 0, 2024-03-04T12:36:20.644 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragmentdir": 19, 2024-03-04T12:36:20.644 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragstats": 0, 2024-03-04T12:36:20.644 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_inodestats": 0 2024-03-04T12:36:20.644 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.644 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log": { 2024-03-04T12:36:20.644 INFO:teuthology.orchestra.run.smithi063.stdout: "evlrg": 1, 2024-03-04T12:36:20.644 INFO:teuthology.orchestra.run.smithi063.stdout: "evadd": 7417, 2024-03-04T12:36:20.644 INFO:teuthology.orchestra.run.smithi063.stdout: "evex": 0, 2024-03-04T12:36:20.644 INFO:teuthology.orchestra.run.smithi063.stdout: "evtrm": 0, 2024-03-04T12:36:20.645 INFO:teuthology.orchestra.run.smithi063.stdout: "ev": 7417, 2024-03-04T12:36:20.645 INFO:teuthology.orchestra.run.smithi063.stdout: "evexg": 0, 2024-03-04T12:36:20.645 INFO:teuthology.orchestra.run.smithi063.stdout: "evexd": 0, 2024-03-04T12:36:20.645 INFO:teuthology.orchestra.run.smithi063.stdout: "segadd": 10, 2024-03-04T12:36:20.645 INFO:teuthology.orchestra.run.smithi063.stdout: "segex": 0, 2024-03-04T12:36:20.645 INFO:teuthology.orchestra.run.smithi063.stdout: "segtrm": 0, 2024-03-04T12:36:20.645 INFO:teuthology.orchestra.run.smithi063.stdout: "seg": 10, 2024-03-04T12:36:20.646 INFO:teuthology.orchestra.run.smithi063.stdout: "segmjr": 2, 2024-03-04T12:36:20.646 INFO:teuthology.orchestra.run.smithi063.stdout: "segexg": 0, 2024-03-04T12:36:20.646 INFO:teuthology.orchestra.run.smithi063.stdout: "segexd": 0, 2024-03-04T12:36:20.646 INFO:teuthology.orchestra.run.smithi063.stdout: "expos": 4194304, 2024-03-04T12:36:20.646 INFO:teuthology.orchestra.run.smithi063.stdout: "wrpos": 20316236, 2024-03-04T12:36:20.646 INFO:teuthology.orchestra.run.smithi063.stdout: "rdpos": 0, 2024-03-04T12:36:20.646 INFO:teuthology.orchestra.run.smithi063.stdout: "jlat": { 2024-03-04T12:36:20.646 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 484, 2024-03-04T12:36:20.646 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 6.055812566, 2024-03-04T12:36:20.647 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.012512009 2024-03-04T12:36:20.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.647 INFO:teuthology.orchestra.run.smithi063.stdout: "replayed": 0 2024-03-04T12:36:20.647 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.647 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_mem": { 2024-03-04T12:36:20.647 INFO:teuthology.orchestra.run.smithi063.stdout: "ino": 703, 2024-03-04T12:36:20.647 INFO:teuthology.orchestra.run.smithi063.stdout: "ino+": 5056, 2024-03-04T12:36:20.647 INFO:teuthology.orchestra.run.smithi063.stdout: "ino-": 4353, 2024-03-04T12:36:20.647 INFO:teuthology.orchestra.run.smithi063.stdout: "dir": 179, 2024-03-04T12:36:20.648 INFO:teuthology.orchestra.run.smithi063.stdout: "dir+": 284, 2024-03-04T12:36:20.648 INFO:teuthology.orchestra.run.smithi063.stdout: "dir-": 105, 2024-03-04T12:36:20.648 INFO:teuthology.orchestra.run.smithi063.stdout: "dn": 1097, 2024-03-04T12:36:20.648 INFO:teuthology.orchestra.run.smithi063.stdout: "dn+": 5681, 2024-03-04T12:36:20.648 INFO:teuthology.orchestra.run.smithi063.stdout: "dn-": 4584, 2024-03-04T12:36:20.648 INFO:teuthology.orchestra.run.smithi063.stdout: "cap": 598, 2024-03-04T12:36:20.648 INFO:teuthology.orchestra.run.smithi063.stdout: "cap+": 4990, 2024-03-04T12:36:20.648 INFO:teuthology.orchestra.run.smithi063.stdout: "cap-": 4392, 2024-03-04T12:36:20.649 INFO:teuthology.orchestra.run.smithi063.stdout: "rss": 84868, 2024-03-04T12:36:20.649 INFO:teuthology.orchestra.run.smithi063.stdout: "heap": 207228 2024-03-04T12:36:20.649 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.649 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server": { 2024-03-04T12:36:20.649 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_client_request": 6132, 2024-03-04T12:36:20.649 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_server_request": 7, 2024-03-04T12:36:20.649 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_request": 5251, 2024-03-04T12:36:20.649 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_session": 96, 2024-03-04T12:36:20.650 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_peer_request": 209, 2024-03-04T12:36:20.650 INFO:teuthology.orchestra.run.smithi063.stdout: "req_create_latency": { 2024-03-04T12:36:20.650 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3918, 2024-03-04T12:36:20.650 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 447.899003579, 2024-03-04T12:36:20.650 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.114318275 2024-03-04T12:36:20.650 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.650 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getattr_latency": { 2024-03-04T12:36:20.650 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 15, 2024-03-04T12:36:20.650 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 1.248404350, 2024-03-04T12:36:20.651 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.083226956 2024-03-04T12:36:20.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.651 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getfilelock_latency": { 2024-03-04T12:36:20.651 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.651 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.651 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.651 INFO:teuthology.orchestra.run.smithi063.stdout: "req_link_latency": { 2024-03-04T12:36:20.651 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.652 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.652 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.652 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookup_latency": { 2024-03-04T12:36:20.652 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 359, 2024-03-04T12:36:20.652 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 1.667412585, 2024-03-04T12:36:20.652 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.004644603 2024-03-04T12:36:20.652 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.653 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookuphash_latency": { 2024-03-04T12:36:20.653 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.653 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.653 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.653 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupino_latency": { 2024-03-04T12:36:20.653 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.653 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.654 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.654 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupname_latency": { 2024-03-04T12:36:20.654 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.654 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.654 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.654 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupparent_latency": { 2024-03-04T12:36:20.654 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.655 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.655 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.655 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupsnap_latency": { 2024-03-04T12:36:20.655 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.655 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.655 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.656 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lssnap_latency": { 2024-03-04T12:36:20.656 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.656 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.657 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.657 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mkdir_latency": { 2024-03-04T12:36:20.657 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 36, 2024-03-04T12:36:20.657 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.408284948, 2024-03-04T12:36:20.658 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.011341248 2024-03-04T12:36:20.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.658 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mknod_latency": { 2024-03-04T12:36:20.658 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.658 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.658 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.658 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mksnap_latency": { 2024-03-04T12:36:20.659 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.659 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.659 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.659 INFO:teuthology.orchestra.run.smithi063.stdout: "req_open_latency": { 2024-03-04T12:36:20.659 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 139, 2024-03-04T12:36:20.659 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.050472781, 2024-03-04T12:36:20.659 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000363113 2024-03-04T12:36:20.659 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.660 INFO:teuthology.orchestra.run.smithi063.stdout: "req_readdir_latency": { 2024-03-04T12:36:20.660 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 101, 2024-03-04T12:36:20.660 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.153182941, 2024-03-04T12:36:20.660 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001516662 2024-03-04T12:36:20.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.660 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rename_latency": { 2024-03-04T12:36:20.660 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 62, 2024-03-04T12:36:20.660 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.117402032, 2024-03-04T12:36:20.660 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001893581 2024-03-04T12:36:20.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.661 INFO:teuthology.orchestra.run.smithi063.stdout: "req_renamesnap_latency": { 2024-03-04T12:36:20.661 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.661 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.661 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.661 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.661 INFO:teuthology.orchestra.run.smithi063.stdout: "req_snapdiff_latency": { 2024-03-04T12:36:20.661 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.662 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.662 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.662 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmdir_latency": { 2024-03-04T12:36:20.662 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 13, 2024-03-04T12:36:20.662 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.096258016, 2024-03-04T12:36:20.662 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.007404462 2024-03-04T12:36:20.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.663 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmsnap_latency": { 2024-03-04T12:36:20.663 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.663 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.663 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.663 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmxattr_latency": { 2024-03-04T12:36:20.663 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.663 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.664 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.664 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setattr_latency": { 2024-03-04T12:36:20.664 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 8, 2024-03-04T12:36:20.665 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.034682560, 2024-03-04T12:36:20.665 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.004335320 2024-03-04T12:36:20.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.665 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setdirlayout_latency": { 2024-03-04T12:36:20.665 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.665 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.665 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.665 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setfilelock_latency": { 2024-03-04T12:36:20.666 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 8, 2024-03-04T12:36:20.666 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.002868601, 2024-03-04T12:36:20.666 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000358575 2024-03-04T12:36:20.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.666 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setlayout_latency": { 2024-03-04T12:36:20.666 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.666 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.667 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.667 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setxattr_latency": { 2024-03-04T12:36:20.667 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.667 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.667 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.667 INFO:teuthology.orchestra.run.smithi063.stdout: "req_symlink_latency": { 2024-03-04T12:36:20.668 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 6, 2024-03-04T12:36:20.668 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.006085945, 2024-03-04T12:36:20.668 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001014324 2024-03-04T12:36:20.668 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.668 INFO:teuthology.orchestra.run.smithi063.stdout: "req_unlink_latency": { 2024-03-04T12:36:20.668 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 567, 2024-03-04T12:36:20.668 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.837213760, 2024-03-04T12:36:20.669 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001476567 2024-03-04T12:36:20.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.669 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_revoke_eviction": 0, 2024-03-04T12:36:20.669 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_acquisition_throttle": 0, 2024-03-04T12:36:20.669 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getvxattr_latency": { 2024-03-04T12:36:20.669 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.669 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.669 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.670 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:20.670 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.670 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sessions": { 2024-03-04T12:36:20.670 INFO:teuthology.orchestra.run.smithi063.stdout: "session_count": 1, 2024-03-04T12:36:20.670 INFO:teuthology.orchestra.run.smithi063.stdout: "session_add": 3, 2024-03-04T12:36:20.670 INFO:teuthology.orchestra.run.smithi063.stdout: "session_remove": 2, 2024-03-04T12:36:20.670 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_open": 1, 2024-03-04T12:36:20.670 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_stale": 0, 2024-03-04T12:36:20.671 INFO:teuthology.orchestra.run.smithi063.stdout: "total_load": 4030, 2024-03-04T12:36:20.671 INFO:teuthology.orchestra.run.smithi063.stdout: "average_load": 4030, 2024-03-04T12:36:20.671 INFO:teuthology.orchestra.run.smithi063.stdout: "avg_session_uptime": 624, 2024-03-04T12:36:20.671 INFO:teuthology.orchestra.run.smithi063.stdout: "mdthresh_evicted": 0 2024-03-04T12:36:20.671 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.671 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-03-04T12:36:20.671 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-03-04T12:36:20.671 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-03-04T12:36:20.671 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-03-04T12:36:20.672 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-03-04T12:36:20.672 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-03-04T12:36:20.672 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-03-04T12:36:20.672 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-04T12:36:20.672 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-03-04T12:36:20.672 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-04T12:36:20.672 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-03-04T12:36:20.672 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-03-04T12:36:20.673 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-03-04T12:36:20.673 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-04T12:36:20.673 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-03-04T12:36:20.673 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-03-04T12:36:20.673 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-03-04T12:36:20.673 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-03-04T12:36:20.673 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-03-04T12:36:20.673 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-04T12:36:20.674 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-03-04T12:36:20.674 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-04T12:36:20.674 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-03-04T12:36:20.675 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-03-04T12:36:20.675 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-03-04T12:36:20.675 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-03-04T12:36:20.675 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-03-04T12:36:20.675 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-04T12:36:20.675 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-03-04T12:36:20.675 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-03-04T12:36:20.675 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-03-04T12:36:20.675 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-03-04T12:36:20.676 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-03-04T12:36:20.676 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-03-04T12:36:20.676 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-03-04T12:36:20.676 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-03-04T12:36:20.676 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-03-04T12:36:20.676 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 341589, 2024-03-04T12:36:20.676 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 813, 2024-03-04T12:36:20.676 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 1760, 2024-03-04T12:36:20.677 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 20, 2024-03-04T12:36:20.677 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-03-04T12:36:20.677 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-03-04T12:36:20.677 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-03-04T12:36:20.677 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-03-04T12:36:20.677 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-03-04T12:36:20.677 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-03-04T12:36:20.677 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 9736, 2024-03-04T12:36:20.677 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 203, 2024-03-04T12:36:20.678 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-03-04T12:36:20.678 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-03-04T12:36:20.678 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-03-04T12:36:20.678 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-03-04T12:36:20.678 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 2946886, 2024-03-04T12:36:20.678 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 34841, 2024-03-04T12:36:20.678 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-03-04T12:36:20.678 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-03-04T12:36:20.679 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-03-04T12:36:20.679 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-03-04T12:36:20.679 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.679 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter": { 2024-03-04T12:36:20.679 INFO:teuthology.orchestra.run.smithi063.stdout: "op_active": 0, 2024-03-04T12:36:20.679 INFO:teuthology.orchestra.run.smithi063.stdout: "op_laggy": 0, 2024-03-04T12:36:20.679 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send": 3046, 2024-03-04T12:36:20.679 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send_bytes": 19125485, 2024-03-04T12:36:20.679 INFO:teuthology.orchestra.run.smithi063.stdout: "op_resend": 27, 2024-03-04T12:36:20.680 INFO:teuthology.orchestra.run.smithi063.stdout: "op_reply": 3046, 2024-03-04T12:36:20.680 INFO:teuthology.orchestra.run.smithi063.stdout: "op_latency": { 2024-03-04T12:36:20.680 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3046, 2024-03-04T12:36:20.680 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 6.533926160, 2024-03-04T12:36:20.681 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.002145084 2024-03-04T12:36:20.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.681 INFO:teuthology.orchestra.run.smithi063.stdout: "op_inflight": 0, 2024-03-04T12:36:20.681 INFO:teuthology.orchestra.run.smithi063.stdout: "oplen_avg": { 2024-03-04T12:36:20.681 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3046, 2024-03-04T12:36:20.681 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 3268 2024-03-04T12:36:20.681 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.681 INFO:teuthology.orchestra.run.smithi063.stdout: "op": 3046, 2024-03-04T12:36:20.681 INFO:teuthology.orchestra.run.smithi063.stdout: "op_r": 1047, 2024-03-04T12:36:20.682 INFO:teuthology.orchestra.run.smithi063.stdout: "op_w": 1999, 2024-03-04T12:36:20.682 INFO:teuthology.orchestra.run.smithi063.stdout: "op_rmw": 0, 2024-03-04T12:36:20.682 INFO:teuthology.orchestra.run.smithi063.stdout: "op_pg": 0, 2024-03-04T12:36:20.682 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_stat": 0, 2024-03-04T12:36:20.682 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_create": 10, 2024-03-04T12:36:20.682 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_read": 19, 2024-03-04T12:36:20.682 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_write": 503, 2024-03-04T12:36:20.682 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writefull": 24, 2024-03-04T12:36:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writesame": 0, 2024-03-04T12:36:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_append": 0, 2024-03-04T12:36:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_zero": 0, 2024-03-04T12:36:20.683 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_truncate": 3, 2024-03-04T12:36:20.684 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_delete": 1248, 2024-03-04T12:36:20.684 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_mapext": 0, 2024-03-04T12:36:20.684 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_sparse_read": 0, 2024-03-04T12:36:20.684 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_clonerange": 0, 2024-03-04T12:36:20.684 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_getxattr": 1028, 2024-03-04T12:36:20.684 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_setxattr": 20, 2024-03-04T12:36:20.685 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_cmpxattr": 0, 2024-03-04T12:36:20.685 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_rmxattr": 0, 2024-03-04T12:36:20.685 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_resetxattrs": 0, 2024-03-04T12:36:20.685 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_call": 0, 2024-03-04T12:36:20.685 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_watch": 0, 2024-03-04T12:36:20.685 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_notify": 0, 2024-03-04T12:36:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_src_cmpxattr": 0, 2024-03-04T12:36:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls": 0, 2024-03-04T12:36:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls_filter": 0, 2024-03-04T12:36:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_other": 8, 2024-03-04T12:36:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_active": 0, 2024-03-04T12:36:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_send": 0, 2024-03-04T12:36:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_resend": 0, 2024-03-04T12:36:20.686 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_ping": 0, 2024-03-04T12:36:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_active": 0, 2024-03-04T12:36:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_send": 0, 2024-03-04T12:36:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_resend": 0, 2024-03-04T12:36:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_active": 0, 2024-03-04T12:36:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_send": 0, 2024-03-04T12:36:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_resend": 0, 2024-03-04T12:36:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_active": 0, 2024-03-04T12:36:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_send": 0, 2024-03-04T12:36:20.687 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_resend": 0, 2024-03-04T12:36:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "command_active": 0, 2024-03-04T12:36:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "command_send": 0, 2024-03-04T12:36:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "command_resend": 0, 2024-03-04T12:36:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "map_epoch": 105, 2024-03-04T12:36:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "map_full": 0, 2024-03-04T12:36:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "map_inc": 16, 2024-03-04T12:36:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_sessions": 12, 2024-03-04T12:36:20.688 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_open": 12, 2024-03-04T12:36:20.689 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_close": 0, 2024-03-04T12:36:20.689 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_laggy": 0, 2024-03-04T12:36:20.689 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_wr": 386, 2024-03-04T12:36:20.689 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_rd": 0, 2024-03-04T12:36:20.689 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_del": 19 2024-03-04T12:36:20.689 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.689 INFO:teuthology.orchestra.run.smithi063.stdout: "oft": { 2024-03-04T12:36:20.689 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_objs": 1, 2024-03-04T12:36:20.689 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_kv_pairs": 0, 2024-03-04T12:36:20.690 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_updates": 0, 2024-03-04T12:36:20.690 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_removes": 0 2024-03-04T12:36:20.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.690 INFO:teuthology.orchestra.run.smithi063.stdout: "osd-slow-ops": { 2024-03-04T12:36:20.690 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_ops_count": 0 2024-03-04T12:36:20.690 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.690 INFO:teuthology.orchestra.run.smithi063.stdout: "purge_queue": { 2024-03-04T12:36:20.690 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops": 0, 2024-03-04T12:36:20.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops_high_water": 10, 2024-03-04T12:36:20.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing": 0, 2024-03-04T12:36:20.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_high_water": 7, 2024-03-04T12:36:20.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed_ops": 1217, 2024-03-04T12:36:20.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed": 614, 2024-03-04T12:36:20.691 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_item_in_journal": 0 2024-03-04T12:36:20.691 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.691 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-04T12:36:20.691 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:20.692 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-04T12:36:20.692 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:20.692 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 13817, 2024-03-04T12:36:20.692 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 6151874, 2024-03-04T12:36:20.692 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:20.692 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 13817, 2024-03-04T12:36:20.692 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-04T12:36:20.692 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-04T12:36:20.693 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 13817, 2024-03-04T12:36:20.693 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 6151874, 2024-03-04T12:36:20.693 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:20.693 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.693 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.693 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.693 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:20.693 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.693 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_bytes": { 2024-03-04T12:36:20.694 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:20.694 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-04T12:36:20.694 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:20.694 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-04T12:36:20.694 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-04T12:36:20.694 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:20.694 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:20.694 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 3046, 2024-03-04T12:36:20.694 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 19192323, 2024-03-04T12:36:20.695 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 3046, 2024-03-04T12:36:20.695 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 19192323, 2024-03-04T12:36:20.695 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:20.695 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.695 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.695 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.695 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:20.695 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.695 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_ops": { 2024-03-04T12:36:20.696 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:20.696 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 1024, 2024-03-04T12:36:20.696 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:20.696 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-04T12:36:20.696 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-04T12:36:20.696 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:20.696 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:20.696 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 3046, 2024-03-04T12:36:20.697 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 3046, 2024-03-04T12:36:20.697 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 3046, 2024-03-04T12:36:20.697 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 3046, 2024-03-04T12:36:20.697 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:20.697 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.697 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.697 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.697 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:20.697 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.698 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle": { 2024-03-04T12:36:20.698 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:20.698 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-04T12:36:20.698 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:20.698 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 614, 2024-03-04T12:36:20.698 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 66838, 2024-03-04T12:36:20.698 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:20.698 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 614, 2024-03-04T12:36:20.698 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-04T12:36:20.699 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-04T12:36:20.699 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 19, 2024-03-04T12:36:20.699 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 66838, 2024-03-04T12:36:20.699 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:20.699 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.699 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.699 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.699 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:20.700 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:20.700 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle-0x564e5cc82640": { 2024-03-04T12:36:20.700 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:20.700 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-04T12:36:20.700 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:20.700 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 7417, 2024-03-04T12:36:20.700 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 16121932, 2024-03-04T12:36:20.700 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:20.700 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 7417, 2024-03-04T12:36:20.701 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-04T12:36:20.701 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-04T12:36:20.701 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 484, 2024-03-04T12:36:20.701 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 16121932, 2024-03-04T12:36:20.701 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:20.701 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:20.701 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:20.701 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:20.701 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:20.702 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:20.702 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-04T12:36:21.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:21 smithi086 ceph-mon[36851]: pgmap v327: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:36:21.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:21 smithi063 ceph-mon[36717]: pgmap v327: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:36:21.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:21 smithi060 ceph-mon[32745]: pgmap v327: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:36:21.451 INFO:tasks.check_counter:Daemon mds.h mds.exported=80 2024-03-04T12:36:21.452 INFO:tasks.check_counter:Daemon mds.h mds.imported=2 2024-03-04T12:36:21.452 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=2 2024-03-04T12:36:21.452 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-04T12:36:21.452 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-04T12:36:21.452 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=19 2024-03-04T12:36:21.452 DEBUG:tasks.check_counter:Getting stats from k 2024-03-04T12:36:21.453 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-04T12:36:21.703 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T12:36:23.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:23 smithi063 ceph-mon[36717]: pgmap v328: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:36:23.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:23 smithi086 ceph-mon[36851]: pgmap v328: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:36:23.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:23 smithi060 ceph-mon[32745]: pgmap v328: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:36:23.452 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-03-04T12:36:23.452 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-03-04T12:36:23.452 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 746, 2024-03-04T12:36:23.452 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 694, 2024-03-04T12:36:23.452 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 247890, 2024-03-04T12:36:23.453 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 380303, 2024-03-04T12:36:23.453 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 10, 2024-03-04T12:36:23.453 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 4, 2024-03-04T12:36:23.453 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.078499438, 2024-03-04T12:36:23.453 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.035114178, 2024-03-04T12:36:23.453 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.039231616, 2024-03-04T12:36:23.453 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.004078275, 2024-03-04T12:36:23.453 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:23.454 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 694, 2024-03-04T12:36:23.454 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.050711830, 2024-03-04T12:36:23.454 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000073071 2024-03-04T12:36:23.454 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.454 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:23.454 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1378, 2024-03-04T12:36:23.454 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000292328, 2024-03-04T12:36:23.454 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000212 2024-03-04T12:36:23.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.455 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-04T12:36:23.455 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 0 2024-03-04T12:36:23.455 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.455 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-03-04T12:36:23.455 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 321, 2024-03-04T12:36:23.455 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 309, 2024-03-04T12:36:23.455 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1842523, 2024-03-04T12:36:23.455 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 152887, 2024-03-04T12:36:23.456 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 6, 2024-03-04T12:36:23.456 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 5, 2024-03-04T12:36:23.456 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.062372907, 2024-03-04T12:36:23.456 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.016721378, 2024-03-04T12:36:23.456 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.054190557, 2024-03-04T12:36:23.456 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.006112849, 2024-03-04T12:36:23.456 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:23.456 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 308, 2024-03-04T12:36:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.028509078, 2024-03-04T12:36:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000092561 2024-03-04T12:36:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 34, 2024-03-04T12:36:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000009319, 2024-03-04T12:36:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000274 2024-03-04T12:36:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.458 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 1839232, 2024-03-04T12:36:23.458 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 147216 2024-03-04T12:36:23.458 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.458 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-03-04T12:36:23.458 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 43, 2024-03-04T12:36:23.458 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 226, 2024-03-04T12:36:23.458 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 17694, 2024-03-04T12:36:23.458 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 564259, 2024-03-04T12:36:23.458 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 21, 2024-03-04T12:36:23.459 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 6, 2024-03-04T12:36:23.459 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.027463027, 2024-03-04T12:36:23.459 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.011190167, 2024-03-04T12:36:23.459 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.012980009, 2024-03-04T12:36:23.459 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000370644, 2024-03-04T12:36:23.459 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:23.459 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 222, 2024-03-04T12:36:23.459 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.022434946, 2024-03-04T12:36:23.460 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000101058 2024-03-04T12:36:23.460 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.460 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:23.460 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 66, 2024-03-04T12:36:23.460 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000012698, 2024-03-04T12:36:23.461 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000192 2024-03-04T12:36:23.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.461 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 7008, 2024-03-04T12:36:23.461 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 451840 2024-03-04T12:36:23.461 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.461 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-03-04T12:36:23.461 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 1, 2024-03-04T12:36:23.462 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-03-04T12:36:23.462 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.462 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-MDSRank": { 2024-03-04T12:36:23.462 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-04T12:36:23.462 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-04T12:36:23.462 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 38, 2024-03-04T12:36:23.462 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.001954088, 2024-03-04T12:36:23.462 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000051423 2024-03-04T12:36:23.462 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:23.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.463 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-PurgeQueue": { 2024-03-04T12:36:23.463 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-03-04T12:36:23.463 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-03-04T12:36:23.463 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2, 2024-03-04T12:36:23.463 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000015954, 2024-03-04T12:36:23.463 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000007977 2024-03-04T12:36:23.463 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:23.464 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.464 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-03-04T12:36:23.464 INFO:teuthology.orchestra.run.smithi063.stdout: "request": 4, 2024-03-04T12:36:23.464 INFO:teuthology.orchestra.run.smithi063.stdout: "reply": 0, 2024-03-04T12:36:23.464 INFO:teuthology.orchestra.run.smithi063.stdout: "reply_latency": { 2024-03-04T12:36:23.464 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.464 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.465 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.465 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.465 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_reply": 0, 2024-03-04T12:36:23.465 INFO:teuthology.orchestra.run.smithi063.stdout: "forward": 4, 2024-03-04T12:36:23.465 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_complete": 0, 2024-03-04T12:36:23.465 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_keys": 0, 2024-03-04T12:36:23.465 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_commit": 11, 2024-03-04T12:36:23.465 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_split": 0, 2024-03-04T12:36:23.466 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_merge": 0, 2024-03-04T12:36:23.466 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes": 17, 2024-03-04T12:36:23.466 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_top": 0, 2024-03-04T12:36:23.466 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_bottom": 17, 2024-03-04T12:36:23.466 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pin_tail": 0, 2024-03-04T12:36:23.466 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pinned": 16, 2024-03-04T12:36:23.466 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_expired": 0, 2024-03-04T12:36:23.466 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_with_caps": 0, 2024-03-04T12:36:23.466 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": 0, 2024-03-04T12:36:23.467 INFO:teuthology.orchestra.run.smithi063.stdout: "subtrees": 2, 2024-03-04T12:36:23.467 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse": 7, 2024-03-04T12:36:23.467 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_hit": 2, 2024-03-04T12:36:23.467 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_forward": 4, 2024-03-04T12:36:23.467 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_discover": 1, 2024-03-04T12:36:23.467 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_dir_fetch": 0, 2024-03-04T12:36:23.467 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_remote_ino": 0, 2024-03-04T12:36:23.467 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_lock": 0, 2024-03-04T12:36:23.468 INFO:teuthology.orchestra.run.smithi063.stdout: "load_cent": 0, 2024-03-04T12:36:23.468 INFO:teuthology.orchestra.run.smithi063.stdout: "q": 0, 2024-03-04T12:36:23.468 INFO:teuthology.orchestra.run.smithi063.stdout: "exported": 0, 2024-03-04T12:36:23.468 INFO:teuthology.orchestra.run.smithi063.stdout: "exported_inodes": 0, 2024-03-04T12:36:23.468 INFO:teuthology.orchestra.run.smithi063.stdout: "imported": 0, 2024-03-04T12:36:23.468 INFO:teuthology.orchestra.run.smithi063.stdout: "imported_inodes": 0, 2024-03-04T12:36:23.468 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_dir_fetch": 0, 2024-03-04T12:36:23.468 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_backtrace_fetch": 0, 2024-03-04T12:36:23.468 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_peer_discover": 0, 2024-03-04T12:36:23.469 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rfiles": 771, 2024-03-04T12:36:23.469 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rbytes": 13163370, 2024-03-04T12:36:23.469 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rsnaps": 2, 2024-03-04T12:36:23.469 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_fetch": 0, 2024-03-04T12:36:23.469 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_set_tag": 0, 2024-03-04T12:36:23.469 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_repaired": 0, 2024-03-04T12:36:23.469 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_inotable_repaired": 0, 2024-03-04T12:36:23.469 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_inodes": 0, 2024-03-04T12:36:23.469 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_base_inodes": 0, 2024-03-04T12:36:23.470 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dirfrag_rstats": 0, 2024-03-04T12:36:23.470 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_file_inodes": 0, 2024-03-04T12:36:23.470 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_inode_file_caps": 0, 2024-03-04T12:36:23.470 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_revoke": 0, 2024-03-04T12:36:23.470 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_grant": 0, 2024-03-04T12:36:23.470 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_trunc": 0, 2024-03-04T12:36:23.470 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-04T12:36:23.470 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-04T12:36:23.471 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps": 0, 2024-03-04T12:36:23.471 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps_dirty": 0, 2024-03-04T12:36:23.471 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_cap_release": 0, 2024-03-04T12:36:23.471 INFO:teuthology.orchestra.run.smithi063.stdout: "process_request_cap_release": 0 2024-03-04T12:36:23.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.471 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache": { 2024-03-04T12:36:23.471 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update": 0, 2024-03-04T12:36:23.471 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update_receipt": 5, 2024-03-04T12:36:23.471 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_try_discover": 3, 2024-03-04T12:36:23.472 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_send_discover": 6, 2024-03-04T12:36:23.472 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_handle_discover": 0, 2024-03-04T12:36:23.472 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays": 0, 2024-03-04T12:36:23.472 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_delayed": 0, 2024-03-04T12:36:23.472 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_enqueuing": 0, 2024-03-04T12:36:23.472 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_created": 0, 2024-03-04T12:36:23.472 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_enqueued": 0, 2024-03-04T12:36:23.472 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_reintegrated": 0, 2024-03-04T12:36:23.472 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_migrated": 0, 2024-03-04T12:36:23.473 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_processing": 0, 2024-03-04T12:36:23.473 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_enqueued": 0, 2024-03-04T12:36:23.473 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_prioritized": 0, 2024-03-04T12:36:23.473 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_started": 0, 2024-03-04T12:36:23.473 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_completed": 0, 2024-03-04T12:36:23.473 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_enqueue_scrub": 0, 2024-03-04T12:36:23.473 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_exportdir": 0, 2024-03-04T12:36:23.473 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_flush": 0, 2024-03-04T12:36:23.474 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragmentdir": 0, 2024-03-04T12:36:23.474 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragstats": 0, 2024-03-04T12:36:23.474 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_inodestats": 0 2024-03-04T12:36:23.474 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.474 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log": { 2024-03-04T12:36:23.474 INFO:teuthology.orchestra.run.smithi063.stdout: "evlrg": 0, 2024-03-04T12:36:23.474 INFO:teuthology.orchestra.run.smithi063.stdout: "evadd": 6, 2024-03-04T12:36:23.474 INFO:teuthology.orchestra.run.smithi063.stdout: "evex": 0, 2024-03-04T12:36:23.475 INFO:teuthology.orchestra.run.smithi063.stdout: "evtrm": 0, 2024-03-04T12:36:23.475 INFO:teuthology.orchestra.run.smithi063.stdout: "ev": 6, 2024-03-04T12:36:23.475 INFO:teuthology.orchestra.run.smithi063.stdout: "evexg": 0, 2024-03-04T12:36:23.475 INFO:teuthology.orchestra.run.smithi063.stdout: "evexd": 0, 2024-03-04T12:36:23.475 INFO:teuthology.orchestra.run.smithi063.stdout: "segadd": 2, 2024-03-04T12:36:23.475 INFO:teuthology.orchestra.run.smithi063.stdout: "segex": 0, 2024-03-04T12:36:23.475 INFO:teuthology.orchestra.run.smithi063.stdout: "segtrm": 0, 2024-03-04T12:36:23.475 INFO:teuthology.orchestra.run.smithi063.stdout: "seg": 2, 2024-03-04T12:36:23.475 INFO:teuthology.orchestra.run.smithi063.stdout: "segmjr": 2, 2024-03-04T12:36:23.476 INFO:teuthology.orchestra.run.smithi063.stdout: "segexg": 0, 2024-03-04T12:36:23.476 INFO:teuthology.orchestra.run.smithi063.stdout: "segexd": 0, 2024-03-04T12:36:23.476 INFO:teuthology.orchestra.run.smithi063.stdout: "expos": 4194304, 2024-03-04T12:36:23.476 INFO:teuthology.orchestra.run.smithi063.stdout: "wrpos": 4196123, 2024-03-04T12:36:23.476 INFO:teuthology.orchestra.run.smithi063.stdout: "rdpos": 0, 2024-03-04T12:36:23.476 INFO:teuthology.orchestra.run.smithi063.stdout: "jlat": { 2024-03-04T12:36:23.476 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 5, 2024-03-04T12:36:23.476 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.008958719, 2024-03-04T12:36:23.477 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001791743 2024-03-04T12:36:23.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.477 INFO:teuthology.orchestra.run.smithi063.stdout: "replayed": 0 2024-03-04T12:36:23.477 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.477 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_mem": { 2024-03-04T12:36:23.477 INFO:teuthology.orchestra.run.smithi063.stdout: "ino": 20, 2024-03-04T12:36:23.477 INFO:teuthology.orchestra.run.smithi063.stdout: "ino+": 20, 2024-03-04T12:36:23.477 INFO:teuthology.orchestra.run.smithi063.stdout: "ino-": 0, 2024-03-04T12:36:23.477 INFO:teuthology.orchestra.run.smithi063.stdout: "dir": 18, 2024-03-04T12:36:23.478 INFO:teuthology.orchestra.run.smithi063.stdout: "dir+": 21, 2024-03-04T12:36:23.478 INFO:teuthology.orchestra.run.smithi063.stdout: "dir-": 3, 2024-03-04T12:36:23.478 INFO:teuthology.orchestra.run.smithi063.stdout: "dn": 17, 2024-03-04T12:36:23.478 INFO:teuthology.orchestra.run.smithi063.stdout: "dn+": 17, 2024-03-04T12:36:23.478 INFO:teuthology.orchestra.run.smithi063.stdout: "dn-": 0, 2024-03-04T12:36:23.478 INFO:teuthology.orchestra.run.smithi063.stdout: "cap": 0, 2024-03-04T12:36:23.478 INFO:teuthology.orchestra.run.smithi063.stdout: "cap+": 0, 2024-03-04T12:36:23.478 INFO:teuthology.orchestra.run.smithi063.stdout: "cap-": 0, 2024-03-04T12:36:23.479 INFO:teuthology.orchestra.run.smithi063.stdout: "rss": 50164, 2024-03-04T12:36:23.479 INFO:teuthology.orchestra.run.smithi063.stdout: "heap": 207228 2024-03-04T12:36:23.479 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.479 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server": { 2024-03-04T12:36:23.479 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_client_request": 4, 2024-03-04T12:36:23.479 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_server_request": 0, 2024-03-04T12:36:23.479 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_request": 4, 2024-03-04T12:36:23.479 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_session": 21, 2024-03-04T12:36:23.479 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_peer_request": 0, 2024-03-04T12:36:23.480 INFO:teuthology.orchestra.run.smithi063.stdout: "req_create_latency": { 2024-03-04T12:36:23.480 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.480 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.480 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.480 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.480 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getattr_latency": { 2024-03-04T12:36:23.480 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.480 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.481 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.481 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getfilelock_latency": { 2024-03-04T12:36:23.481 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.481 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.481 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.481 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.481 INFO:teuthology.orchestra.run.smithi063.stdout: "req_link_latency": { 2024-03-04T12:36:23.482 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.482 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.482 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.482 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookup_latency": { 2024-03-04T12:36:23.482 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.482 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.482 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.482 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.483 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookuphash_latency": { 2024-03-04T12:36:23.483 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.483 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.483 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.483 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.483 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupino_latency": { 2024-03-04T12:36:23.483 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.483 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.484 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.484 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupname_latency": { 2024-03-04T12:36:23.484 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.484 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.484 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.484 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.484 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupparent_latency": { 2024-03-04T12:36:23.485 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.485 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.485 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.485 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupsnap_latency": { 2024-03-04T12:36:23.485 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.485 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.485 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.485 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.486 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lssnap_latency": { 2024-03-04T12:36:23.486 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.486 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.486 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.486 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.486 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mkdir_latency": { 2024-03-04T12:36:23.486 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.486 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.487 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.487 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.487 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mknod_latency": { 2024-03-04T12:36:23.487 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.488 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.488 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.488 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.488 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mksnap_latency": { 2024-03-04T12:36:23.488 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.488 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.488 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.489 INFO:teuthology.orchestra.run.smithi063.stdout: "req_open_latency": { 2024-03-04T12:36:23.489 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.489 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.489 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.489 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.489 INFO:teuthology.orchestra.run.smithi063.stdout: "req_readdir_latency": { 2024-03-04T12:36:23.489 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.490 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.490 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.490 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rename_latency": { 2024-03-04T12:36:23.490 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.490 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.490 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.490 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.490 INFO:teuthology.orchestra.run.smithi063.stdout: "req_renamesnap_latency": { 2024-03-04T12:36:23.491 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.491 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.491 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.491 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.491 INFO:teuthology.orchestra.run.smithi063.stdout: "req_snapdiff_latency": { 2024-03-04T12:36:23.491 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.491 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.491 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.492 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmdir_latency": { 2024-03-04T12:36:23.492 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.492 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.492 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.492 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.492 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmsnap_latency": { 2024-03-04T12:36:23.492 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.493 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.493 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.493 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmxattr_latency": { 2024-03-04T12:36:23.493 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.493 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.493 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.493 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.494 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setattr_latency": { 2024-03-04T12:36:23.494 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.494 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.494 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.494 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.494 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setdirlayout_latency": { 2024-03-04T12:36:23.494 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.494 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.494 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.495 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setfilelock_latency": { 2024-03-04T12:36:23.495 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.495 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.495 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.495 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.495 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setlayout_latency": { 2024-03-04T12:36:23.495 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.496 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.496 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.496 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setxattr_latency": { 2024-03-04T12:36:23.496 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.496 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.496 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.496 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.497 INFO:teuthology.orchestra.run.smithi063.stdout: "req_symlink_latency": { 2024-03-04T12:36:23.497 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.497 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.497 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.497 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.497 INFO:teuthology.orchestra.run.smithi063.stdout: "req_unlink_latency": { 2024-03-04T12:36:23.497 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.497 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.497 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.498 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_revoke_eviction": 0, 2024-03-04T12:36:23.498 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_acquisition_throttle": 0, 2024-03-04T12:36:23.498 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getvxattr_latency": { 2024-03-04T12:36:23.498 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.498 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.498 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.498 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:23.498 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.499 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sessions": { 2024-03-04T12:36:23.499 INFO:teuthology.orchestra.run.smithi063.stdout: "session_count": 2, 2024-03-04T12:36:23.499 INFO:teuthology.orchestra.run.smithi063.stdout: "session_add": 3, 2024-03-04T12:36:23.499 INFO:teuthology.orchestra.run.smithi063.stdout: "session_remove": 1, 2024-03-04T12:36:23.499 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_open": 2, 2024-03-04T12:36:23.499 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_stale": 0, 2024-03-04T12:36:23.499 INFO:teuthology.orchestra.run.smithi063.stdout: "total_load": 0, 2024-03-04T12:36:23.499 INFO:teuthology.orchestra.run.smithi063.stdout: "average_load": 0, 2024-03-04T12:36:23.499 INFO:teuthology.orchestra.run.smithi063.stdout: "avg_session_uptime": 569, 2024-03-04T12:36:23.500 INFO:teuthology.orchestra.run.smithi063.stdout: "mdthresh_evicted": 0 2024-03-04T12:36:23.500 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.500 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-03-04T12:36:23.500 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-03-04T12:36:23.500 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-03-04T12:36:23.500 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-03-04T12:36:23.500 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-03-04T12:36:23.500 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-03-04T12:36:23.501 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-03-04T12:36:23.501 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-04T12:36:23.501 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-03-04T12:36:23.501 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-04T12:36:23.501 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-03-04T12:36:23.501 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-03-04T12:36:23.501 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-03-04T12:36:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-04T12:36:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-03-04T12:36:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-03-04T12:36:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-03-04T12:36:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-03-04T12:36:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-03-04T12:36:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-04T12:36:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-03-04T12:36:23.502 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-04T12:36:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-03-04T12:36:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-03-04T12:36:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-03-04T12:36:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-03-04T12:36:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-03-04T12:36:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-04T12:36:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-03-04T12:36:23.503 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-03-04T12:36:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-03-04T12:36:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-03-04T12:36:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-03-04T12:36:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-03-04T12:36:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-03-04T12:36:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-03-04T12:36:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-03-04T12:36:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 232224, 2024-03-04T12:36:23.504 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 88, 2024-03-04T12:36:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-03-04T12:36:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-03-04T12:36:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-03-04T12:36:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-03-04T12:36:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-03-04T12:36:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-03-04T12:36:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-03-04T12:36:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-03-04T12:36:23.505 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 9736, 2024-03-04T12:36:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 203, 2024-03-04T12:36:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-03-04T12:36:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-03-04T12:36:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-03-04T12:36:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-03-04T12:36:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 87626, 2024-03-04T12:36:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 419, 2024-03-04T12:36:23.506 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-03-04T12:36:23.507 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-03-04T12:36:23.507 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-03-04T12:36:23.507 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-03-04T12:36:23.507 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.507 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter": { 2024-03-04T12:36:23.507 INFO:teuthology.orchestra.run.smithi063.stdout: "op_active": 0, 2024-03-04T12:36:23.507 INFO:teuthology.orchestra.run.smithi063.stdout: "op_laggy": 0, 2024-03-04T12:36:23.507 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send": 42, 2024-03-04T12:36:23.508 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send_bytes": 12211, 2024-03-04T12:36:23.508 INFO:teuthology.orchestra.run.smithi063.stdout: "op_resend": 32, 2024-03-04T12:36:23.508 INFO:teuthology.orchestra.run.smithi063.stdout: "op_reply": 42, 2024-03-04T12:36:23.508 INFO:teuthology.orchestra.run.smithi063.stdout: "op_latency": { 2024-03-04T12:36:23.508 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 42, 2024-03-04T12:36:23.508 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.230997370, 2024-03-04T12:36:23.508 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.005499937 2024-03-04T12:36:23.508 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.509 INFO:teuthology.orchestra.run.smithi063.stdout: "op_inflight": 0, 2024-03-04T12:36:23.509 INFO:teuthology.orchestra.run.smithi063.stdout: "oplen_avg": { 2024-03-04T12:36:23.509 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 42, 2024-03-04T12:36:23.509 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 64 2024-03-04T12:36:23.509 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.509 INFO:teuthology.orchestra.run.smithi063.stdout: "op": 42, 2024-03-04T12:36:23.509 INFO:teuthology.orchestra.run.smithi063.stdout: "op_r": 0, 2024-03-04T12:36:23.509 INFO:teuthology.orchestra.run.smithi063.stdout: "op_w": 42, 2024-03-04T12:36:23.509 INFO:teuthology.orchestra.run.smithi063.stdout: "op_rmw": 0, 2024-03-04T12:36:23.510 INFO:teuthology.orchestra.run.smithi063.stdout: "op_pg": 0, 2024-03-04T12:36:23.510 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_stat": 0, 2024-03-04T12:36:23.510 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_create": 10, 2024-03-04T12:36:23.510 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_read": 0, 2024-03-04T12:36:23.510 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_write": 5, 2024-03-04T12:36:23.510 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writefull": 8, 2024-03-04T12:36:23.510 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writesame": 0, 2024-03-04T12:36:23.510 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_append": 0, 2024-03-04T12:36:23.511 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_zero": 0, 2024-03-04T12:36:23.511 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_truncate": 0, 2024-03-04T12:36:23.511 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_delete": 6, 2024-03-04T12:36:23.511 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_mapext": 0, 2024-03-04T12:36:23.511 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_sparse_read": 0, 2024-03-04T12:36:23.511 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_clonerange": 0, 2024-03-04T12:36:23.511 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_getxattr": 0, 2024-03-04T12:36:23.511 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_setxattr": 20, 2024-03-04T12:36:23.512 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_cmpxattr": 0, 2024-03-04T12:36:23.512 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_rmxattr": 0, 2024-03-04T12:36:23.512 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_resetxattrs": 0, 2024-03-04T12:36:23.512 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_call": 0, 2024-03-04T12:36:23.512 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_watch": 0, 2024-03-04T12:36:23.512 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_notify": 0, 2024-03-04T12:36:23.512 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_src_cmpxattr": 0, 2024-03-04T12:36:23.512 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls": 0, 2024-03-04T12:36:23.513 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls_filter": 0, 2024-03-04T12:36:23.513 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_other": 0, 2024-03-04T12:36:23.513 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_active": 0, 2024-03-04T12:36:23.513 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_send": 0, 2024-03-04T12:36:23.513 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_resend": 0, 2024-03-04T12:36:23.513 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_ping": 0, 2024-03-04T12:36:23.513 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_active": 0, 2024-03-04T12:36:23.513 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_send": 0, 2024-03-04T12:36:23.514 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_resend": 0, 2024-03-04T12:36:23.514 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_active": 0, 2024-03-04T12:36:23.514 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_send": 0, 2024-03-04T12:36:23.514 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_resend": 0, 2024-03-04T12:36:23.514 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_active": 0, 2024-03-04T12:36:23.514 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_send": 0, 2024-03-04T12:36:23.514 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_resend": 0, 2024-03-04T12:36:23.514 INFO:teuthology.orchestra.run.smithi063.stdout: "command_active": 0, 2024-03-04T12:36:23.514 INFO:teuthology.orchestra.run.smithi063.stdout: "command_send": 0, 2024-03-04T12:36:23.515 INFO:teuthology.orchestra.run.smithi063.stdout: "command_resend": 0, 2024-03-04T12:36:23.515 INFO:teuthology.orchestra.run.smithi063.stdout: "map_epoch": 105, 2024-03-04T12:36:23.515 INFO:teuthology.orchestra.run.smithi063.stdout: "map_full": 0, 2024-03-04T12:36:23.515 INFO:teuthology.orchestra.run.smithi063.stdout: "map_inc": 16, 2024-03-04T12:36:23.515 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_sessions": 8, 2024-03-04T12:36:23.515 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_open": 8, 2024-03-04T12:36:23.515 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_close": 0, 2024-03-04T12:36:23.516 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_laggy": 0, 2024-03-04T12:36:23.516 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_wr": 14, 2024-03-04T12:36:23.516 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_rd": 0, 2024-03-04T12:36:23.516 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_del": 1 2024-03-04T12:36:23.516 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.516 INFO:teuthology.orchestra.run.smithi063.stdout: "oft": { 2024-03-04T12:36:23.517 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_objs": 1, 2024-03-04T12:36:23.517 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_kv_pairs": 0, 2024-03-04T12:36:23.517 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_updates": 0, 2024-03-04T12:36:23.517 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_removes": 0 2024-03-04T12:36:23.517 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.517 INFO:teuthology.orchestra.run.smithi063.stdout: "osd-slow-ops": { 2024-03-04T12:36:23.517 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_ops_count": 0 2024-03-04T12:36:23.518 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.518 INFO:teuthology.orchestra.run.smithi063.stdout: "purge_queue": { 2024-03-04T12:36:23.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops": 0, 2024-03-04T12:36:23.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops_high_water": 0, 2024-03-04T12:36:23.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing": 0, 2024-03-04T12:36:23.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_high_water": 0, 2024-03-04T12:36:23.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed_ops": 0, 2024-03-04T12:36:23.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed": 0, 2024-03-04T12:36:23.518 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_item_in_journal": 0 2024-03-04T12:36:23.519 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.519 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-04T12:36:23.519 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:23.519 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-04T12:36:23.519 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:23.519 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 1106, 2024-03-04T12:36:23.519 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 1989303, 2024-03-04T12:36:23.519 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:23.520 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 1106, 2024-03-04T12:36:23.520 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-04T12:36:23.520 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-04T12:36:23.520 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 1106, 2024-03-04T12:36:23.520 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 1989303, 2024-03-04T12:36:23.520 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:23.520 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.520 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.521 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.521 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:23.521 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.521 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_bytes": { 2024-03-04T12:36:23.521 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:23.521 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-03-04T12:36:23.521 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:23.521 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-04T12:36:23.522 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-04T12:36:23.522 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:23.522 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:23.522 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 42, 2024-03-04T12:36:23.522 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 12211, 2024-03-04T12:36:23.522 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 42, 2024-03-04T12:36:23.522 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 12211, 2024-03-04T12:36:23.522 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:23.522 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.523 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.523 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.523 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:23.523 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.523 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_ops": { 2024-03-04T12:36:23.523 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:23.523 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 1024, 2024-03-04T12:36:23.524 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:23.524 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-04T12:36:23.524 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-04T12:36:23.524 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:23.524 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:23.524 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 42, 2024-03-04T12:36:23.524 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 42, 2024-03-04T12:36:23.524 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 42, 2024-03-04T12:36:23.525 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 42, 2024-03-04T12:36:23.525 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:23.525 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.525 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.525 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.525 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:23.525 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.525 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle": { 2024-03-04T12:36:23.525 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:23.526 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-04T12:36:23.526 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:23.526 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-03-04T12:36:23.526 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-03-04T12:36:23.526 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:23.526 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:23.526 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-04T12:36:23.526 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-04T12:36:23.526 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-03-04T12:36:23.527 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-03-04T12:36:23.527 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:23.527 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.527 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.527 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.527 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:23.527 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-03-04T12:36:23.527 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle-0x55c76a1c2640": { 2024-03-04T12:36:23.527 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-03-04T12:36:23.528 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-03-04T12:36:23.528 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-03-04T12:36:23.528 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 6, 2024-03-04T12:36:23.528 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 1819, 2024-03-04T12:36:23.528 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:23.528 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 6, 2024-03-04T12:36:23.528 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-03-04T12:36:23.528 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-03-04T12:36:23.528 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 5, 2024-03-04T12:36:23.529 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 1819, 2024-03-04T12:36:23.529 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-03-04T12:36:23.529 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-03-04T12:36:23.529 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-03-04T12:36:23.529 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-03-04T12:36:23.529 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:23.529 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-03-04T12:36:23.529 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-03-04T12:36:24.265 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-04T12:36:24.265 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-04T12:36:24.266 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=2 2024-03-04T12:36:24.266 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-04T12:36:24.266 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-04T12:36:24.266 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-03-04T12:36:24.266 DEBUG:tasks.check_counter:Getting stats from c 2024-03-04T12:36:24.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-04T12:36:24.514 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:36:25.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:25 smithi086 ceph-mon[36851]: pgmap v329: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:36:25.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:25 smithi063 ceph-mon[36717]: pgmap v329: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:36:25.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:25 smithi060 ceph-mon[32745]: pgmap v329: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-04T12:36:26.346 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-04T12:36:26.346 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-0": { 2024-03-04T12:36:26.346 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 1171, 2024-03-04T12:36:26.347 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 1173, 2024-03-04T12:36:26.347 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 4190482, 2024-03-04T12:36:26.347 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 372388, 2024-03-04T12:36:26.347 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 6, 2024-03-04T12:36:26.347 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 2, 2024-03-04T12:36:26.347 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.120858432, 2024-03-04T12:36:26.347 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.038412306, 2024-03-04T12:36:26.347 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.034055862, 2024-03-04T12:36:26.348 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.036501256, 2024-03-04T12:36:26.348 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:26.348 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 1171, 2024-03-04T12:36:26.348 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.047881864, 2024-03-04T12:36:26.348 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000040889 2024-03-04T12:36:26.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.348 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:26.348 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.348 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.349 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.349 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-04T12:36:26.349 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 0 2024-03-04T12:36:26.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.349 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-1": { 2024-03-04T12:36:26.349 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 186, 2024-03-04T12:36:26.349 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 173, 2024-03-04T12:36:26.350 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 409105, 2024-03-04T12:36:26.350 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 94506, 2024-03-04T12:36:26.350 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 4, 2024-03-04T12:36:26.350 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 3, 2024-03-04T12:36:26.350 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.028037843, 2024-03-04T12:36:26.350 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.010942978, 2024-03-04T12:36:26.350 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.013323106, 2024-03-04T12:36:26.350 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.003899459, 2024-03-04T12:36:26.350 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:26.351 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 173, 2024-03-04T12:36:26.351 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.011673253, 2024-03-04T12:36:26.351 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000067475 2024-03-04T12:36:26.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.351 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:26.351 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.351 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.352 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.352 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 406000, 2024-03-04T12:36:26.352 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 93168 2024-03-04T12:36:26.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.352 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-2": { 2024-03-04T12:36:26.352 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 554, 2024-03-04T12:36:26.352 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 667, 2024-03-04T12:36:26.353 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 136435, 2024-03-04T12:36:26.353 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 460111, 2024-03-04T12:36:26.353 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 4, 2024-03-04T12:36:26.353 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 3, 2024-03-04T12:36:26.353 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.061374856, 2024-03-04T12:36:26.353 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.024322997, 2024-03-04T12:36:26.353 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.012058188, 2024-03-04T12:36:26.353 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.018583360, 2024-03-04T12:36:26.354 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:26.354 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 665, 2024-03-04T12:36:26.354 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.028033904, 2024-03-04T12:36:26.354 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000042156 2024-03-04T12:36:26.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.354 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:26.354 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.354 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.354 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.355 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-04T12:36:26.355 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 284256 2024-03-04T12:36:26.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.355 INFO:teuthology.orchestra.run.smithi086.stdout: "cct": { 2024-03-04T12:36:26.355 INFO:teuthology.orchestra.run.smithi086.stdout: "total_workers": 1, 2024-03-04T12:36:26.355 INFO:teuthology.orchestra.run.smithi086.stdout: "unhealthy_workers": 0 2024-03-04T12:36:26.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.355 INFO:teuthology.orchestra.run.smithi086.stdout: "finisher-MDSRank": { 2024-03-04T12:36:26.356 INFO:teuthology.orchestra.run.smithi086.stdout: "queue_len": 0, 2024-03-04T12:36:26.356 INFO:teuthology.orchestra.run.smithi086.stdout: "complete_latency": { 2024-03-04T12:36:26.356 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 2835, 2024-03-04T12:36:26.356 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.089016721, 2024-03-04T12:36:26.356 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000031399 2024-03-04T12:36:26.356 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:26.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.356 INFO:teuthology.orchestra.run.smithi086.stdout: "finisher-PurgeQueue": { 2024-03-04T12:36:26.357 INFO:teuthology.orchestra.run.smithi086.stdout: "queue_len": 0, 2024-03-04T12:36:26.357 INFO:teuthology.orchestra.run.smithi086.stdout: "complete_latency": { 2024-03-04T12:36:26.357 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.357 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.357 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.357 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:26.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.357 INFO:teuthology.orchestra.run.smithi086.stdout: "mds": { 2024-03-04T12:36:26.358 INFO:teuthology.orchestra.run.smithi086.stdout: "request": 0, 2024-03-04T12:36:26.358 INFO:teuthology.orchestra.run.smithi086.stdout: "reply": 0, 2024-03-04T12:36:26.358 INFO:teuthology.orchestra.run.smithi086.stdout: "reply_latency": { 2024-03-04T12:36:26.358 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.358 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.358 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.358 INFO:teuthology.orchestra.run.smithi086.stdout: "slow_reply": 0, 2024-03-04T12:36:26.358 INFO:teuthology.orchestra.run.smithi086.stdout: "forward": 0, 2024-03-04T12:36:26.359 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_fetch_complete": 0, 2024-03-04T12:36:26.359 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_fetch_keys": 0, 2024-03-04T12:36:26.359 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_commit": 0, 2024-03-04T12:36:26.359 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_split": 0, 2024-03-04T12:36:26.359 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_merge": 0, 2024-03-04T12:36:26.359 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes": 39, 2024-03-04T12:36:26.359 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_top": 16, 2024-03-04T12:36:26.359 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_bottom": 23, 2024-03-04T12:36:26.359 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_pin_tail": 0, 2024-03-04T12:36:26.360 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_pinned": 17, 2024-03-04T12:36:26.360 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_expired": 0, 2024-03-04T12:36:26.360 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_with_caps": 0, 2024-03-04T12:36:26.360 INFO:teuthology.orchestra.run.smithi086.stdout: "caps": 0, 2024-03-04T12:36:26.360 INFO:teuthology.orchestra.run.smithi086.stdout: "subtrees": 4, 2024-03-04T12:36:26.360 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse": 0, 2024-03-04T12:36:26.360 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_hit": 0, 2024-03-04T12:36:26.360 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_forward": 0, 2024-03-04T12:36:26.361 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_discover": 0, 2024-03-04T12:36:26.361 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_dir_fetch": 0, 2024-03-04T12:36:26.361 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_remote_ino": 0, 2024-03-04T12:36:26.361 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_lock": 0, 2024-03-04T12:36:26.361 INFO:teuthology.orchestra.run.smithi086.stdout: "load_cent": 0, 2024-03-04T12:36:26.361 INFO:teuthology.orchestra.run.smithi086.stdout: "q": 0, 2024-03-04T12:36:26.361 INFO:teuthology.orchestra.run.smithi086.stdout: "exported": 0, 2024-03-04T12:36:26.361 INFO:teuthology.orchestra.run.smithi086.stdout: "exported_inodes": 0, 2024-03-04T12:36:26.361 INFO:teuthology.orchestra.run.smithi086.stdout: "imported": 0, 2024-03-04T12:36:26.362 INFO:teuthology.orchestra.run.smithi086.stdout: "imported_inodes": 0, 2024-03-04T12:36:26.362 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_dir_fetch": 0, 2024-03-04T12:36:26.362 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_backtrace_fetch": 0, 2024-03-04T12:36:26.362 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_peer_discover": 0, 2024-03-04T12:36:26.362 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rfiles": 3452, 2024-03-04T12:36:26.362 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rbytes": 26508703, 2024-03-04T12:36:26.362 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rsnaps": 6, 2024-03-04T12:36:26.362 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_backtrace_fetch": 0, 2024-03-04T12:36:26.362 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_set_tag": 0, 2024-03-04T12:36:26.363 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_backtrace_repaired": 0, 2024-03-04T12:36:26.363 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_inotable_repaired": 0, 2024-03-04T12:36:26.363 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dir_inodes": 0, 2024-03-04T12:36:26.363 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dir_base_inodes": 0, 2024-03-04T12:36:26.363 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dirfrag_rstats": 0, 2024-03-04T12:36:26.363 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_file_inodes": 0, 2024-03-04T12:36:26.363 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_inode_file_caps": 0, 2024-03-04T12:36:26.363 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_revoke": 0, 2024-03-04T12:36:26.364 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_grant": 0, 2024-03-04T12:36:26.364 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_trunc": 0, 2024-03-04T12:36:26.364 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-04T12:36:26.364 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-04T12:36:26.364 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_caps": 0, 2024-03-04T12:36:26.364 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_caps_dirty": 0, 2024-03-04T12:36:26.364 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_cap_release": 0, 2024-03-04T12:36:26.364 INFO:teuthology.orchestra.run.smithi086.stdout: "process_request_cap_release": 0 2024-03-04T12:36:26.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.365 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache": { 2024-03-04T12:36:26.365 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_update": 0, 2024-03-04T12:36:26.365 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_update_receipt": 0, 2024-03-04T12:36:26.365 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_try_discover": 0, 2024-03-04T12:36:26.365 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_send_discover": 0, 2024-03-04T12:36:26.365 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_handle_discover": 0, 2024-03-04T12:36:26.365 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays": 0, 2024-03-04T12:36:26.365 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays_delayed": 0, 2024-03-04T12:36:26.365 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays_enqueuing": 0, 2024-03-04T12:36:26.366 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_created": 0, 2024-03-04T12:36:26.366 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_enqueued": 0, 2024-03-04T12:36:26.366 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_reintegrated": 0, 2024-03-04T12:36:26.366 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_migrated": 0, 2024-03-04T12:36:26.366 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_processing": 0, 2024-03-04T12:36:26.366 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_enqueued": 0, 2024-03-04T12:36:26.366 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_prioritized": 0, 2024-03-04T12:36:26.366 INFO:teuthology.orchestra.run.smithi086.stdout: "recovery_started": 0, 2024-03-04T12:36:26.366 INFO:teuthology.orchestra.run.smithi086.stdout: "recovery_completed": 0, 2024-03-04T12:36:26.367 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_enqueue_scrub": 0, 2024-03-04T12:36:26.367 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_exportdir": 0, 2024-03-04T12:36:26.367 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_flush": 0, 2024-03-04T12:36:26.367 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_fragmentdir": 0, 2024-03-04T12:36:26.367 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_fragstats": 0, 2024-03-04T12:36:26.367 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_inodestats": 0 2024-03-04T12:36:26.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.367 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log": { 2024-03-04T12:36:26.367 INFO:teuthology.orchestra.run.smithi086.stdout: "evlrg": 0, 2024-03-04T12:36:26.368 INFO:teuthology.orchestra.run.smithi086.stdout: "evadd": 0, 2024-03-04T12:36:26.368 INFO:teuthology.orchestra.run.smithi086.stdout: "evex": 0, 2024-03-04T12:36:26.368 INFO:teuthology.orchestra.run.smithi086.stdout: "evtrm": 0, 2024-03-04T12:36:26.368 INFO:teuthology.orchestra.run.smithi086.stdout: "ev": 1638, 2024-03-04T12:36:26.368 INFO:teuthology.orchestra.run.smithi086.stdout: "evexg": 0, 2024-03-04T12:36:26.368 INFO:teuthology.orchestra.run.smithi086.stdout: "evexd": 0, 2024-03-04T12:36:26.368 INFO:teuthology.orchestra.run.smithi086.stdout: "segadd": 0, 2024-03-04T12:36:26.368 INFO:teuthology.orchestra.run.smithi086.stdout: "segex": 0, 2024-03-04T12:36:26.369 INFO:teuthology.orchestra.run.smithi086.stdout: "segtrm": 0, 2024-03-04T12:36:26.369 INFO:teuthology.orchestra.run.smithi086.stdout: "seg": 2, 2024-03-04T12:36:26.369 INFO:teuthology.orchestra.run.smithi086.stdout: "segmjr": 5, 2024-03-04T12:36:26.369 INFO:teuthology.orchestra.run.smithi086.stdout: "segexg": 0, 2024-03-04T12:36:26.369 INFO:teuthology.orchestra.run.smithi086.stdout: "segexd": 0, 2024-03-04T12:36:26.369 INFO:teuthology.orchestra.run.smithi086.stdout: "expos": 7945131, 2024-03-04T12:36:26.369 INFO:teuthology.orchestra.run.smithi086.stdout: "wrpos": 8144277, 2024-03-04T12:36:26.369 INFO:teuthology.orchestra.run.smithi086.stdout: "rdpos": 8144141, 2024-03-04T12:36:26.369 INFO:teuthology.orchestra.run.smithi086.stdout: "jlat": { 2024-03-04T12:36:26.370 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.370 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.370 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.370 INFO:teuthology.orchestra.run.smithi086.stdout: "replayed": 1638 2024-03-04T12:36:26.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.370 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_mem": { 2024-03-04T12:36:26.370 INFO:teuthology.orchestra.run.smithi086.stdout: "ino": 21, 2024-03-04T12:36:26.370 INFO:teuthology.orchestra.run.smithi086.stdout: "ino+": 1212, 2024-03-04T12:36:26.371 INFO:teuthology.orchestra.run.smithi086.stdout: "ino-": 1191, 2024-03-04T12:36:26.371 INFO:teuthology.orchestra.run.smithi086.stdout: "dir": 13, 2024-03-04T12:36:26.371 INFO:teuthology.orchestra.run.smithi086.stdout: "dir+": 75, 2024-03-04T12:36:26.371 INFO:teuthology.orchestra.run.smithi086.stdout: "dir-": 62, 2024-03-04T12:36:26.371 INFO:teuthology.orchestra.run.smithi086.stdout: "dn": 39, 2024-03-04T12:36:26.371 INFO:teuthology.orchestra.run.smithi086.stdout: "dn+": 1223, 2024-03-04T12:36:26.371 INFO:teuthology.orchestra.run.smithi086.stdout: "dn-": 1184, 2024-03-04T12:36:26.371 INFO:teuthology.orchestra.run.smithi086.stdout: "cap": 0, 2024-03-04T12:36:26.371 INFO:teuthology.orchestra.run.smithi086.stdout: "cap+": 0, 2024-03-04T12:36:26.371 INFO:teuthology.orchestra.run.smithi086.stdout: "cap-": 0, 2024-03-04T12:36:26.372 INFO:teuthology.orchestra.run.smithi086.stdout: "rss": 52752, 2024-03-04T12:36:26.372 INFO:teuthology.orchestra.run.smithi086.stdout: "heap": 215420 2024-03-04T12:36:26.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.372 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_server": { 2024-03-04T12:36:26.372 INFO:teuthology.orchestra.run.smithi086.stdout: "dispatch_client_request": 0, 2024-03-04T12:36:26.372 INFO:teuthology.orchestra.run.smithi086.stdout: "dispatch_server_request": 0, 2024-03-04T12:36:26.372 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_request": 0, 2024-03-04T12:36:26.372 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_session": 0, 2024-03-04T12:36:26.373 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_peer_request": 0, 2024-03-04T12:36:26.373 INFO:teuthology.orchestra.run.smithi086.stdout: "req_create_latency": { 2024-03-04T12:36:26.373 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.373 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.373 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.373 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getattr_latency": { 2024-03-04T12:36:26.374 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.374 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.374 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.374 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getfilelock_latency": { 2024-03-04T12:36:26.374 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.374 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.374 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.375 INFO:teuthology.orchestra.run.smithi086.stdout: "req_link_latency": { 2024-03-04T12:36:26.375 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.375 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.375 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.375 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookup_latency": { 2024-03-04T12:36:26.375 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.375 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.375 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.376 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookuphash_latency": { 2024-03-04T12:36:26.376 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.376 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.376 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.376 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupino_latency": { 2024-03-04T12:36:26.376 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.376 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.377 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.377 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupname_latency": { 2024-03-04T12:36:26.377 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.377 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.377 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.377 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupparent_latency": { 2024-03-04T12:36:26.377 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.378 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.378 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.378 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupsnap_latency": { 2024-03-04T12:36:26.378 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.378 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.378 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.378 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lssnap_latency": { 2024-03-04T12:36:26.379 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.379 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.379 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.379 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mkdir_latency": { 2024-03-04T12:36:26.379 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.379 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.380 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.380 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mknod_latency": { 2024-03-04T12:36:26.381 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.381 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.381 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.381 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mksnap_latency": { 2024-03-04T12:36:26.381 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.381 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.381 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.382 INFO:teuthology.orchestra.run.smithi086.stdout: "req_open_latency": { 2024-03-04T12:36:26.382 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.382 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.382 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.382 INFO:teuthology.orchestra.run.smithi086.stdout: "req_readdir_latency": { 2024-03-04T12:36:26.382 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.383 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.383 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.383 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rename_latency": { 2024-03-04T12:36:26.383 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.383 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.383 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.384 INFO:teuthology.orchestra.run.smithi086.stdout: "req_renamesnap_latency": { 2024-03-04T12:36:26.384 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.384 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.384 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.384 INFO:teuthology.orchestra.run.smithi086.stdout: "req_snapdiff_latency": { 2024-03-04T12:36:26.384 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.384 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.384 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.385 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmdir_latency": { 2024-03-04T12:36:26.385 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.385 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.385 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.385 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmsnap_latency": { 2024-03-04T12:36:26.385 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.385 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.386 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.386 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmxattr_latency": { 2024-03-04T12:36:26.386 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.386 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.386 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.386 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setattr_latency": { 2024-03-04T12:36:26.386 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.386 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.387 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.387 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setdirlayout_latency": { 2024-03-04T12:36:26.387 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.388 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.388 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.389 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setfilelock_latency": { 2024-03-04T12:36:26.389 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.389 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.389 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.389 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setlayout_latency": { 2024-03-04T12:36:26.389 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.389 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.390 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.390 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setxattr_latency": { 2024-03-04T12:36:26.390 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.390 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.391 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.391 INFO:teuthology.orchestra.run.smithi086.stdout: "req_symlink_latency": { 2024-03-04T12:36:26.391 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.392 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.392 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.392 INFO:teuthology.orchestra.run.smithi086.stdout: "req_unlink_latency": { 2024-03-04T12:36:26.392 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.392 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.392 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.393 INFO:teuthology.orchestra.run.smithi086.stdout: "cap_revoke_eviction": 0, 2024-03-04T12:36:26.393 INFO:teuthology.orchestra.run.smithi086.stdout: "cap_acquisition_throttle": 0, 2024-03-04T12:36:26.393 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getvxattr_latency": { 2024-03-04T12:36:26.393 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.393 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.393 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.393 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:26.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.394 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_sessions": { 2024-03-04T12:36:26.394 INFO:teuthology.orchestra.run.smithi086.stdout: "session_count": 2, 2024-03-04T12:36:26.394 INFO:teuthology.orchestra.run.smithi086.stdout: "session_add": 4, 2024-03-04T12:36:26.394 INFO:teuthology.orchestra.run.smithi086.stdout: "session_remove": 2, 2024-03-04T12:36:26.394 INFO:teuthology.orchestra.run.smithi086.stdout: "sessions_open": 0, 2024-03-04T12:36:26.394 INFO:teuthology.orchestra.run.smithi086.stdout: "sessions_stale": 0, 2024-03-04T12:36:26.395 INFO:teuthology.orchestra.run.smithi086.stdout: "total_load": 0, 2024-03-04T12:36:26.395 INFO:teuthology.orchestra.run.smithi086.stdout: "average_load": 0, 2024-03-04T12:36:26.395 INFO:teuthology.orchestra.run.smithi086.stdout: "avg_session_uptime": 3568, 2024-03-04T12:36:26.395 INFO:teuthology.orchestra.run.smithi086.stdout: "mdthresh_evicted": 0 2024-03-04T12:36:26.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.395 INFO:teuthology.orchestra.run.smithi086.stdout: "mempool": { 2024-03-04T12:36:26.395 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_bytes": 0, 2024-03-04T12:36:26.396 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_items": 0, 2024-03-04T12:36:26.396 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_bytes": 0, 2024-03-04T12:36:26.396 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_items": 0, 2024-03-04T12:36:26.396 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_bytes": 0, 2024-03-04T12:36:26.396 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_items": 0, 2024-03-04T12:36:26.396 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-04T12:36:26.397 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_items": 0, 2024-03-04T12:36:26.397 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-04T12:36:26.397 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_items": 0, 2024-03-04T12:36:26.397 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_bytes": 0, 2024-03-04T12:36:26.397 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_items": 0, 2024-03-04T12:36:26.397 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-04T12:36:26.397 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_items": 0, 2024-03-04T12:36:26.397 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_bytes": 0, 2024-03-04T12:36:26.398 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_items": 0, 2024-03-04T12:36:26.398 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_bytes": 0, 2024-03-04T12:36:26.398 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_items": 0, 2024-03-04T12:36:26.398 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-04T12:36:26.398 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_items": 0, 2024-03-04T12:36:26.398 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-04T12:36:26.398 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_items": 0, 2024-03-04T12:36:26.399 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_bytes": 0, 2024-03-04T12:36:26.399 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_items": 0, 2024-03-04T12:36:26.399 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_bytes": 0, 2024-03-04T12:36:26.399 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_items": 0, 2024-03-04T12:36:26.399 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-04T12:36:26.399 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_items": 0, 2024-03-04T12:36:26.399 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_bytes": 0, 2024-03-04T12:36:26.399 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_items": 0, 2024-03-04T12:36:26.400 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_bytes": 0, 2024-03-04T12:36:26.400 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_items": 0, 2024-03-04T12:36:26.400 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_bytes": 0, 2024-03-04T12:36:26.400 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_items": 0, 2024-03-04T12:36:26.400 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_bytes": 0, 2024-03-04T12:36:26.400 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_items": 0, 2024-03-04T12:36:26.400 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_bytes": 190123, 2024-03-04T12:36:26.400 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_items": 70, 2024-03-04T12:36:26.401 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_bytes": 0, 2024-03-04T12:36:26.401 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_items": 0, 2024-03-04T12:36:26.401 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bytes": 0, 2024-03-04T12:36:26.401 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_items": 0, 2024-03-04T12:36:26.401 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_bytes": 0, 2024-03-04T12:36:26.401 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_items": 0, 2024-03-04T12:36:26.402 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_bytes": 0, 2024-03-04T12:36:26.402 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_items": 0, 2024-03-04T12:36:26.402 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_bytes": 9736, 2024-03-04T12:36:26.402 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_items": 203, 2024-03-04T12:36:26.402 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_bytes": 0, 2024-03-04T12:36:26.402 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_items": 0, 2024-03-04T12:36:26.402 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_bytes": 0, 2024-03-04T12:36:26.403 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_items": 0, 2024-03-04T12:36:26.403 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_bytes": 112369, 2024-03-04T12:36:26.403 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_items": 774, 2024-03-04T12:36:26.403 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_bytes": 0, 2024-03-04T12:36:26.403 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_items": 0, 2024-03-04T12:36:26.403 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_bytes": 0, 2024-03-04T12:36:26.403 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_items": 0 2024-03-04T12:36:26.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.404 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter": { 2024-03-04T12:36:26.404 INFO:teuthology.orchestra.run.smithi086.stdout: "op_active": 0, 2024-03-04T12:36:26.404 INFO:teuthology.orchestra.run.smithi086.stdout: "op_laggy": 0, 2024-03-04T12:36:26.404 INFO:teuthology.orchestra.run.smithi086.stdout: "op_send": 1728, 2024-03-04T12:36:26.404 INFO:teuthology.orchestra.run.smithi086.stdout: "op_send_bytes": 26, 2024-03-04T12:36:26.404 INFO:teuthology.orchestra.run.smithi086.stdout: "op_resend": 4, 2024-03-04T12:36:26.404 INFO:teuthology.orchestra.run.smithi086.stdout: "op_reply": 1728, 2024-03-04T12:36:26.405 INFO:teuthology.orchestra.run.smithi086.stdout: "op_latency": { 2024-03-04T12:36:26.405 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 1728, 2024-03-04T12:36:26.405 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.993977837, 2024-03-04T12:36:26.405 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000575218 2024-03-04T12:36:26.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.405 INFO:teuthology.orchestra.run.smithi086.stdout: "op_inflight": 0, 2024-03-04T12:36:26.405 INFO:teuthology.orchestra.run.smithi086.stdout: "oplen_avg": { 2024-03-04T12:36:26.406 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 1728, 2024-03-04T12:36:26.406 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 1729 2024-03-04T12:36:26.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.406 INFO:teuthology.orchestra.run.smithi086.stdout: "op": 1728, 2024-03-04T12:36:26.406 INFO:teuthology.orchestra.run.smithi086.stdout: "op_r": 1728, 2024-03-04T12:36:26.406 INFO:teuthology.orchestra.run.smithi086.stdout: "op_w": 0, 2024-03-04T12:36:26.406 INFO:teuthology.orchestra.run.smithi086.stdout: "op_rmw": 0, 2024-03-04T12:36:26.406 INFO:teuthology.orchestra.run.smithi086.stdout: "op_pg": 0, 2024-03-04T12:36:26.407 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_stat": 1104, 2024-03-04T12:36:26.407 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_create": 0, 2024-03-04T12:36:26.407 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_read": 621, 2024-03-04T12:36:26.407 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_write": 0, 2024-03-04T12:36:26.407 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_writefull": 0, 2024-03-04T12:36:26.407 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_writesame": 0, 2024-03-04T12:36:26.407 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_append": 0, 2024-03-04T12:36:26.408 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_zero": 0, 2024-03-04T12:36:26.408 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_truncate": 0, 2024-03-04T12:36:26.408 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_delete": 0, 2024-03-04T12:36:26.408 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_mapext": 0, 2024-03-04T12:36:26.408 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_sparse_read": 0, 2024-03-04T12:36:26.408 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_clonerange": 0, 2024-03-04T12:36:26.408 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_getxattr": 2, 2024-03-04T12:36:26.409 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_setxattr": 0, 2024-03-04T12:36:26.409 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_cmpxattr": 0, 2024-03-04T12:36:26.409 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_rmxattr": 0, 2024-03-04T12:36:26.409 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_resetxattrs": 0, 2024-03-04T12:36:26.409 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_call": 0, 2024-03-04T12:36:26.409 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_watch": 0, 2024-03-04T12:36:26.409 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_notify": 0, 2024-03-04T12:36:26.410 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_src_cmpxattr": 0, 2024-03-04T12:36:26.410 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_pgls": 0, 2024-03-04T12:36:26.410 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_pgls_filter": 0, 2024-03-04T12:36:26.410 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_other": 0, 2024-03-04T12:36:26.410 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_active": 0, 2024-03-04T12:36:26.410 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_send": 0, 2024-03-04T12:36:26.410 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_resend": 0, 2024-03-04T12:36:26.410 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_ping": 0, 2024-03-04T12:36:26.411 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_active": 0, 2024-03-04T12:36:26.411 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_send": 0, 2024-03-04T12:36:26.411 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_resend": 0, 2024-03-04T12:36:26.411 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_active": 0, 2024-03-04T12:36:26.411 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_send": 0, 2024-03-04T12:36:26.411 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_resend": 0, 2024-03-04T12:36:26.411 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_active": 0, 2024-03-04T12:36:26.412 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_send": 0, 2024-03-04T12:36:26.412 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_resend": 0, 2024-03-04T12:36:26.412 INFO:teuthology.orchestra.run.smithi086.stdout: "command_active": 0, 2024-03-04T12:36:26.412 INFO:teuthology.orchestra.run.smithi086.stdout: "command_send": 0, 2024-03-04T12:36:26.412 INFO:teuthology.orchestra.run.smithi086.stdout: "command_resend": 0, 2024-03-04T12:36:26.412 INFO:teuthology.orchestra.run.smithi086.stdout: "map_epoch": 105, 2024-03-04T12:36:26.412 INFO:teuthology.orchestra.run.smithi086.stdout: "map_full": 0, 2024-03-04T12:36:26.413 INFO:teuthology.orchestra.run.smithi086.stdout: "map_inc": 10, 2024-03-04T12:36:26.413 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_sessions": 5, 2024-03-04T12:36:26.413 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_session_open": 5, 2024-03-04T12:36:26.413 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_session_close": 0, 2024-03-04T12:36:26.413 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_laggy": 0, 2024-03-04T12:36:26.413 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_wr": 0, 2024-03-04T12:36:26.413 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_rd": 2, 2024-03-04T12:36:26.413 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_del": 0 2024-03-04T12:36:26.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.414 INFO:teuthology.orchestra.run.smithi086.stdout: "oft": { 2024-03-04T12:36:26.414 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_objs": 0, 2024-03-04T12:36:26.414 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_kv_pairs": 0, 2024-03-04T12:36:26.414 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_updates": 0, 2024-03-04T12:36:26.414 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_removes": 0 2024-03-04T12:36:26.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.415 INFO:teuthology.orchestra.run.smithi086.stdout: "osd-slow-ops": { 2024-03-04T12:36:26.415 INFO:teuthology.orchestra.run.smithi086.stdout: "slow_ops_count": 0 2024-03-04T12:36:26.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.415 INFO:teuthology.orchestra.run.smithi086.stdout: "purge_queue": { 2024-03-04T12:36:26.415 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_ops": 0, 2024-03-04T12:36:26.415 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_ops_high_water": 0, 2024-03-04T12:36:26.415 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing": 0, 2024-03-04T12:36:26.415 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_high_water": 0, 2024-03-04T12:36:26.416 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executed_ops": 0, 2024-03-04T12:36:26.416 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executed": 0, 2024-03-04T12:36:26.416 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_item_in_journal": 0 2024-03-04T12:36:26.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.416 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-04T12:36:26.416 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:26.416 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-03-04T12:36:26.417 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:26.417 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 1907, 2024-03-04T12:36:26.417 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 4552321, 2024-03-04T12:36:26.417 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:26.417 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 1907, 2024-03-04T12:36:26.417 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-03-04T12:36:26.417 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-03-04T12:36:26.417 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 1907, 2024-03-04T12:36:26.418 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 4552321, 2024-03-04T12:36:26.418 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:26.418 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.418 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.418 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.418 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:26.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.418 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-objecter_bytes": { 2024-03-04T12:36:26.419 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:26.419 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-03-04T12:36:26.419 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:26.419 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:26.419 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:26.419 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:26.419 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:26.420 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 1728, 2024-03-04T12:36:26.420 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 3842654, 2024-03-04T12:36:26.420 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 1728, 2024-03-04T12:36:26.420 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 3842654, 2024-03-04T12:36:26.420 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:26.420 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.421 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.421 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.421 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:26.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.421 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-objecter_ops": { 2024-03-04T12:36:26.421 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:26.421 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 1024, 2024-03-04T12:36:26.421 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:26.422 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:26.422 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:26.422 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:26.422 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:26.422 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 1728, 2024-03-04T12:36:26.422 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 1728, 2024-03-04T12:36:26.422 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 1728, 2024-03-04T12:36:26.423 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 1728, 2024-03-04T12:36:26.423 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:26.423 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.423 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.423 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.423 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:26.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.423 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-write_buf_throttle": { 2024-03-04T12:36:26.424 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:26.424 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 3758096384, 2024-03-04T12:36:26.424 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:26.424 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:26.424 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:26.424 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:26.424 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:26.425 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-03-04T12:36:26.425 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-03-04T12:36:26.425 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 0, 2024-03-04T12:36:26.425 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 0, 2024-03-04T12:36:26.425 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:26.425 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.425 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.426 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.426 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:26.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:26.426 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-write_buf_throttle-0x55cba04c6960": { 2024-03-04T12:36:26.426 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:26.426 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 3758096384, 2024-03-04T12:36:26.426 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:26.426 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:26.427 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:26.427 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:26.427 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:26.427 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-03-04T12:36:26.427 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-03-04T12:36:26.427 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 0, 2024-03-04T12:36:26.427 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 0, 2024-03-04T12:36:26.427 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:26.428 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:26.428 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:26.428 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:26.428 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:26.428 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:26.428 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-04T12:36:27.084 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-04T12:36:27.084 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-04T12:36:27.084 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-04T12:36:27.085 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-04T12:36:27.085 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-04T12:36:27.085 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-04T12:36:27.086 DEBUG:tasks.check_counter:Getting stats from f 2024-03-04T12:36:27.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-04T12:36:27.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:27 smithi063 ceph-mon[36717]: pgmap v330: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:36:27.341 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:36:27.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:27 smithi086 ceph-mon[36851]: pgmap v330: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:36:27.406 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:27 smithi060 ceph-mon[32745]: pgmap v330: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-04T12:36:29.079 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-04T12:36:29.079 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-0": { 2024-03-04T12:36:29.079 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 1753, 2024-03-04T12:36:29.080 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 1755, 2024-03-04T12:36:29.080 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 24024384, 2024-03-04T12:36:29.080 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 557461, 2024-03-04T12:36:29.080 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 7, 2024-03-04T12:36:29.080 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 3, 2024-03-04T12:36:29.080 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.187192815, 2024-03-04T12:36:29.080 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.055280663, 2024-03-04T12:36:29.080 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.057771590, 2024-03-04T12:36:29.081 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.057988419, 2024-03-04T12:36:29.081 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:29.081 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 1753, 2024-03-04T12:36:29.081 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.068813873, 2024-03-04T12:36:29.081 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000039254 2024-03-04T12:36:29.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.081 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:29.082 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.082 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.082 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.082 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-04T12:36:29.082 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 0 2024-03-04T12:36:29.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.082 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-1": { 2024-03-04T12:36:29.083 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 825, 2024-03-04T12:36:29.083 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 794, 2024-03-04T12:36:29.083 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 10365398, 2024-03-04T12:36:29.083 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 311184, 2024-03-04T12:36:29.083 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 4, 2024-03-04T12:36:29.083 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 3, 2024-03-04T12:36:29.083 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.117572365, 2024-03-04T12:36:29.083 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.031355042, 2024-03-04T12:36:29.084 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.061663043, 2024-03-04T12:36:29.084 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.029329878, 2024-03-04T12:36:29.084 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:29.084 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 794, 2024-03-04T12:36:29.084 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.039952822, 2024-03-04T12:36:29.084 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000050318 2024-03-04T12:36:29.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.084 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:29.085 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.085 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.085 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.085 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 1839456, 2024-03-04T12:36:29.085 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 135904 2024-03-04T12:36:29.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.085 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-2": { 2024-03-04T12:36:29.085 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 13, 2024-03-04T12:36:29.086 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 179, 2024-03-04T12:36:29.086 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 4197464, 2024-03-04T12:36:29.086 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 449626, 2024-03-04T12:36:29.086 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 7, 2024-03-04T12:36:29.086 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 5, 2024-03-04T12:36:29.086 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.015195196, 2024-03-04T12:36:29.086 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.007015151, 2024-03-04T12:36:29.086 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.005827968, 2024-03-04T12:36:29.087 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.000256539, 2024-03-04T12:36:29.087 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:29.087 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 176, 2024-03-04T12:36:29.087 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.015077746, 2024-03-04T12:36:29.087 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000085669 2024-03-04T12:36:29.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.087 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:29.087 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.088 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.088 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.088 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-04T12:36:29.088 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 446128 2024-03-04T12:36:29.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.088 INFO:teuthology.orchestra.run.smithi086.stdout: "cct": { 2024-03-04T12:36:29.088 INFO:teuthology.orchestra.run.smithi086.stdout: "total_workers": 1, 2024-03-04T12:36:29.089 INFO:teuthology.orchestra.run.smithi086.stdout: "unhealthy_workers": 0 2024-03-04T12:36:29.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.089 INFO:teuthology.orchestra.run.smithi086.stdout: "finisher-MDSRank": { 2024-03-04T12:36:29.089 INFO:teuthology.orchestra.run.smithi086.stdout: "queue_len": 0, 2024-03-04T12:36:29.089 INFO:teuthology.orchestra.run.smithi086.stdout: "complete_latency": { 2024-03-04T12:36:29.089 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 3788, 2024-03-04T12:36:29.089 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.117144141, 2024-03-04T12:36:29.089 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000030925 2024-03-04T12:36:29.090 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:29.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.090 INFO:teuthology.orchestra.run.smithi086.stdout: "finisher-PurgeQueue": { 2024-03-04T12:36:29.090 INFO:teuthology.orchestra.run.smithi086.stdout: "queue_len": 0, 2024-03-04T12:36:29.090 INFO:teuthology.orchestra.run.smithi086.stdout: "complete_latency": { 2024-03-04T12:36:29.090 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.090 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.090 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.090 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:29.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.091 INFO:teuthology.orchestra.run.smithi086.stdout: "mds": { 2024-03-04T12:36:29.091 INFO:teuthology.orchestra.run.smithi086.stdout: "request": 0, 2024-03-04T12:36:29.091 INFO:teuthology.orchestra.run.smithi086.stdout: "reply": 0, 2024-03-04T12:36:29.091 INFO:teuthology.orchestra.run.smithi086.stdout: "reply_latency": { 2024-03-04T12:36:29.091 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.091 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.091 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.092 INFO:teuthology.orchestra.run.smithi086.stdout: "slow_reply": 0, 2024-03-04T12:36:29.092 INFO:teuthology.orchestra.run.smithi086.stdout: "forward": 0, 2024-03-04T12:36:29.092 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_fetch_complete": 0, 2024-03-04T12:36:29.092 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_fetch_keys": 0, 2024-03-04T12:36:29.092 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_commit": 0, 2024-03-04T12:36:29.092 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_split": 0, 2024-03-04T12:36:29.092 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_merge": 0, 2024-03-04T12:36:29.093 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes": 6585, 2024-03-04T12:36:29.093 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_top": 7, 2024-03-04T12:36:29.093 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_bottom": 6578, 2024-03-04T12:36:29.093 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_pin_tail": 0, 2024-03-04T12:36:29.093 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_pinned": 6575, 2024-03-04T12:36:29.093 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_expired": 0, 2024-03-04T12:36:29.093 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_with_caps": 0, 2024-03-04T12:36:29.093 INFO:teuthology.orchestra.run.smithi086.stdout: "caps": 0, 2024-03-04T12:36:29.093 INFO:teuthology.orchestra.run.smithi086.stdout: "subtrees": 4, 2024-03-04T12:36:29.094 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse": 0, 2024-03-04T12:36:29.094 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_hit": 0, 2024-03-04T12:36:29.094 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_forward": 0, 2024-03-04T12:36:29.094 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_discover": 0, 2024-03-04T12:36:29.094 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_dir_fetch": 0, 2024-03-04T12:36:29.095 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_remote_ino": 0, 2024-03-04T12:36:29.095 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_lock": 0, 2024-03-04T12:36:29.095 INFO:teuthology.orchestra.run.smithi086.stdout: "load_cent": 0, 2024-03-04T12:36:29.095 INFO:teuthology.orchestra.run.smithi086.stdout: "q": 0, 2024-03-04T12:36:29.095 INFO:teuthology.orchestra.run.smithi086.stdout: "exported": 0, 2024-03-04T12:36:29.095 INFO:teuthology.orchestra.run.smithi086.stdout: "exported_inodes": 0, 2024-03-04T12:36:29.095 INFO:teuthology.orchestra.run.smithi086.stdout: "imported": 0, 2024-03-04T12:36:29.095 INFO:teuthology.orchestra.run.smithi086.stdout: "imported_inodes": 0, 2024-03-04T12:36:29.096 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_dir_fetch": 0, 2024-03-04T12:36:29.096 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_backtrace_fetch": 0, 2024-03-04T12:36:29.096 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_peer_discover": 0, 2024-03-04T12:36:29.096 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rfiles": 0, 2024-03-04T12:36:29.096 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rbytes": 0, 2024-03-04T12:36:29.096 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rsnaps": 0, 2024-03-04T12:36:29.096 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_backtrace_fetch": 0, 2024-03-04T12:36:29.096 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_set_tag": 0, 2024-03-04T12:36:29.096 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_backtrace_repaired": 0, 2024-03-04T12:36:29.097 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_inotable_repaired": 0, 2024-03-04T12:36:29.097 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dir_inodes": 0, 2024-03-04T12:36:29.097 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dir_base_inodes": 0, 2024-03-04T12:36:29.097 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dirfrag_rstats": 0, 2024-03-04T12:36:29.097 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_file_inodes": 0, 2024-03-04T12:36:29.097 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_inode_file_caps": 0, 2024-03-04T12:36:29.098 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_revoke": 0, 2024-03-04T12:36:29.098 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_grant": 0, 2024-03-04T12:36:29.098 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_trunc": 0, 2024-03-04T12:36:29.098 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-04T12:36:29.098 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-04T12:36:29.098 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_caps": 0, 2024-03-04T12:36:29.098 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_caps_dirty": 0, 2024-03-04T12:36:29.098 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_cap_release": 0, 2024-03-04T12:36:29.099 INFO:teuthology.orchestra.run.smithi086.stdout: "process_request_cap_release": 0 2024-03-04T12:36:29.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.099 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache": { 2024-03-04T12:36:29.099 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_update": 0, 2024-03-04T12:36:29.099 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_update_receipt": 0, 2024-03-04T12:36:29.099 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_try_discover": 0, 2024-03-04T12:36:29.099 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_send_discover": 0, 2024-03-04T12:36:29.099 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_handle_discover": 0, 2024-03-04T12:36:29.100 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays": 0, 2024-03-04T12:36:29.100 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays_delayed": 0, 2024-03-04T12:36:29.100 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays_enqueuing": 0, 2024-03-04T12:36:29.100 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_created": 0, 2024-03-04T12:36:29.100 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_enqueued": 0, 2024-03-04T12:36:29.100 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_reintegrated": 0, 2024-03-04T12:36:29.100 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_migrated": 0, 2024-03-04T12:36:29.100 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_processing": 0, 2024-03-04T12:36:29.101 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_enqueued": 0, 2024-03-04T12:36:29.101 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_prioritized": 0, 2024-03-04T12:36:29.101 INFO:teuthology.orchestra.run.smithi086.stdout: "recovery_started": 0, 2024-03-04T12:36:29.101 INFO:teuthology.orchestra.run.smithi086.stdout: "recovery_completed": 0, 2024-03-04T12:36:29.101 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_enqueue_scrub": 0, 2024-03-04T12:36:29.101 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_exportdir": 0, 2024-03-04T12:36:29.101 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_flush": 0, 2024-03-04T12:36:29.101 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_fragmentdir": 0, 2024-03-04T12:36:29.102 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_fragstats": 0, 2024-03-04T12:36:29.102 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_inodestats": 0 2024-03-04T12:36:29.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.102 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log": { 2024-03-04T12:36:29.102 INFO:teuthology.orchestra.run.smithi086.stdout: "evlrg": 0, 2024-03-04T12:36:29.102 INFO:teuthology.orchestra.run.smithi086.stdout: "evadd": 0, 2024-03-04T12:36:29.102 INFO:teuthology.orchestra.run.smithi086.stdout: "evex": 0, 2024-03-04T12:36:29.102 INFO:teuthology.orchestra.run.smithi086.stdout: "evtrm": 0, 2024-03-04T12:36:29.102 INFO:teuthology.orchestra.run.smithi086.stdout: "ev": 15412, 2024-03-04T12:36:29.103 INFO:teuthology.orchestra.run.smithi086.stdout: "evexg": 0, 2024-03-04T12:36:29.103 INFO:teuthology.orchestra.run.smithi086.stdout: "evexd": 0, 2024-03-04T12:36:29.103 INFO:teuthology.orchestra.run.smithi086.stdout: "segadd": 0, 2024-03-04T12:36:29.103 INFO:teuthology.orchestra.run.smithi086.stdout: "segex": 0, 2024-03-04T12:36:29.103 INFO:teuthology.orchestra.run.smithi086.stdout: "segtrm": 0, 2024-03-04T12:36:29.103 INFO:teuthology.orchestra.run.smithi086.stdout: "seg": 23, 2024-03-04T12:36:29.103 INFO:teuthology.orchestra.run.smithi086.stdout: "segmjr": 3, 2024-03-04T12:36:29.103 INFO:teuthology.orchestra.run.smithi086.stdout: "segexg": 0, 2024-03-04T12:36:29.104 INFO:teuthology.orchestra.run.smithi086.stdout: "segexd": 0, 2024-03-04T12:36:29.104 INFO:teuthology.orchestra.run.smithi086.stdout: "expos": 4194304, 2024-03-04T12:36:29.104 INFO:teuthology.orchestra.run.smithi086.stdout: "wrpos": 40281423, 2024-03-04T12:36:29.104 INFO:teuthology.orchestra.run.smithi086.stdout: "rdpos": 40280849, 2024-03-04T12:36:29.104 INFO:teuthology.orchestra.run.smithi086.stdout: "jlat": { 2024-03-04T12:36:29.104 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.104 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.104 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.105 INFO:teuthology.orchestra.run.smithi086.stdout: "replayed": 15412 2024-03-04T12:36:29.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.105 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_mem": { 2024-03-04T12:36:29.105 INFO:teuthology.orchestra.run.smithi086.stdout: "ino": 2992, 2024-03-04T12:36:29.105 INFO:teuthology.orchestra.run.smithi086.stdout: "ino+": 5574, 2024-03-04T12:36:29.105 INFO:teuthology.orchestra.run.smithi086.stdout: "ino-": 2582, 2024-03-04T12:36:29.105 INFO:teuthology.orchestra.run.smithi086.stdout: "dir": 185, 2024-03-04T12:36:29.105 INFO:teuthology.orchestra.run.smithi086.stdout: "dir+": 207, 2024-03-04T12:36:29.105 INFO:teuthology.orchestra.run.smithi086.stdout: "dir-": 22, 2024-03-04T12:36:29.106 INFO:teuthology.orchestra.run.smithi086.stdout: "dn": 6585, 2024-03-04T12:36:29.106 INFO:teuthology.orchestra.run.smithi086.stdout: "dn+": 6614, 2024-03-04T12:36:29.106 INFO:teuthology.orchestra.run.smithi086.stdout: "dn-": 29, 2024-03-04T12:36:29.106 INFO:teuthology.orchestra.run.smithi086.stdout: "cap": 0, 2024-03-04T12:36:29.106 INFO:teuthology.orchestra.run.smithi086.stdout: "cap+": 0, 2024-03-04T12:36:29.106 INFO:teuthology.orchestra.run.smithi086.stdout: "cap-": 0, 2024-03-04T12:36:29.106 INFO:teuthology.orchestra.run.smithi086.stdout: "rss": 72032, 2024-03-04T12:36:29.106 INFO:teuthology.orchestra.run.smithi086.stdout: "heap": 215420 2024-03-04T12:36:29.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.106 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_server": { 2024-03-04T12:36:29.107 INFO:teuthology.orchestra.run.smithi086.stdout: "dispatch_client_request": 0, 2024-03-04T12:36:29.107 INFO:teuthology.orchestra.run.smithi086.stdout: "dispatch_server_request": 0, 2024-03-04T12:36:29.107 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_request": 0, 2024-03-04T12:36:29.107 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_session": 0, 2024-03-04T12:36:29.107 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_peer_request": 0, 2024-03-04T12:36:29.107 INFO:teuthology.orchestra.run.smithi086.stdout: "req_create_latency": { 2024-03-04T12:36:29.107 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.107 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.107 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.108 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getattr_latency": { 2024-03-04T12:36:29.108 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.108 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.108 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.111 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getfilelock_latency": { 2024-03-04T12:36:29.111 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.111 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.111 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.112 INFO:teuthology.orchestra.run.smithi086.stdout: "req_link_latency": { 2024-03-04T12:36:29.112 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.112 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.112 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.112 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookup_latency": { 2024-03-04T12:36:29.112 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.113 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.113 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.113 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookuphash_latency": { 2024-03-04T12:36:29.113 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.113 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.113 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.113 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupino_latency": { 2024-03-04T12:36:29.114 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.114 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.114 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.114 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupname_latency": { 2024-03-04T12:36:29.114 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.114 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.114 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.115 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupparent_latency": { 2024-03-04T12:36:29.115 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.115 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.115 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.115 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupsnap_latency": { 2024-03-04T12:36:29.115 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.115 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.115 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.116 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lssnap_latency": { 2024-03-04T12:36:29.116 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.116 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.116 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.116 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mkdir_latency": { 2024-03-04T12:36:29.116 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.117 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.117 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.117 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mknod_latency": { 2024-03-04T12:36:29.118 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.118 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.118 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.118 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mksnap_latency": { 2024-03-04T12:36:29.118 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.118 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.118 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.119 INFO:teuthology.orchestra.run.smithi086.stdout: "req_open_latency": { 2024-03-04T12:36:29.119 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.119 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.119 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.119 INFO:teuthology.orchestra.run.smithi086.stdout: "req_readdir_latency": { 2024-03-04T12:36:29.119 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.119 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.120 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.120 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rename_latency": { 2024-03-04T12:36:29.120 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.120 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.120 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.120 INFO:teuthology.orchestra.run.smithi086.stdout: "req_renamesnap_latency": { 2024-03-04T12:36:29.121 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.121 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.121 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.121 INFO:teuthology.orchestra.run.smithi086.stdout: "req_snapdiff_latency": { 2024-03-04T12:36:29.121 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.121 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.121 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.122 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmdir_latency": { 2024-03-04T12:36:29.122 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.122 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.122 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.122 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmsnap_latency": { 2024-03-04T12:36:29.122 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.122 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.123 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.123 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmxattr_latency": { 2024-03-04T12:36:29.123 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.123 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.123 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.123 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setattr_latency": { 2024-03-04T12:36:29.124 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.124 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.124 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.124 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setdirlayout_latency": { 2024-03-04T12:36:29.124 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.124 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.124 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.125 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setfilelock_latency": { 2024-03-04T12:36:29.125 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.125 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.125 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.125 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setlayout_latency": { 2024-03-04T12:36:29.125 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.125 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.125 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.126 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setxattr_latency": { 2024-03-04T12:36:29.126 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.126 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.126 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.126 INFO:teuthology.orchestra.run.smithi086.stdout: "req_symlink_latency": { 2024-03-04T12:36:29.126 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.126 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.127 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.127 INFO:teuthology.orchestra.run.smithi086.stdout: "req_unlink_latency": { 2024-03-04T12:36:29.127 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.127 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.127 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.127 INFO:teuthology.orchestra.run.smithi086.stdout: "cap_revoke_eviction": 0, 2024-03-04T12:36:29.127 INFO:teuthology.orchestra.run.smithi086.stdout: "cap_acquisition_throttle": 0, 2024-03-04T12:36:29.128 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getvxattr_latency": { 2024-03-04T12:36:29.128 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.128 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.128 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.128 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:29.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.128 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_sessions": { 2024-03-04T12:36:29.128 INFO:teuthology.orchestra.run.smithi086.stdout: "session_count": 1, 2024-03-04T12:36:29.128 INFO:teuthology.orchestra.run.smithi086.stdout: "session_add": 4, 2024-03-04T12:36:29.129 INFO:teuthology.orchestra.run.smithi086.stdout: "session_remove": 3, 2024-03-04T12:36:29.129 INFO:teuthology.orchestra.run.smithi086.stdout: "sessions_open": 3, 2024-03-04T12:36:29.129 INFO:teuthology.orchestra.run.smithi086.stdout: "sessions_stale": 0, 2024-03-04T12:36:29.129 INFO:teuthology.orchestra.run.smithi086.stdout: "total_load": 0, 2024-03-04T12:36:29.129 INFO:teuthology.orchestra.run.smithi086.stdout: "average_load": 0, 2024-03-04T12:36:29.129 INFO:teuthology.orchestra.run.smithi086.stdout: "avg_session_uptime": 4918, 2024-03-04T12:36:29.129 INFO:teuthology.orchestra.run.smithi086.stdout: "mdthresh_evicted": 0 2024-03-04T12:36:29.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.129 INFO:teuthology.orchestra.run.smithi086.stdout: "mempool": { 2024-03-04T12:36:29.130 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_bytes": 0, 2024-03-04T12:36:29.130 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_items": 0, 2024-03-04T12:36:29.130 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_bytes": 0, 2024-03-04T12:36:29.130 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_items": 0, 2024-03-04T12:36:29.130 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_bytes": 0, 2024-03-04T12:36:29.130 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_items": 0, 2024-03-04T12:36:29.130 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-04T12:36:29.130 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_items": 0, 2024-03-04T12:36:29.130 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-04T12:36:29.131 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_items": 0, 2024-03-04T12:36:29.131 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_bytes": 0, 2024-03-04T12:36:29.131 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_items": 0, 2024-03-04T12:36:29.131 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-04T12:36:29.131 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_items": 0, 2024-03-04T12:36:29.131 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_bytes": 0, 2024-03-04T12:36:29.131 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_items": 0, 2024-03-04T12:36:29.131 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_bytes": 0, 2024-03-04T12:36:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_items": 0, 2024-03-04T12:36:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-04T12:36:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_items": 0, 2024-03-04T12:36:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-04T12:36:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_items": 0, 2024-03-04T12:36:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_bytes": 0, 2024-03-04T12:36:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_items": 0, 2024-03-04T12:36:29.132 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_bytes": 0, 2024-03-04T12:36:29.133 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_items": 0, 2024-03-04T12:36:29.133 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-04T12:36:29.133 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_items": 0, 2024-03-04T12:36:29.133 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_bytes": 0, 2024-03-04T12:36:29.133 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_items": 0, 2024-03-04T12:36:29.133 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_bytes": 0, 2024-03-04T12:36:29.133 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_items": 0, 2024-03-04T12:36:29.133 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_bytes": 0, 2024-03-04T12:36:29.133 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_items": 0, 2024-03-04T12:36:29.134 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_bytes": 0, 2024-03-04T12:36:29.134 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_items": 0, 2024-03-04T12:36:29.134 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_bytes": 316748, 2024-03-04T12:36:29.134 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_items": 3045, 2024-03-04T12:36:29.134 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_bytes": 0, 2024-03-04T12:36:29.134 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_items": 0, 2024-03-04T12:36:29.134 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bytes": 0, 2024-03-04T12:36:29.134 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_items": 0, 2024-03-04T12:36:29.134 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_bytes": 0, 2024-03-04T12:36:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_items": 0, 2024-03-04T12:36:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_bytes": 0, 2024-03-04T12:36:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_items": 0, 2024-03-04T12:36:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_bytes": 9736, 2024-03-04T12:36:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_items": 203, 2024-03-04T12:36:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_bytes": 0, 2024-03-04T12:36:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_items": 0, 2024-03-04T12:36:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_bytes": 0, 2024-03-04T12:36:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_items": 0, 2024-03-04T12:36:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_bytes": 11666050, 2024-03-04T12:36:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_items": 99166, 2024-03-04T12:36:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_bytes": 0, 2024-03-04T12:36:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_items": 0, 2024-03-04T12:36:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_bytes": 0, 2024-03-04T12:36:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_items": 0 2024-03-04T12:36:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter": { 2024-03-04T12:36:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "op_active": 0, 2024-03-04T12:36:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "op_laggy": 0, 2024-03-04T12:36:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "op_send": 2315, 2024-03-04T12:36:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "op_send_bytes": 21, 2024-03-04T12:36:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "op_resend": 3, 2024-03-04T12:36:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "op_reply": 2315, 2024-03-04T12:36:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "op_latency": { 2024-03-04T12:36:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 2315, 2024-03-04T12:36:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 2.146951840, 2024-03-04T12:36:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000927409 2024-03-04T12:36:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "op_inflight": 0, 2024-03-04T12:36:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "oplen_avg": { 2024-03-04T12:36:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 2315, 2024-03-04T12:36:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 2316 2024-03-04T12:36:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "op": 2315, 2024-03-04T12:36:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "op_r": 2315, 2024-03-04T12:36:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "op_w": 0, 2024-03-04T12:36:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "op_rmw": 0, 2024-03-04T12:36:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "op_pg": 0, 2024-03-04T12:36:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_stat": 1501, 2024-03-04T12:36:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_create": 0, 2024-03-04T12:36:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_read": 812, 2024-03-04T12:36:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_write": 0, 2024-03-04T12:36:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_writefull": 0, 2024-03-04T12:36:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_writesame": 0, 2024-03-04T12:36:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_append": 0, 2024-03-04T12:36:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_zero": 0, 2024-03-04T12:36:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_truncate": 0, 2024-03-04T12:36:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_delete": 0, 2024-03-04T12:36:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_mapext": 0, 2024-03-04T12:36:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_sparse_read": 0, 2024-03-04T12:36:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_clonerange": 0, 2024-03-04T12:36:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_getxattr": 1, 2024-03-04T12:36:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_setxattr": 0, 2024-03-04T12:36:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_cmpxattr": 0, 2024-03-04T12:36:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_rmxattr": 0, 2024-03-04T12:36:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_resetxattrs": 0, 2024-03-04T12:36:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_call": 0, 2024-03-04T12:36:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_watch": 0, 2024-03-04T12:36:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_notify": 0, 2024-03-04T12:36:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_src_cmpxattr": 0, 2024-03-04T12:36:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_pgls": 0, 2024-03-04T12:36:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_pgls_filter": 0, 2024-03-04T12:36:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_other": 0, 2024-03-04T12:36:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_active": 0, 2024-03-04T12:36:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_send": 0, 2024-03-04T12:36:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_resend": 0, 2024-03-04T12:36:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_ping": 0, 2024-03-04T12:36:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_active": 0, 2024-03-04T12:36:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_send": 0, 2024-03-04T12:36:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_resend": 0, 2024-03-04T12:36:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_active": 0, 2024-03-04T12:36:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_send": 0, 2024-03-04T12:36:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_resend": 0, 2024-03-04T12:36:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_active": 0, 2024-03-04T12:36:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_send": 0, 2024-03-04T12:36:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_resend": 0, 2024-03-04T12:36:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "command_active": 0, 2024-03-04T12:36:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "command_send": 0, 2024-03-04T12:36:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "command_resend": 0, 2024-03-04T12:36:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "map_epoch": 105, 2024-03-04T12:36:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "map_full": 0, 2024-03-04T12:36:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "map_inc": 16, 2024-03-04T12:36:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_sessions": 7, 2024-03-04T12:36:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_session_open": 7, 2024-03-04T12:36:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_session_close": 0, 2024-03-04T12:36:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_laggy": 0, 2024-03-04T12:36:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_wr": 0, 2024-03-04T12:36:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_rd": 2, 2024-03-04T12:36:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_del": 0 2024-03-04T12:36:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "oft": { 2024-03-04T12:36:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_objs": 0, 2024-03-04T12:36:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_kv_pairs": 0, 2024-03-04T12:36:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_updates": 0, 2024-03-04T12:36:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_removes": 0 2024-03-04T12:36:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "osd-slow-ops": { 2024-03-04T12:36:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "slow_ops_count": 0 2024-03-04T12:36:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "purge_queue": { 2024-03-04T12:36:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_ops": 0, 2024-03-04T12:36:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_ops_high_water": 0, 2024-03-04T12:36:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing": 0, 2024-03-04T12:36:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_high_water": 0, 2024-03-04T12:36:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executed_ops": 0, 2024-03-04T12:36:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executed": 0, 2024-03-04T12:36:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_item_in_journal": 0 2024-03-04T12:36:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-04T12:36:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-03-04T12:36:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 2587, 2024-03-04T12:36:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 38337629, 2024-03-04T12:36:29.148 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:29.148 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 2587, 2024-03-04T12:36:29.148 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-03-04T12:36:29.148 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-03-04T12:36:29.148 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 2587, 2024-03-04T12:36:29.148 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 38337629, 2024-03-04T12:36:29.148 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:29.148 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.148 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.149 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.149 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:29.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.149 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-objecter_bytes": { 2024-03-04T12:36:29.149 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:29.149 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-03-04T12:36:29.149 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:29.149 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:29.149 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:29.150 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:29.150 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:29.150 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 2315, 2024-03-04T12:36:29.150 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 36094880, 2024-03-04T12:36:29.150 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 2315, 2024-03-04T12:36:29.150 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 36094880, 2024-03-04T12:36:29.150 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:29.150 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.151 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:29 smithi086 ceph-mon[36851]: pgmap v331: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-04T12:36:29.151 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.151 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.151 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:29.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.151 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-objecter_ops": { 2024-03-04T12:36:29.152 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:29.152 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 1024, 2024-03-04T12:36:29.152 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:29.152 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:29.152 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:29.152 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:29.152 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:29.152 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 2315, 2024-03-04T12:36:29.153 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 2315, 2024-03-04T12:36:29.153 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 2315, 2024-03-04T12:36:29.153 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 2315, 2024-03-04T12:36:29.153 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:29.153 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.153 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.153 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.153 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:29.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.153 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-write_buf_throttle": { 2024-03-04T12:36:29.154 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:29.154 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 3758096384, 2024-03-04T12:36:29.154 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:29.154 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:29.154 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:29.154 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:29.154 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:29.154 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-03-04T12:36:29.155 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-03-04T12:36:29.155 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 0, 2024-03-04T12:36:29.155 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 0, 2024-03-04T12:36:29.155 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:29.155 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.155 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.155 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.155 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:29.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:29.156 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-write_buf_throttle-0x5560241f2960": { 2024-03-04T12:36:29.156 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:29.156 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 3758096384, 2024-03-04T12:36:29.156 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:29.156 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:29.156 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:29.156 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:29.156 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:29.157 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-03-04T12:36:29.157 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-03-04T12:36:29.157 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 0, 2024-03-04T12:36:29.157 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 0, 2024-03-04T12:36:29.157 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:29.157 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:29.157 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:29.157 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:29.157 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:29.158 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:29.158 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-04T12:36:29.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:29 smithi063 ceph-mon[36717]: pgmap v331: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-04T12:36:29.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:29 smithi060 ceph-mon[32745]: pgmap v331: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-04T12:36:29.922 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-04T12:36:29.923 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-04T12:36:29.923 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-04T12:36:29.923 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-04T12:36:29.923 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-04T12:36:29.923 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-04T12:36:29.923 DEBUG:tasks.check_counter:Getting stats from i 2024-03-04T12:36:29.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-04T12:36:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:36:31.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:31 smithi086 ceph-mon[36851]: pgmap v332: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-03-04T12:36:31.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:31 smithi063 ceph-mon[36717]: pgmap v332: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-03-04T12:36:31.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:31 smithi060 ceph-mon[32745]: pgmap v332: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-03-04T12:36:31.842 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-04T12:36:31.843 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-0": { 2024-03-04T12:36:31.843 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 2611, 2024-03-04T12:36:31.843 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 2471, 2024-03-04T12:36:31.843 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 886610, 2024-03-04T12:36:31.843 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 23918066, 2024-03-04T12:36:31.843 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 11, 2024-03-04T12:36:31.843 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 6, 2024-03-04T12:36:31.843 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.284447532, 2024-03-04T12:36:31.844 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.130479049, 2024-03-04T12:36:31.844 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.099710209, 2024-03-04T12:36:31.844 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.052755527, 2024-03-04T12:36:31.844 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:31.844 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 2469, 2024-03-04T12:36:31.844 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.225812981, 2024-03-04T12:36:31.844 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000091459 2024-03-04T12:36:31.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.845 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:31.845 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 2241, 2024-03-04T12:36:31.845 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000487709, 2024-03-04T12:36:31.845 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000217 2024-03-04T12:36:31.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.845 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-04T12:36:31.845 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 314448 2024-03-04T12:36:31.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.845 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-1": { 2024-03-04T12:36:31.846 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 18085, 2024-03-04T12:36:31.846 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 28046, 2024-03-04T12:36:31.846 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 7889947, 2024-03-04T12:36:31.846 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 16948401, 2024-03-04T12:36:31.846 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 18, 2024-03-04T12:36:31.846 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 6, 2024-03-04T12:36:31.846 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 1.096008351, 2024-03-04T12:36:31.847 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.515601529, 2024-03-04T12:36:31.847 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 1.491213646, 2024-03-04T12:36:31.847 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.059832861, 2024-03-04T12:36:31.847 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:31.847 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 28046, 2024-03-04T12:36:31.847 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 1.065148074, 2024-03-04T12:36:31.847 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000037978 2024-03-04T12:36:31.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.848 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:31.848 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 38157, 2024-03-04T12:36:31.848 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.005910366, 2024-03-04T12:36:31.848 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000154 2024-03-04T12:36:31.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.848 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 7486272, 2024-03-04T12:36:31.848 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 14264928 2024-03-04T12:36:31.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.848 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-2": { 2024-03-04T12:36:31.849 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 3384, 2024-03-04T12:36:31.849 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 3797, 2024-03-04T12:36:31.849 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 5070241, 2024-03-04T12:36:31.849 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 6377807, 2024-03-04T12:36:31.849 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 12, 2024-03-04T12:36:31.849 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 7, 2024-03-04T12:36:31.849 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.316211165, 2024-03-04T12:36:31.849 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.128299414, 2024-03-04T12:36:31.850 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.097962289, 2024-03-04T12:36:31.850 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.077015368, 2024-03-04T12:36:31.850 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:31.850 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 3794, 2024-03-04T12:36:31.850 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.276973670, 2024-03-04T12:36:31.850 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000073003 2024-03-04T12:36:31.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.851 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:31.851 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 1782, 2024-03-04T12:36:31.851 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000267011, 2024-03-04T12:36:31.851 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000149 2024-03-04T12:36:31.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.851 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 4176, 2024-03-04T12:36:31.851 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 141552 2024-03-04T12:36:31.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.852 INFO:teuthology.orchestra.run.smithi086.stdout: "cct": { 2024-03-04T12:36:31.852 INFO:teuthology.orchestra.run.smithi086.stdout: "total_workers": 1, 2024-03-04T12:36:31.852 INFO:teuthology.orchestra.run.smithi086.stdout: "unhealthy_workers": 0 2024-03-04T12:36:31.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.852 INFO:teuthology.orchestra.run.smithi086.stdout: "finisher-MDSRank": { 2024-03-04T12:36:31.852 INFO:teuthology.orchestra.run.smithi086.stdout: "queue_len": 0, 2024-03-04T12:36:31.852 INFO:teuthology.orchestra.run.smithi086.stdout: "complete_latency": { 2024-03-04T12:36:31.852 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 333, 2024-03-04T12:36:31.852 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 8.717911987, 2024-03-04T12:36:31.853 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.026179915 2024-03-04T12:36:31.853 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:31.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.853 INFO:teuthology.orchestra.run.smithi086.stdout: "finisher-PurgeQueue": { 2024-03-04T12:36:31.853 INFO:teuthology.orchestra.run.smithi086.stdout: "queue_len": 0, 2024-03-04T12:36:31.853 INFO:teuthology.orchestra.run.smithi086.stdout: "complete_latency": { 2024-03-04T12:36:31.853 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 4600, 2024-03-04T12:36:31.853 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 1.240072742, 2024-03-04T12:36:31.854 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000269581 2024-03-04T12:36:31.854 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:31.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.854 INFO:teuthology.orchestra.run.smithi086.stdout: "mds": { 2024-03-04T12:36:31.854 INFO:teuthology.orchestra.run.smithi086.stdout: "request": 7433, 2024-03-04T12:36:31.854 INFO:teuthology.orchestra.run.smithi086.stdout: "reply": 7366, 2024-03-04T12:36:31.854 INFO:teuthology.orchestra.run.smithi086.stdout: "reply_latency": { 2024-03-04T12:36:31.854 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 7366, 2024-03-04T12:36:31.854 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 14.605322719, 2024-03-04T12:36:31.855 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.001982802 2024-03-04T12:36:31.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.855 INFO:teuthology.orchestra.run.smithi086.stdout: "slow_reply": 0, 2024-03-04T12:36:31.855 INFO:teuthology.orchestra.run.smithi086.stdout: "forward": 67, 2024-03-04T12:36:31.855 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_fetch_complete": 0, 2024-03-04T12:36:31.855 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_fetch_keys": 0, 2024-03-04T12:36:31.855 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_commit": 37, 2024-03-04T12:36:31.855 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_split": 3, 2024-03-04T12:36:31.855 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_merge": 2, 2024-03-04T12:36:31.856 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes": 4589, 2024-03-04T12:36:31.856 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_top": 217, 2024-03-04T12:36:31.856 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_bottom": 4372, 2024-03-04T12:36:31.856 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_pin_tail": 0, 2024-03-04T12:36:31.856 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_pinned": 4279, 2024-03-04T12:36:31.856 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_expired": 1, 2024-03-04T12:36:31.856 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_with_caps": 2069, 2024-03-04T12:36:31.856 INFO:teuthology.orchestra.run.smithi086.stdout: "caps": 2069, 2024-03-04T12:36:31.857 INFO:teuthology.orchestra.run.smithi086.stdout: "subtrees": 5, 2024-03-04T12:36:31.857 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse": 10909, 2024-03-04T12:36:31.857 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_hit": 10477, 2024-03-04T12:36:31.857 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_forward": 65, 2024-03-04T12:36:31.857 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_discover": 225, 2024-03-04T12:36:31.857 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_dir_fetch": 0, 2024-03-04T12:36:31.857 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_remote_ino": 0, 2024-03-04T12:36:31.857 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_lock": 0, 2024-03-04T12:36:31.857 INFO:teuthology.orchestra.run.smithi086.stdout: "load_cent": 0, 2024-03-04T12:36:31.858 INFO:teuthology.orchestra.run.smithi086.stdout: "q": 0, 2024-03-04T12:36:31.858 INFO:teuthology.orchestra.run.smithi086.stdout: "exported": 0, 2024-03-04T12:36:31.858 INFO:teuthology.orchestra.run.smithi086.stdout: "exported_inodes": 0, 2024-03-04T12:36:31.858 INFO:teuthology.orchestra.run.smithi086.stdout: "imported": 89, 2024-03-04T12:36:31.858 INFO:teuthology.orchestra.run.smithi086.stdout: "imported_inodes": 3936, 2024-03-04T12:36:31.858 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_dir_fetch": 0, 2024-03-04T12:36:31.858 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_backtrace_fetch": 0, 2024-03-04T12:36:31.858 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_peer_discover": 0, 2024-03-04T12:36:31.858 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rfiles": 771, 2024-03-04T12:36:31.859 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rbytes": 13163370, 2024-03-04T12:36:31.859 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rsnaps": 2, 2024-03-04T12:36:31.859 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_backtrace_fetch": 0, 2024-03-04T12:36:31.859 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_set_tag": 0, 2024-03-04T12:36:31.859 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_backtrace_repaired": 0, 2024-03-04T12:36:31.859 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_inotable_repaired": 0, 2024-03-04T12:36:31.859 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dir_inodes": 0, 2024-03-04T12:36:31.859 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dir_base_inodes": 0, 2024-03-04T12:36:31.860 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dirfrag_rstats": 0, 2024-03-04T12:36:31.860 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_file_inodes": 0, 2024-03-04T12:36:31.860 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_inode_file_caps": 4, 2024-03-04T12:36:31.860 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_revoke": 40, 2024-03-04T12:36:31.860 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_grant": 5107, 2024-03-04T12:36:31.860 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_trunc": 0, 2024-03-04T12:36:31.860 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-04T12:36:31.860 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-04T12:36:31.860 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_caps": 8200, 2024-03-04T12:36:31.861 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_caps_dirty": 5481, 2024-03-04T12:36:31.861 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_cap_release": 933, 2024-03-04T12:36:31.861 INFO:teuthology.orchestra.run.smithi086.stdout: "process_request_cap_release": 2905 2024-03-04T12:36:31.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.861 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache": { 2024-03-04T12:36:31.861 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_update": 0, 2024-03-04T12:36:31.861 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_update_receipt": 0, 2024-03-04T12:36:31.861 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_try_discover": 0, 2024-03-04T12:36:31.862 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_send_discover": 114, 2024-03-04T12:36:31.862 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_handle_discover": 0, 2024-03-04T12:36:31.862 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays": 0, 2024-03-04T12:36:31.862 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays_delayed": 0, 2024-03-04T12:36:31.862 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays_enqueuing": 0, 2024-03-04T12:36:31.862 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_created": 2582, 2024-03-04T12:36:31.862 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_enqueued": 2582, 2024-03-04T12:36:31.862 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_reintegrated": 0, 2024-03-04T12:36:31.862 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_migrated": 0, 2024-03-04T12:36:31.863 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_processing": 0, 2024-03-04T12:36:31.863 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_enqueued": 0, 2024-03-04T12:36:31.863 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_prioritized": 0, 2024-03-04T12:36:31.863 INFO:teuthology.orchestra.run.smithi086.stdout: "recovery_started": 0, 2024-03-04T12:36:31.863 INFO:teuthology.orchestra.run.smithi086.stdout: "recovery_completed": 0, 2024-03-04T12:36:31.863 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_enqueue_scrub": 0, 2024-03-04T12:36:31.863 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_exportdir": 0, 2024-03-04T12:36:31.863 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_flush": 0, 2024-03-04T12:36:31.863 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_fragmentdir": 5, 2024-03-04T12:36:31.864 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_fragstats": 0, 2024-03-04T12:36:31.864 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_inodestats": 0 2024-03-04T12:36:31.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.864 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log": { 2024-03-04T12:36:31.864 INFO:teuthology.orchestra.run.smithi086.stdout: "evlrg": 1, 2024-03-04T12:36:31.864 INFO:teuthology.orchestra.run.smithi086.stdout: "evadd": 15412, 2024-03-04T12:36:31.864 INFO:teuthology.orchestra.run.smithi086.stdout: "evex": 0, 2024-03-04T12:36:31.864 INFO:teuthology.orchestra.run.smithi086.stdout: "evtrm": 0, 2024-03-04T12:36:31.864 INFO:teuthology.orchestra.run.smithi086.stdout: "ev": 15412, 2024-03-04T12:36:31.865 INFO:teuthology.orchestra.run.smithi086.stdout: "evexg": 0, 2024-03-04T12:36:31.865 INFO:teuthology.orchestra.run.smithi086.stdout: "evexd": 0, 2024-03-04T12:36:31.865 INFO:teuthology.orchestra.run.smithi086.stdout: "segadd": 23, 2024-03-04T12:36:31.865 INFO:teuthology.orchestra.run.smithi086.stdout: "segex": 0, 2024-03-04T12:36:31.865 INFO:teuthology.orchestra.run.smithi086.stdout: "segtrm": 0, 2024-03-04T12:36:31.865 INFO:teuthology.orchestra.run.smithi086.stdout: "seg": 23, 2024-03-04T12:36:31.865 INFO:teuthology.orchestra.run.smithi086.stdout: "segmjr": 3, 2024-03-04T12:36:31.865 INFO:teuthology.orchestra.run.smithi086.stdout: "segexg": 0, 2024-03-04T12:36:31.865 INFO:teuthology.orchestra.run.smithi086.stdout: "segexd": 0, 2024-03-04T12:36:31.866 INFO:teuthology.orchestra.run.smithi086.stdout: "expos": 4194304, 2024-03-04T12:36:31.866 INFO:teuthology.orchestra.run.smithi086.stdout: "wrpos": 40281423, 2024-03-04T12:36:31.866 INFO:teuthology.orchestra.run.smithi086.stdout: "rdpos": 0, 2024-03-04T12:36:31.866 INFO:teuthology.orchestra.run.smithi086.stdout: "jlat": { 2024-03-04T12:36:31.866 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 166, 2024-03-04T12:36:31.866 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 11.713803652, 2024-03-04T12:36:31.866 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.070565082 2024-03-04T12:36:31.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.866 INFO:teuthology.orchestra.run.smithi086.stdout: "replayed": 0 2024-03-04T12:36:31.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.867 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_mem": { 2024-03-04T12:36:31.867 INFO:teuthology.orchestra.run.smithi086.stdout: "ino": 2993, 2024-03-04T12:36:31.867 INFO:teuthology.orchestra.run.smithi086.stdout: "ino+": 5575, 2024-03-04T12:36:31.867 INFO:teuthology.orchestra.run.smithi086.stdout: "ino-": 2582, 2024-03-04T12:36:31.867 INFO:teuthology.orchestra.run.smithi086.stdout: "dir": 186, 2024-03-04T12:36:31.867 INFO:teuthology.orchestra.run.smithi086.stdout: "dir+": 209, 2024-03-04T12:36:31.867 INFO:teuthology.orchestra.run.smithi086.stdout: "dir-": 23, 2024-03-04T12:36:31.867 INFO:teuthology.orchestra.run.smithi086.stdout: "dn": 4589, 2024-03-04T12:36:31.867 INFO:teuthology.orchestra.run.smithi086.stdout: "dn+": 6634, 2024-03-04T12:36:31.868 INFO:teuthology.orchestra.run.smithi086.stdout: "dn-": 2045, 2024-03-04T12:36:31.868 INFO:teuthology.orchestra.run.smithi086.stdout: "cap": 2069, 2024-03-04T12:36:31.868 INFO:teuthology.orchestra.run.smithi086.stdout: "cap+": 5504, 2024-03-04T12:36:31.868 INFO:teuthology.orchestra.run.smithi086.stdout: "cap-": 3435, 2024-03-04T12:36:31.868 INFO:teuthology.orchestra.run.smithi086.stdout: "rss": 98368, 2024-03-04T12:36:31.868 INFO:teuthology.orchestra.run.smithi086.stdout: "heap": 207228 2024-03-04T12:36:31.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.868 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_server": { 2024-03-04T12:36:31.868 INFO:teuthology.orchestra.run.smithi086.stdout: "dispatch_client_request": 7581, 2024-03-04T12:36:31.869 INFO:teuthology.orchestra.run.smithi086.stdout: "dispatch_server_request": 11, 2024-03-04T12:36:31.869 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_request": 7433, 2024-03-04T12:36:31.869 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_session": 21, 2024-03-04T12:36:31.869 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_peer_request": 27, 2024-03-04T12:36:31.869 INFO:teuthology.orchestra.run.smithi086.stdout: "req_create_latency": { 2024-03-04T12:36:31.869 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 1572, 2024-03-04T12:36:31.869 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 3.343062316, 2024-03-04T12:36:31.869 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.002126629 2024-03-04T12:36:31.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.870 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getattr_latency": { 2024-03-04T12:36:31.870 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 7, 2024-03-04T12:36:31.870 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.005180358, 2024-03-04T12:36:31.870 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000740051 2024-03-04T12:36:31.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.870 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getfilelock_latency": { 2024-03-04T12:36:31.870 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.871 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.871 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.871 INFO:teuthology.orchestra.run.smithi086.stdout: "req_link_latency": { 2024-03-04T12:36:31.871 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.871 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.871 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.871 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookup_latency": { 2024-03-04T12:36:31.872 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 3187, 2024-03-04T12:36:31.872 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 4.908840594, 2024-03-04T12:36:31.872 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.001540270 2024-03-04T12:36:31.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.872 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookuphash_latency": { 2024-03-04T12:36:31.872 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.872 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.872 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.873 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupino_latency": { 2024-03-04T12:36:31.873 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 1, 2024-03-04T12:36:31.873 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000195945, 2024-03-04T12:36:31.873 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000195945 2024-03-04T12:36:31.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.873 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupname_latency": { 2024-03-04T12:36:31.873 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.873 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.874 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.874 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupparent_latency": { 2024-03-04T12:36:31.874 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.874 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.874 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.874 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupsnap_latency": { 2024-03-04T12:36:31.874 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.875 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.875 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.875 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lssnap_latency": { 2024-03-04T12:36:31.875 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.875 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.876 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.876 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mkdir_latency": { 2024-03-04T12:36:31.876 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.876 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.876 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.877 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mknod_latency": { 2024-03-04T12:36:31.877 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.877 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.877 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.877 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mksnap_latency": { 2024-03-04T12:36:31.877 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.877 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.878 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.878 INFO:teuthology.orchestra.run.smithi086.stdout: "req_open_latency": { 2024-03-04T12:36:31.878 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 10, 2024-03-04T12:36:31.878 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.002052702, 2024-03-04T12:36:31.878 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000205270 2024-03-04T12:36:31.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.878 INFO:teuthology.orchestra.run.smithi086.stdout: "req_readdir_latency": { 2024-03-04T12:36:31.878 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 1, 2024-03-04T12:36:31.879 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000707704, 2024-03-04T12:36:31.879 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000707704 2024-03-04T12:36:31.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.879 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rename_latency": { 2024-03-04T12:36:31.879 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.879 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.879 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.879 INFO:teuthology.orchestra.run.smithi086.stdout: "req_renamesnap_latency": { 2024-03-04T12:36:31.880 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.880 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.880 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.880 INFO:teuthology.orchestra.run.smithi086.stdout: "req_snapdiff_latency": { 2024-03-04T12:36:31.880 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.880 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.880 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.881 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmdir_latency": { 2024-03-04T12:36:31.881 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 9, 2024-03-04T12:36:31.881 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.665475710, 2024-03-04T12:36:31.881 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.073941745 2024-03-04T12:36:31.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.881 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmsnap_latency": { 2024-03-04T12:36:31.881 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.881 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.881 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.882 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmxattr_latency": { 2024-03-04T12:36:31.882 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.882 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.882 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.882 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setattr_latency": { 2024-03-04T12:36:31.882 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.882 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.883 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.883 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setdirlayout_latency": { 2024-03-04T12:36:31.883 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.883 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.883 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.883 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setfilelock_latency": { 2024-03-04T12:36:31.883 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.883 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.884 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.884 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setlayout_latency": { 2024-03-04T12:36:31.884 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.884 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.884 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.884 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setxattr_latency": { 2024-03-04T12:36:31.884 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.885 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.885 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.885 INFO:teuthology.orchestra.run.smithi086.stdout: "req_symlink_latency": { 2024-03-04T12:36:31.885 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.885 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.885 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.885 INFO:teuthology.orchestra.run.smithi086.stdout: "req_unlink_latency": { 2024-03-04T12:36:31.886 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 2579, 2024-03-04T12:36:31.886 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 5.679807390, 2024-03-04T12:36:31.886 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.002202329 2024-03-04T12:36:31.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.886 INFO:teuthology.orchestra.run.smithi086.stdout: "cap_revoke_eviction": 0, 2024-03-04T12:36:31.886 INFO:teuthology.orchestra.run.smithi086.stdout: "cap_acquisition_throttle": 0, 2024-03-04T12:36:31.886 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getvxattr_latency": { 2024-03-04T12:36:31.886 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.886 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.886 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.887 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:31.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.887 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_sessions": { 2024-03-04T12:36:31.887 INFO:teuthology.orchestra.run.smithi086.stdout: "session_count": 1, 2024-03-04T12:36:31.887 INFO:teuthology.orchestra.run.smithi086.stdout: "session_add": 4, 2024-03-04T12:36:31.887 INFO:teuthology.orchestra.run.smithi086.stdout: "session_remove": 3, 2024-03-04T12:36:31.887 INFO:teuthology.orchestra.run.smithi086.stdout: "sessions_open": 1, 2024-03-04T12:36:31.887 INFO:teuthology.orchestra.run.smithi086.stdout: "sessions_stale": 0, 2024-03-04T12:36:31.888 INFO:teuthology.orchestra.run.smithi086.stdout: "total_load": 5951, 2024-03-04T12:36:31.888 INFO:teuthology.orchestra.run.smithi086.stdout: "average_load": 5951, 2024-03-04T12:36:31.888 INFO:teuthology.orchestra.run.smithi086.stdout: "avg_session_uptime": 651, 2024-03-04T12:36:31.888 INFO:teuthology.orchestra.run.smithi086.stdout: "mdthresh_evicted": 0 2024-03-04T12:36:31.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.888 INFO:teuthology.orchestra.run.smithi086.stdout: "mempool": { 2024-03-04T12:36:31.888 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_bytes": 0, 2024-03-04T12:36:31.888 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_items": 0, 2024-03-04T12:36:31.888 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_bytes": 0, 2024-03-04T12:36:31.889 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_items": 0, 2024-03-04T12:36:31.889 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_bytes": 0, 2024-03-04T12:36:31.889 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_items": 0, 2024-03-04T12:36:31.889 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-04T12:36:31.889 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_items": 0, 2024-03-04T12:36:31.889 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-04T12:36:31.889 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_items": 0, 2024-03-04T12:36:31.889 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_bytes": 0, 2024-03-04T12:36:31.890 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_items": 0, 2024-03-04T12:36:31.890 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-04T12:36:31.890 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_items": 0, 2024-03-04T12:36:31.890 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_bytes": 0, 2024-03-04T12:36:31.890 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_items": 0, 2024-03-04T12:36:31.890 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_bytes": 0, 2024-03-04T12:36:31.890 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_items": 0, 2024-03-04T12:36:31.890 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-04T12:36:31.891 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_items": 0, 2024-03-04T12:36:31.891 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-04T12:36:31.891 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_items": 0, 2024-03-04T12:36:31.891 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_bytes": 0, 2024-03-04T12:36:31.891 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_items": 0, 2024-03-04T12:36:31.891 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_bytes": 0, 2024-03-04T12:36:31.891 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_items": 0, 2024-03-04T12:36:31.891 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-04T12:36:31.891 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_items": 0, 2024-03-04T12:36:31.892 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_bytes": 0, 2024-03-04T12:36:31.892 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_items": 0, 2024-03-04T12:36:31.892 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_bytes": 0, 2024-03-04T12:36:31.892 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_items": 0, 2024-03-04T12:36:31.892 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_bytes": 0, 2024-03-04T12:36:31.892 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_items": 0, 2024-03-04T12:36:31.893 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_bytes": 0, 2024-03-04T12:36:31.893 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_items": 0, 2024-03-04T12:36:31.893 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_bytes": 400623, 2024-03-04T12:36:31.893 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_items": 3053, 2024-03-04T12:36:31.893 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_bytes": 880, 2024-03-04T12:36:31.893 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_items": 10, 2024-03-04T12:36:31.893 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bytes": 0, 2024-03-04T12:36:31.893 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_items": 0, 2024-03-04T12:36:31.894 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_bytes": 0, 2024-03-04T12:36:31.894 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_items": 0, 2024-03-04T12:36:31.894 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_bytes": 0, 2024-03-04T12:36:31.894 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_items": 0, 2024-03-04T12:36:31.894 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_bytes": 9736, 2024-03-04T12:36:31.894 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_items": 203, 2024-03-04T12:36:31.894 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_bytes": 0, 2024-03-04T12:36:31.894 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_items": 0, 2024-03-04T12:36:31.894 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_bytes": 0, 2024-03-04T12:36:31.895 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_items": 0, 2024-03-04T12:36:31.895 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_bytes": 11355752, 2024-03-04T12:36:31.895 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_items": 135860, 2024-03-04T12:36:31.895 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_bytes": 0, 2024-03-04T12:36:31.895 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_items": 0, 2024-03-04T12:36:31.895 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_bytes": 0, 2024-03-04T12:36:31.895 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_items": 0 2024-03-04T12:36:31.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.895 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter": { 2024-03-04T12:36:31.896 INFO:teuthology.orchestra.run.smithi086.stdout: "op_active": 0, 2024-03-04T12:36:31.896 INFO:teuthology.orchestra.run.smithi086.stdout: "op_laggy": 0, 2024-03-04T12:36:31.896 INFO:teuthology.orchestra.run.smithi086.stdout: "op_send": 5520, 2024-03-04T12:36:31.896 INFO:teuthology.orchestra.run.smithi086.stdout: "op_send_bytes": 37390384, 2024-03-04T12:36:31.896 INFO:teuthology.orchestra.run.smithi086.stdout: "op_resend": 28, 2024-03-04T12:36:31.896 INFO:teuthology.orchestra.run.smithi086.stdout: "op_reply": 5520, 2024-03-04T12:36:31.896 INFO:teuthology.orchestra.run.smithi086.stdout: "op_latency": { 2024-03-04T12:36:31.896 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 5520, 2024-03-04T12:36:31.896 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 18.253586125, 2024-03-04T12:36:31.897 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.003306809 2024-03-04T12:36:31.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.897 INFO:teuthology.orchestra.run.smithi086.stdout: "op_inflight": 0, 2024-03-04T12:36:31.897 INFO:teuthology.orchestra.run.smithi086.stdout: "oplen_avg": { 2024-03-04T12:36:31.897 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 5520, 2024-03-04T12:36:31.897 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 5604 2024-03-04T12:36:31.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.897 INFO:teuthology.orchestra.run.smithi086.stdout: "op": 5520, 2024-03-04T12:36:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "op_r": 27, 2024-03-04T12:36:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "op_w": 5493, 2024-03-04T12:36:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "op_rmw": 0, 2024-03-04T12:36:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "op_pg": 0, 2024-03-04T12:36:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_stat": 0, 2024-03-04T12:36:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_create": 10, 2024-03-04T12:36:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_read": 27, 2024-03-04T12:36:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_write": 193, 2024-03-04T12:36:31.898 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_writefull": 26, 2024-03-04T12:36:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_writesame": 0, 2024-03-04T12:36:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_append": 0, 2024-03-04T12:36:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_zero": 0, 2024-03-04T12:36:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_truncate": 3, 2024-03-04T12:36:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_delete": 5197, 2024-03-04T12:36:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_mapext": 0, 2024-03-04T12:36:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_sparse_read": 0, 2024-03-04T12:36:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_clonerange": 0, 2024-03-04T12:36:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_getxattr": 0, 2024-03-04T12:36:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_setxattr": 20, 2024-03-04T12:36:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_cmpxattr": 0, 2024-03-04T12:36:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_rmxattr": 0, 2024-03-04T12:36:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_resetxattrs": 0, 2024-03-04T12:36:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_call": 0, 2024-03-04T12:36:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_watch": 0, 2024-03-04T12:36:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_notify": 0, 2024-03-04T12:36:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_src_cmpxattr": 0, 2024-03-04T12:36:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_pgls": 0, 2024-03-04T12:36:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_pgls_filter": 0, 2024-03-04T12:36:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_other": 0, 2024-03-04T12:36:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_active": 0, 2024-03-04T12:36:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_send": 0, 2024-03-04T12:36:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_resend": 0, 2024-03-04T12:36:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_ping": 0, 2024-03-04T12:36:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_active": 0, 2024-03-04T12:36:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_send": 0, 2024-03-04T12:36:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_resend": 0, 2024-03-04T12:36:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_active": 0, 2024-03-04T12:36:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_send": 0, 2024-03-04T12:36:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_resend": 0, 2024-03-04T12:36:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_active": 0, 2024-03-04T12:36:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_send": 0, 2024-03-04T12:36:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_resend": 0, 2024-03-04T12:36:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "command_active": 0, 2024-03-04T12:36:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "command_send": 0, 2024-03-04T12:36:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "command_resend": 0, 2024-03-04T12:36:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "map_epoch": 105, 2024-03-04T12:36:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "map_full": 0, 2024-03-04T12:36:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "map_inc": 16, 2024-03-04T12:36:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_sessions": 12, 2024-03-04T12:36:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_session_open": 12, 2024-03-04T12:36:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_session_close": 0, 2024-03-04T12:36:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_laggy": 0, 2024-03-04T12:36:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_wr": 110, 2024-03-04T12:36:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_rd": 0, 2024-03-04T12:36:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_del": 18 2024-03-04T12:36:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "oft": { 2024-03-04T12:36:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_objs": 1, 2024-03-04T12:36:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_kv_pairs": 0, 2024-03-04T12:36:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_updates": 0, 2024-03-04T12:36:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_removes": 0 2024-03-04T12:36:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: "osd-slow-ops": { 2024-03-04T12:36:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: "slow_ops_count": 0 2024-03-04T12:36:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: "purge_queue": { 2024-03-04T12:36:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_ops": 0, 2024-03-04T12:36:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_ops_high_water": 10, 2024-03-04T12:36:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing": 0, 2024-03-04T12:36:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_high_water": 5, 2024-03-04T12:36:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executed_ops": 5161, 2024-03-04T12:36:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executed": 2582, 2024-03-04T12:36:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_item_in_journal": 0 2024-03-04T12:36:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-04T12:36:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-03-04T12:36:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 24076, 2024-03-04T12:36:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 10950556, 2024-03-04T12:36:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 24076, 2024-03-04T12:36:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-03-04T12:36:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-03-04T12:36:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 24076, 2024-03-04T12:36:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 10950556, 2024-03-04T12:36:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-objecter_bytes": { 2024-03-04T12:36:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-03-04T12:36:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 5520, 2024-03-04T12:36:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 37671798, 2024-03-04T12:36:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 5520, 2024-03-04T12:36:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 37671798, 2024-03-04T12:36:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-objecter_ops": { 2024-03-04T12:36:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 1024, 2024-03-04T12:36:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 5520, 2024-03-04T12:36:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 5520, 2024-03-04T12:36:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 5520, 2024-03-04T12:36:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 5520, 2024-03-04T12:36:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-write_buf_throttle": { 2024-03-04T12:36:31.915 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:31.916 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 3758096384, 2024-03-04T12:36:31.916 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:31.916 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 2582, 2024-03-04T12:36:31.916 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 281414, 2024-03-04T12:36:31.916 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:31.916 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 2582, 2024-03-04T12:36:31.916 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-03-04T12:36:31.916 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-03-04T12:36:31.916 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 27, 2024-03-04T12:36:31.917 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 281414, 2024-03-04T12:36:31.917 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:31.917 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.917 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.917 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.917 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:31.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:31.917 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-write_buf_throttle-0x55b62a116640": { 2024-03-04T12:36:31.918 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:31.918 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 3758096384, 2024-03-04T12:36:31.918 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:31.918 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 15412, 2024-03-04T12:36:31.918 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 36087119, 2024-03-04T12:36:31.918 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:31.918 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 15412, 2024-03-04T12:36:31.918 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-03-04T12:36:31.918 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-03-04T12:36:31.919 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 166, 2024-03-04T12:36:31.919 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 36087119, 2024-03-04T12:36:31.919 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:31.919 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:31.919 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:31.919 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:31.919 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:31.919 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:31.920 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-04T12:36:32.679 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-04T12:36:32.680 INFO:tasks.check_counter:Daemon mds.i mds.imported=89 2024-03-04T12:36:32.680 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=2 2024-03-04T12:36:32.680 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-04T12:36:32.680 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-04T12:36:32.680 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=3 2024-03-04T12:36:32.680 DEBUG:tasks.check_counter:Getting stats from l 2024-03-04T12:36:32.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0 shell --fsid a1c49686-da20-11ee-95c6-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-04T12:36:32.918 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T12:36:33.291 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:33 smithi086 ceph-mon[36851]: pgmap v333: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-04T12:36:33.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:33 smithi063 ceph-mon[36717]: pgmap v333: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-04T12:36:33.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:33 smithi060 ceph-mon[32745]: pgmap v333: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-04T12:36:34.617 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-04T12:36:34.617 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-0": { 2024-03-04T12:36:34.618 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 29, 2024-03-04T12:36:34.618 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 162, 2024-03-04T12:36:34.618 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 6076, 2024-03-04T12:36:34.618 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 440735, 2024-03-04T12:36:34.618 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 21, 2024-03-04T12:36:34.618 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 6, 2024-03-04T12:36:34.618 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.014303723, 2024-03-04T12:36:34.618 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.005611783, 2024-03-04T12:36:34.619 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.005170786, 2024-03-04T12:36:34.619 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.000394489, 2024-03-04T12:36:34.619 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:34.619 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 158, 2024-03-04T12:36:34.619 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.019800060, 2024-03-04T12:36:34.619 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000125316 2024-03-04T12:36:34.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.619 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:34.620 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 12, 2024-03-04T12:36:34.620 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000001294, 2024-03-04T12:36:34.620 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000107 2024-03-04T12:36:34.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.620 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-04T12:36:34.620 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 316224 2024-03-04T12:36:34.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.620 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-1": { 2024-03-04T12:36:34.620 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 1039, 2024-03-04T12:36:34.621 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 959, 2024-03-04T12:36:34.621 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 2077008, 2024-03-04T12:36:34.621 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 420127, 2024-03-04T12:36:34.621 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 6, 2024-03-04T12:36:34.621 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 4, 2024-03-04T12:36:34.621 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.130327093, 2024-03-04T12:36:34.621 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.045970023, 2024-03-04T12:36:34.621 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.091671798, 2024-03-04T12:36:34.621 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.010208462, 2024-03-04T12:36:34.622 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:34.622 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 961, 2024-03-04T12:36:34.622 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.050563685, 2024-03-04T12:36:34.622 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000052615 2024-03-04T12:36:34.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.622 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:34.622 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 1387, 2024-03-04T12:36:34.622 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000313689, 2024-03-04T12:36:34.622 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000226 2024-03-04T12:36:34.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.623 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 1832400, 2024-03-04T12:36:34.623 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 144448 2024-03-04T12:36:34.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.623 INFO:teuthology.orchestra.run.smithi086.stdout: "AsyncMessenger::Worker-2": { 2024-03-04T12:36:34.623 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_messages": 39, 2024-03-04T12:36:34.623 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages": 115, 2024-03-04T12:36:34.623 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_bytes": 16104, 2024-03-04T12:36:34.624 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_bytes": 161612, 2024-03-04T12:36:34.624 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_created_connections": 10, 2024-03-04T12:36:34.624 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_active_connections": 5, 2024-03-04T12:36:34.624 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_total_time": 0.009615515, 2024-03-04T12:36:34.624 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_send_time": 0.003439205, 2024-03-04T12:36:34.624 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_recv_time": 0.005023766, 2024-03-04T12:36:34.624 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_running_fast_dispatch_time": 0.000368117, 2024-03-04T12:36:34.624 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_messages_queue_lat": { 2024-03-04T12:36:34.624 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 112, 2024-03-04T12:36:34.625 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.017106580, 2024-03-04T12:36:34.625 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000152737 2024-03-04T12:36:34.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.625 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_handle_ack_lat": { 2024-03-04T12:36:34.625 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 73, 2024-03-04T12:36:34.625 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000011937, 2024-03-04T12:36:34.625 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000163 2024-03-04T12:36:34.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.626 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_recv_encrypted_bytes": 4384, 2024-03-04T12:36:34.626 INFO:teuthology.orchestra.run.smithi086.stdout: "msgr_send_encrypted_bytes": 147392 2024-03-04T12:36:34.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.626 INFO:teuthology.orchestra.run.smithi086.stdout: "cct": { 2024-03-04T12:36:34.626 INFO:teuthology.orchestra.run.smithi086.stdout: "total_workers": 1, 2024-03-04T12:36:34.626 INFO:teuthology.orchestra.run.smithi086.stdout: "unhealthy_workers": 0 2024-03-04T12:36:34.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.626 INFO:teuthology.orchestra.run.smithi086.stdout: "finisher-MDSRank": { 2024-03-04T12:36:34.626 INFO:teuthology.orchestra.run.smithi086.stdout: "queue_len": 0, 2024-03-04T12:36:34.627 INFO:teuthology.orchestra.run.smithi086.stdout: "complete_latency": { 2024-03-04T12:36:34.627 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 41, 2024-03-04T12:36:34.627 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.001968488, 2024-03-04T12:36:34.627 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000048011 2024-03-04T12:36:34.627 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:34.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.627 INFO:teuthology.orchestra.run.smithi086.stdout: "finisher-PurgeQueue": { 2024-03-04T12:36:34.627 INFO:teuthology.orchestra.run.smithi086.stdout: "queue_len": 0, 2024-03-04T12:36:34.627 INFO:teuthology.orchestra.run.smithi086.stdout: "complete_latency": { 2024-03-04T12:36:34.628 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 2, 2024-03-04T12:36:34.628 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000007050, 2024-03-04T12:36:34.628 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000003525 2024-03-04T12:36:34.628 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:34.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.628 INFO:teuthology.orchestra.run.smithi086.stdout: "mds": { 2024-03-04T12:36:34.628 INFO:teuthology.orchestra.run.smithi086.stdout: "request": 5, 2024-03-04T12:36:34.628 INFO:teuthology.orchestra.run.smithi086.stdout: "reply": 3, 2024-03-04T12:36:34.629 INFO:teuthology.orchestra.run.smithi086.stdout: "reply_latency": { 2024-03-04T12:36:34.629 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 3, 2024-03-04T12:36:34.629 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000595117, 2024-03-04T12:36:34.629 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000198372 2024-03-04T12:36:34.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.629 INFO:teuthology.orchestra.run.smithi086.stdout: "slow_reply": 0, 2024-03-04T12:36:34.629 INFO:teuthology.orchestra.run.smithi086.stdout: "forward": 2, 2024-03-04T12:36:34.629 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_fetch_complete": 0, 2024-03-04T12:36:34.629 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_fetch_keys": 0, 2024-03-04T12:36:34.630 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_commit": 11, 2024-03-04T12:36:34.630 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_split": 0, 2024-03-04T12:36:34.630 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_merge": 0, 2024-03-04T12:36:34.630 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes": 17, 2024-03-04T12:36:34.630 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_top": 0, 2024-03-04T12:36:34.630 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_bottom": 17, 2024-03-04T12:36:34.630 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_pin_tail": 0, 2024-03-04T12:36:34.630 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_pinned": 16, 2024-03-04T12:36:34.630 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_expired": 0, 2024-03-04T12:36:34.631 INFO:teuthology.orchestra.run.smithi086.stdout: "inodes_with_caps": 1, 2024-03-04T12:36:34.631 INFO:teuthology.orchestra.run.smithi086.stdout: "caps": 1, 2024-03-04T12:36:34.631 INFO:teuthology.orchestra.run.smithi086.stdout: "subtrees": 2, 2024-03-04T12:36:34.631 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse": 10, 2024-03-04T12:36:34.631 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_hit": 8, 2024-03-04T12:36:34.631 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_forward": 0, 2024-03-04T12:36:34.631 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_discover": 1, 2024-03-04T12:36:34.631 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_dir_fetch": 0, 2024-03-04T12:36:34.631 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_remote_ino": 0, 2024-03-04T12:36:34.632 INFO:teuthology.orchestra.run.smithi086.stdout: "traverse_lock": 0, 2024-03-04T12:36:34.632 INFO:teuthology.orchestra.run.smithi086.stdout: "load_cent": 0, 2024-03-04T12:36:34.632 INFO:teuthology.orchestra.run.smithi086.stdout: "q": 0, 2024-03-04T12:36:34.632 INFO:teuthology.orchestra.run.smithi086.stdout: "exported": 0, 2024-03-04T12:36:34.632 INFO:teuthology.orchestra.run.smithi086.stdout: "exported_inodes": 0, 2024-03-04T12:36:34.632 INFO:teuthology.orchestra.run.smithi086.stdout: "imported": 0, 2024-03-04T12:36:34.632 INFO:teuthology.orchestra.run.smithi086.stdout: "imported_inodes": 0, 2024-03-04T12:36:34.633 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_dir_fetch": 0, 2024-03-04T12:36:34.633 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_backtrace_fetch": 0, 2024-03-04T12:36:34.633 INFO:teuthology.orchestra.run.smithi086.stdout: "openino_peer_discover": 0, 2024-03-04T12:36:34.633 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rfiles": 771, 2024-03-04T12:36:34.633 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rbytes": 13163370, 2024-03-04T12:36:34.633 INFO:teuthology.orchestra.run.smithi086.stdout: "root_rsnaps": 2, 2024-03-04T12:36:34.633 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_backtrace_fetch": 0, 2024-03-04T12:36:34.634 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_set_tag": 0, 2024-03-04T12:36:34.634 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_backtrace_repaired": 0, 2024-03-04T12:36:34.634 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_inotable_repaired": 0, 2024-03-04T12:36:34.634 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dir_inodes": 0, 2024-03-04T12:36:34.634 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dir_base_inodes": 0, 2024-03-04T12:36:34.634 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_dirfrag_rstats": 0, 2024-03-04T12:36:34.634 INFO:teuthology.orchestra.run.smithi086.stdout: "scrub_file_inodes": 0, 2024-03-04T12:36:34.634 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_inode_file_caps": 0, 2024-03-04T12:36:34.634 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_revoke": 0, 2024-03-04T12:36:34.635 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_grant": 0, 2024-03-04T12:36:34.635 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_trunc": 0, 2024-03-04T12:36:34.635 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-04T12:36:34.635 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-04T12:36:34.635 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_caps": 0, 2024-03-04T12:36:34.635 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_caps_dirty": 0, 2024-03-04T12:36:34.635 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_cap_release": 0, 2024-03-04T12:36:34.635 INFO:teuthology.orchestra.run.smithi086.stdout: "process_request_cap_release": 0 2024-03-04T12:36:34.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.636 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache": { 2024-03-04T12:36:34.636 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_update": 0, 2024-03-04T12:36:34.636 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_update_receipt": 5, 2024-03-04T12:36:34.636 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_try_discover": 3, 2024-03-04T12:36:34.636 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_send_discover": 6, 2024-03-04T12:36:34.636 INFO:teuthology.orchestra.run.smithi086.stdout: "dir_handle_discover": 0, 2024-03-04T12:36:34.636 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays": 0, 2024-03-04T12:36:34.636 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays_delayed": 0, 2024-03-04T12:36:34.637 INFO:teuthology.orchestra.run.smithi086.stdout: "num_strays_enqueuing": 0, 2024-03-04T12:36:34.637 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_created": 0, 2024-03-04T12:36:34.637 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_enqueued": 0, 2024-03-04T12:36:34.637 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_reintegrated": 0, 2024-03-04T12:36:34.637 INFO:teuthology.orchestra.run.smithi086.stdout: "strays_migrated": 0, 2024-03-04T12:36:34.637 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_processing": 0, 2024-03-04T12:36:34.637 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_enqueued": 0, 2024-03-04T12:36:34.637 INFO:teuthology.orchestra.run.smithi086.stdout: "num_recovering_prioritized": 0, 2024-03-04T12:36:34.637 INFO:teuthology.orchestra.run.smithi086.stdout: "recovery_started": 0, 2024-03-04T12:36:34.638 INFO:teuthology.orchestra.run.smithi086.stdout: "recovery_completed": 0, 2024-03-04T12:36:34.638 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_enqueue_scrub": 0, 2024-03-04T12:36:34.638 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_exportdir": 0, 2024-03-04T12:36:34.638 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_flush": 0, 2024-03-04T12:36:34.638 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_fragmentdir": 0, 2024-03-04T12:36:34.638 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_fragstats": 0, 2024-03-04T12:36:34.638 INFO:teuthology.orchestra.run.smithi086.stdout: "ireq_inodestats": 0 2024-03-04T12:36:34.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.638 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log": { 2024-03-04T12:36:34.639 INFO:teuthology.orchestra.run.smithi086.stdout: "evlrg": 0, 2024-03-04T12:36:34.639 INFO:teuthology.orchestra.run.smithi086.stdout: "evadd": 5, 2024-03-04T12:36:34.639 INFO:teuthology.orchestra.run.smithi086.stdout: "evex": 0, 2024-03-04T12:36:34.639 INFO:teuthology.orchestra.run.smithi086.stdout: "evtrm": 0, 2024-03-04T12:36:34.639 INFO:teuthology.orchestra.run.smithi086.stdout: "ev": 5, 2024-03-04T12:36:34.639 INFO:teuthology.orchestra.run.smithi086.stdout: "evexg": 0, 2024-03-04T12:36:34.639 INFO:teuthology.orchestra.run.smithi086.stdout: "evexd": 0, 2024-03-04T12:36:34.639 INFO:teuthology.orchestra.run.smithi086.stdout: "segadd": 2, 2024-03-04T12:36:34.639 INFO:teuthology.orchestra.run.smithi086.stdout: "segex": 0, 2024-03-04T12:36:34.640 INFO:teuthology.orchestra.run.smithi086.stdout: "segtrm": 0, 2024-03-04T12:36:34.640 INFO:teuthology.orchestra.run.smithi086.stdout: "seg": 2, 2024-03-04T12:36:34.640 INFO:teuthology.orchestra.run.smithi086.stdout: "segmjr": 2, 2024-03-04T12:36:34.640 INFO:teuthology.orchestra.run.smithi086.stdout: "segexg": 0, 2024-03-04T12:36:34.640 INFO:teuthology.orchestra.run.smithi086.stdout: "segexd": 0, 2024-03-04T12:36:34.640 INFO:teuthology.orchestra.run.smithi086.stdout: "expos": 4194304, 2024-03-04T12:36:34.640 INFO:teuthology.orchestra.run.smithi086.stdout: "wrpos": 4195815, 2024-03-04T12:36:34.640 INFO:teuthology.orchestra.run.smithi086.stdout: "rdpos": 0, 2024-03-04T12:36:34.640 INFO:teuthology.orchestra.run.smithi086.stdout: "jlat": { 2024-03-04T12:36:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 4, 2024-03-04T12:36:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.008533865, 2024-03-04T12:36:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.002133466 2024-03-04T12:36:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: "replayed": 0 2024-03-04T12:36:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_mem": { 2024-03-04T12:36:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: "ino": 20, 2024-03-04T12:36:34.641 INFO:teuthology.orchestra.run.smithi086.stdout: "ino+": 20, 2024-03-04T12:36:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: "ino-": 0, 2024-03-04T12:36:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: "dir": 18, 2024-03-04T12:36:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: "dir+": 21, 2024-03-04T12:36:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: "dir-": 3, 2024-03-04T12:36:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: "dn": 17, 2024-03-04T12:36:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: "dn+": 17, 2024-03-04T12:36:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: "dn-": 0, 2024-03-04T12:36:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: "cap": 1, 2024-03-04T12:36:34.642 INFO:teuthology.orchestra.run.smithi086.stdout: "cap+": 1, 2024-03-04T12:36:34.643 INFO:teuthology.orchestra.run.smithi086.stdout: "cap-": 0, 2024-03-04T12:36:34.643 INFO:teuthology.orchestra.run.smithi086.stdout: "rss": 51580, 2024-03-04T12:36:34.643 INFO:teuthology.orchestra.run.smithi086.stdout: "heap": 207228 2024-03-04T12:36:34.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.643 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_server": { 2024-03-04T12:36:34.643 INFO:teuthology.orchestra.run.smithi086.stdout: "dispatch_client_request": 5, 2024-03-04T12:36:34.643 INFO:teuthology.orchestra.run.smithi086.stdout: "dispatch_server_request": 0, 2024-03-04T12:36:34.643 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_request": 5, 2024-03-04T12:36:34.643 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_client_session": 13, 2024-03-04T12:36:34.644 INFO:teuthology.orchestra.run.smithi086.stdout: "handle_peer_request": 0, 2024-03-04T12:36:34.644 INFO:teuthology.orchestra.run.smithi086.stdout: "req_create_latency": { 2024-03-04T12:36:34.644 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.644 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.644 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.644 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getattr_latency": { 2024-03-04T12:36:34.644 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 3, 2024-03-04T12:36:34.644 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000595117, 2024-03-04T12:36:34.645 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000198372 2024-03-04T12:36:34.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.645 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getfilelock_latency": { 2024-03-04T12:36:34.645 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.645 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.645 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.646 INFO:teuthology.orchestra.run.smithi086.stdout: "req_link_latency": { 2024-03-04T12:36:34.646 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.646 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.646 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.646 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookup_latency": { 2024-03-04T12:36:34.646 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.646 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.647 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.647 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookuphash_latency": { 2024-03-04T12:36:34.647 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.647 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.647 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.647 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupino_latency": { 2024-03-04T12:36:34.647 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.648 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.648 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.648 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupname_latency": { 2024-03-04T12:36:34.648 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.648 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.648 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.648 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupparent_latency": { 2024-03-04T12:36:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lookupsnap_latency": { 2024-03-04T12:36:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.650 INFO:teuthology.orchestra.run.smithi086.stdout: "req_lssnap_latency": { 2024-03-04T12:36:34.650 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.650 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.650 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.650 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mkdir_latency": { 2024-03-04T12:36:34.650 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.650 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.651 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.651 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mknod_latency": { 2024-03-04T12:36:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: "req_mksnap_latency": { 2024-03-04T12:36:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.652 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: "req_open_latency": { 2024-03-04T12:36:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: "req_readdir_latency": { 2024-03-04T12:36:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.653 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.654 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.655 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rename_latency": { 2024-03-04T12:36:34.656 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.656 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.656 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.656 INFO:teuthology.orchestra.run.smithi086.stdout: "req_renamesnap_latency": { 2024-03-04T12:36:34.656 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.656 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.657 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.657 INFO:teuthology.orchestra.run.smithi086.stdout: "req_snapdiff_latency": { 2024-03-04T12:36:34.657 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.657 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.657 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.657 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmdir_latency": { 2024-03-04T12:36:34.657 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.658 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.658 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.658 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmsnap_latency": { 2024-03-04T12:36:34.658 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.658 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.658 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.659 INFO:teuthology.orchestra.run.smithi086.stdout: "req_rmxattr_latency": { 2024-03-04T12:36:34.659 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.659 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.659 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.659 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setattr_latency": { 2024-03-04T12:36:34.659 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.659 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.659 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.660 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setdirlayout_latency": { 2024-03-04T12:36:34.660 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.660 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.660 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.660 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setfilelock_latency": { 2024-03-04T12:36:34.660 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.661 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.661 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.661 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setlayout_latency": { 2024-03-04T12:36:34.661 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.661 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.661 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.661 INFO:teuthology.orchestra.run.smithi086.stdout: "req_setxattr_latency": { 2024-03-04T12:36:34.662 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.662 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.662 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.662 INFO:teuthology.orchestra.run.smithi086.stdout: "req_symlink_latency": { 2024-03-04T12:36:34.662 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.663 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.663 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.663 INFO:teuthology.orchestra.run.smithi086.stdout: "req_unlink_latency": { 2024-03-04T12:36:34.663 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.663 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.663 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.664 INFO:teuthology.orchestra.run.smithi086.stdout: "cap_revoke_eviction": 0, 2024-03-04T12:36:34.664 INFO:teuthology.orchestra.run.smithi086.stdout: "cap_acquisition_throttle": 0, 2024-03-04T12:36:34.664 INFO:teuthology.orchestra.run.smithi086.stdout: "req_getvxattr_latency": { 2024-03-04T12:36:34.664 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.664 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.664 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.664 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:34.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.664 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_sessions": { 2024-03-04T12:36:34.665 INFO:teuthology.orchestra.run.smithi086.stdout: "session_count": 1, 2024-03-04T12:36:34.665 INFO:teuthology.orchestra.run.smithi086.stdout: "session_add": 2, 2024-03-04T12:36:34.665 INFO:teuthology.orchestra.run.smithi086.stdout: "session_remove": 1, 2024-03-04T12:36:34.665 INFO:teuthology.orchestra.run.smithi086.stdout: "sessions_open": 1, 2024-03-04T12:36:34.665 INFO:teuthology.orchestra.run.smithi086.stdout: "sessions_stale": 0, 2024-03-04T12:36:34.665 INFO:teuthology.orchestra.run.smithi086.stdout: "total_load": 2, 2024-03-04T12:36:34.665 INFO:teuthology.orchestra.run.smithi086.stdout: "average_load": 1, 2024-03-04T12:36:34.665 INFO:teuthology.orchestra.run.smithi086.stdout: "avg_session_uptime": 628, 2024-03-04T12:36:34.665 INFO:teuthology.orchestra.run.smithi086.stdout: "mdthresh_evicted": 0 2024-03-04T12:36:34.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.666 INFO:teuthology.orchestra.run.smithi086.stdout: "mempool": { 2024-03-04T12:36:34.666 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_bytes": 0, 2024-03-04T12:36:34.666 INFO:teuthology.orchestra.run.smithi086.stdout: "bloom_filter_items": 0, 2024-03-04T12:36:34.666 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_bytes": 0, 2024-03-04T12:36:34.666 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_items": 0, 2024-03-04T12:36:34.667 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_bytes": 0, 2024-03-04T12:36:34.667 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_data_items": 0, 2024-03-04T12:36:34.667 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-04T12:36:34.667 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_onode_items": 0, 2024-03-04T12:36:34.667 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-04T12:36:34.667 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_items": 0, 2024-03-04T12:36:34.667 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_bytes": 0, 2024-03-04T12:36:34.667 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_other_items": 0, 2024-03-04T12:36:34.667 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-04T12:36:34.668 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_buffer_items": 0, 2024-03-04T12:36:34.668 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_bytes": 0, 2024-03-04T12:36:34.668 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_items": 0, 2024-03-04T12:36:34.668 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_bytes": 0, 2024-03-04T12:36:34.668 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blob_items": 0, 2024-03-04T12:36:34.668 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-04T12:36:34.669 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_shared_blob_items": 0, 2024-03-04T12:36:34.669 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-04T12:36:34.669 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_inline_bl_items": 0, 2024-03-04T12:36:34.669 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_bytes": 0, 2024-03-04T12:36:34.669 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_items": 0, 2024-03-04T12:36:34.669 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_bytes": 0, 2024-03-04T12:36:34.669 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_txc_items": 0, 2024-03-04T12:36:34.670 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-04T12:36:34.670 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_deferred_items": 0, 2024-03-04T12:36:34.670 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_bytes": 0, 2024-03-04T12:36:34.670 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_writing_items": 0, 2024-03-04T12:36:34.670 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_bytes": 0, 2024-03-04T12:36:34.670 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_items": 0, 2024-03-04T12:36:34.670 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_bytes": 0, 2024-03-04T12:36:34.670 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_reader_items": 0, 2024-03-04T12:36:34.671 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_bytes": 0, 2024-03-04T12:36:34.671 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_file_writer_items": 0, 2024-03-04T12:36:34.671 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_bytes": 248272, 2024-03-04T12:36:34.671 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_anon_items": 93, 2024-03-04T12:36:34.671 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_bytes": 88, 2024-03-04T12:36:34.671 INFO:teuthology.orchestra.run.smithi086.stdout: "buffer_meta_items": 1, 2024-03-04T12:36:34.671 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bytes": 0, 2024-03-04T12:36:34.671 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_items": 0, 2024-03-04T12:36:34.671 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_bytes": 0, 2024-03-04T12:36:34.672 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mapbl_items": 0, 2024-03-04T12:36:34.672 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_bytes": 0, 2024-03-04T12:36:34.672 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pglog_items": 0, 2024-03-04T12:36:34.672 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_bytes": 9736, 2024-03-04T12:36:34.672 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_items": 203, 2024-03-04T12:36:34.672 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_bytes": 0, 2024-03-04T12:36:34.672 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap_mapping_items": 0, 2024-03-04T12:36:34.672 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_bytes": 0, 2024-03-04T12:36:34.673 INFO:teuthology.orchestra.run.smithi086.stdout: "pgmap_items": 0, 2024-03-04T12:36:34.673 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_bytes": 88002, 2024-03-04T12:36:34.673 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_co_items": 422, 2024-03-04T12:36:34.673 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_bytes": 0, 2024-03-04T12:36:34.673 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_1_items": 0, 2024-03-04T12:36:34.673 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_bytes": 0, 2024-03-04T12:36:34.673 INFO:teuthology.orchestra.run.smithi086.stdout: "unittest_2_items": 0 2024-03-04T12:36:34.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.673 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter": { 2024-03-04T12:36:34.674 INFO:teuthology.orchestra.run.smithi086.stdout: "op_active": 0, 2024-03-04T12:36:34.674 INFO:teuthology.orchestra.run.smithi086.stdout: "op_laggy": 0, 2024-03-04T12:36:34.674 INFO:teuthology.orchestra.run.smithi086.stdout: "op_send": 40, 2024-03-04T12:36:34.674 INFO:teuthology.orchestra.run.smithi086.stdout: "op_send_bytes": 11813, 2024-03-04T12:36:34.674 INFO:teuthology.orchestra.run.smithi086.stdout: "op_resend": 33, 2024-03-04T12:36:34.674 INFO:teuthology.orchestra.run.smithi086.stdout: "op_reply": 40, 2024-03-04T12:36:34.674 INFO:teuthology.orchestra.run.smithi086.stdout: "op_latency": { 2024-03-04T12:36:34.674 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 40, 2024-03-04T12:36:34.674 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.214995369, 2024-03-04T12:36:34.675 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.005374884 2024-03-04T12:36:34.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.675 INFO:teuthology.orchestra.run.smithi086.stdout: "op_inflight": 0, 2024-03-04T12:36:34.675 INFO:teuthology.orchestra.run.smithi086.stdout: "oplen_avg": { 2024-03-04T12:36:34.675 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 40, 2024-03-04T12:36:34.675 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 62 2024-03-04T12:36:34.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.675 INFO:teuthology.orchestra.run.smithi086.stdout: "op": 40, 2024-03-04T12:36:34.675 INFO:teuthology.orchestra.run.smithi086.stdout: "op_r": 0, 2024-03-04T12:36:34.676 INFO:teuthology.orchestra.run.smithi086.stdout: "op_w": 40, 2024-03-04T12:36:34.676 INFO:teuthology.orchestra.run.smithi086.stdout: "op_rmw": 0, 2024-03-04T12:36:34.676 INFO:teuthology.orchestra.run.smithi086.stdout: "op_pg": 0, 2024-03-04T12:36:34.676 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_stat": 0, 2024-03-04T12:36:34.676 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_create": 10, 2024-03-04T12:36:34.676 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_read": 0, 2024-03-04T12:36:34.676 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_write": 4, 2024-03-04T12:36:34.676 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_writefull": 7, 2024-03-04T12:36:34.677 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_writesame": 0, 2024-03-04T12:36:34.677 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_append": 0, 2024-03-04T12:36:34.677 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_zero": 0, 2024-03-04T12:36:34.677 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_truncate": 0, 2024-03-04T12:36:34.677 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_delete": 6, 2024-03-04T12:36:34.677 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_mapext": 0, 2024-03-04T12:36:34.677 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_sparse_read": 0, 2024-03-04T12:36:34.677 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_clonerange": 0, 2024-03-04T12:36:34.677 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_getxattr": 0, 2024-03-04T12:36:34.678 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_setxattr": 20, 2024-03-04T12:36:34.678 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_cmpxattr": 0, 2024-03-04T12:36:34.678 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_rmxattr": 0, 2024-03-04T12:36:34.678 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_resetxattrs": 0, 2024-03-04T12:36:34.678 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_call": 0, 2024-03-04T12:36:34.678 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_watch": 0, 2024-03-04T12:36:34.678 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_notify": 0, 2024-03-04T12:36:34.678 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_src_cmpxattr": 0, 2024-03-04T12:36:34.678 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_pgls": 0, 2024-03-04T12:36:34.679 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_pgls_filter": 0, 2024-03-04T12:36:34.679 INFO:teuthology.orchestra.run.smithi086.stdout: "osdop_other": 0, 2024-03-04T12:36:34.679 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_active": 0, 2024-03-04T12:36:34.679 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_send": 0, 2024-03-04T12:36:34.679 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_resend": 0, 2024-03-04T12:36:34.679 INFO:teuthology.orchestra.run.smithi086.stdout: "linger_ping": 0, 2024-03-04T12:36:34.679 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_active": 0, 2024-03-04T12:36:34.679 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_send": 0, 2024-03-04T12:36:34.679 INFO:teuthology.orchestra.run.smithi086.stdout: "poolop_resend": 0, 2024-03-04T12:36:34.680 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_active": 0, 2024-03-04T12:36:34.680 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_send": 0, 2024-03-04T12:36:34.680 INFO:teuthology.orchestra.run.smithi086.stdout: "poolstat_resend": 0, 2024-03-04T12:36:34.680 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_active": 0, 2024-03-04T12:36:34.680 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_send": 0, 2024-03-04T12:36:34.680 INFO:teuthology.orchestra.run.smithi086.stdout: "statfs_resend": 0, 2024-03-04T12:36:34.680 INFO:teuthology.orchestra.run.smithi086.stdout: "command_active": 0, 2024-03-04T12:36:34.680 INFO:teuthology.orchestra.run.smithi086.stdout: "command_send": 0, 2024-03-04T12:36:34.681 INFO:teuthology.orchestra.run.smithi086.stdout: "command_resend": 0, 2024-03-04T12:36:34.681 INFO:teuthology.orchestra.run.smithi086.stdout: "map_epoch": 105, 2024-03-04T12:36:34.681 INFO:teuthology.orchestra.run.smithi086.stdout: "map_full": 0, 2024-03-04T12:36:34.681 INFO:teuthology.orchestra.run.smithi086.stdout: "map_inc": 16, 2024-03-04T12:36:34.681 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_sessions": 9, 2024-03-04T12:36:34.681 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_session_open": 9, 2024-03-04T12:36:34.681 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_session_close": 0, 2024-03-04T12:36:34.681 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_laggy": 0, 2024-03-04T12:36:34.681 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_wr": 14, 2024-03-04T12:36:34.682 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_rd": 0, 2024-03-04T12:36:34.682 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_del": 1 2024-03-04T12:36:34.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.682 INFO:teuthology.orchestra.run.smithi086.stdout: "oft": { 2024-03-04T12:36:34.682 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_objs": 1, 2024-03-04T12:36:34.683 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_kv_pairs": 0, 2024-03-04T12:36:34.683 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_updates": 0, 2024-03-04T12:36:34.683 INFO:teuthology.orchestra.run.smithi086.stdout: "omap_total_removes": 0 2024-03-04T12:36:34.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.683 INFO:teuthology.orchestra.run.smithi086.stdout: "osd-slow-ops": { 2024-03-04T12:36:34.683 INFO:teuthology.orchestra.run.smithi086.stdout: "slow_ops_count": 0 2024-03-04T12:36:34.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.683 INFO:teuthology.orchestra.run.smithi086.stdout: "purge_queue": { 2024-03-04T12:36:34.683 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_ops": 0, 2024-03-04T12:36:34.684 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_ops_high_water": 0, 2024-03-04T12:36:34.684 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing": 0, 2024-03-04T12:36:34.684 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executing_high_water": 0, 2024-03-04T12:36:34.684 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executed_ops": 0, 2024-03-04T12:36:34.684 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_executed": 0, 2024-03-04T12:36:34.684 INFO:teuthology.orchestra.run.smithi086.stdout: "pq_item_in_journal": 0 2024-03-04T12:36:34.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.684 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-04T12:36:34.685 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:34.685 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-03-04T12:36:34.685 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:34.685 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 1103, 2024-03-04T12:36:34.685 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 1980806, 2024-03-04T12:36:34.685 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:34.685 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 1103, 2024-03-04T12:36:34.685 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-03-04T12:36:34.685 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-03-04T12:36:34.686 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 1103, 2024-03-04T12:36:34.686 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 1980806, 2024-03-04T12:36:34.686 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:34.686 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.686 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.686 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.686 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:34.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.687 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-objecter_bytes": { 2024-03-04T12:36:34.687 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:34.687 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 104857600, 2024-03-04T12:36:34.687 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:34.687 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:34.687 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:34.688 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:34.688 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:34.688 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 40, 2024-03-04T12:36:34.688 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 11813, 2024-03-04T12:36:34.688 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 40, 2024-03-04T12:36:34.688 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 11813, 2024-03-04T12:36:34.688 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:34.688 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.688 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.689 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.689 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:34.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.689 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-objecter_ops": { 2024-03-04T12:36:34.689 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:34.689 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 1024, 2024-03-04T12:36:34.689 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:34.689 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:34.689 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:34.690 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:34.690 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:34.690 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 40, 2024-03-04T12:36:34.690 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 40, 2024-03-04T12:36:34.690 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 40, 2024-03-04T12:36:34.690 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 40, 2024-03-04T12:36:34.690 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:34.690 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.690 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.691 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.691 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:34.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.691 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-write_buf_throttle": { 2024-03-04T12:36:34.691 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:34.691 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 3758096384, 2024-03-04T12:36:34.691 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:34.691 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 0, 2024-03-04T12:36:34.692 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 0, 2024-03-04T12:36:34.692 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:34.692 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 0, 2024-03-04T12:36:34.692 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-03-04T12:36:34.692 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-03-04T12:36:34.692 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 0, 2024-03-04T12:36:34.692 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 0, 2024-03-04T12:36:34.692 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:34.692 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.693 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.693 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.693 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:34.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-04T12:36:34.693 INFO:teuthology.orchestra.run.smithi086.stdout: "throttle-write_buf_throttle-0x56201113a640": { 2024-03-04T12:36:34.693 INFO:teuthology.orchestra.run.smithi086.stdout: "val": 0, 2024-03-04T12:36:34.693 INFO:teuthology.orchestra.run.smithi086.stdout: "max": 3758096384, 2024-03-04T12:36:34.693 INFO:teuthology.orchestra.run.smithi086.stdout: "get_started": 0, 2024-03-04T12:36:34.693 INFO:teuthology.orchestra.run.smithi086.stdout: "get": 5, 2024-03-04T12:36:34.694 INFO:teuthology.orchestra.run.smithi086.stdout: "get_sum": 1511, 2024-03-04T12:36:34.694 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_fail": 0, 2024-03-04T12:36:34.694 INFO:teuthology.orchestra.run.smithi086.stdout: "get_or_fail_success": 5, 2024-03-04T12:36:34.694 INFO:teuthology.orchestra.run.smithi086.stdout: "take": 0, 2024-03-04T12:36:34.694 INFO:teuthology.orchestra.run.smithi086.stdout: "take_sum": 0, 2024-03-04T12:36:34.694 INFO:teuthology.orchestra.run.smithi086.stdout: "put": 4, 2024-03-04T12:36:34.694 INFO:teuthology.orchestra.run.smithi086.stdout: "put_sum": 1511, 2024-03-04T12:36:34.694 INFO:teuthology.orchestra.run.smithi086.stdout: "wait": { 2024-03-04T12:36:34.694 INFO:teuthology.orchestra.run.smithi086.stdout: "avgcount": 0, 2024-03-04T12:36:34.695 INFO:teuthology.orchestra.run.smithi086.stdout: "sum": 0.000000000, 2024-03-04T12:36:34.695 INFO:teuthology.orchestra.run.smithi086.stdout: "avgtime": 0.000000000 2024-03-04T12:36:34.695 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:34.695 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-04T12:36:34.695 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-04T12:36:35.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:35 smithi086 ceph-mon[36851]: pgmap v334: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-04T12:36:35.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:35 smithi063 ceph-mon[36717]: pgmap v334: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-04T12:36:35.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:35 smithi060 ceph-mon[32745]: pgmap v334: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-04T12:36:35.447 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-04T12:36:35.447 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-04T12:36:35.448 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=2 2024-03-04T12:36:35.448 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-04T12:36:35.448 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-04T12:36:35.448 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-03-04T12:36:35.448 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-04T12:36:35.460 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-04T12:36:35.460 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:36:35.460 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T12:36:35.482 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:36:35.482 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T12:36:35.541 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:36:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.692+0000 7fa773043640 1 -- 172.21.15.60:0/2327344828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c105520 msgr2=0x7fa76c10d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.692+0000 7fa773043640 1 --2- 172.21.15.60:0/2327344828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c105520 0x7fa76c10d0f0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009900 tx=0x7fa75c02bc30 comp rx=0 tx=0).stop 2024-03-04T12:36:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.692+0000 7fa773043640 1 -- 172.21.15.60:0/2327344828 shutdown_connections 2024-03-04T12:36:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.692+0000 7fa773043640 1 --2- 172.21.15.60:0/2327344828 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c105520 0x7fa76c10d0f0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.692+0000 7fa773043640 1 --2- 172.21.15.60:0/2327344828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa76c100e00 0x7fa76c104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.692+0000 7fa773043640 1 --2- 172.21.15.60:0/2327344828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa76c100450 0x7fa76c100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:35.701 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.692+0000 7fa773043640 1 -- 172.21.15.60:0/2327344828 >> 172.21.15.60:0/2327344828 conn(0x7fa76c0773b0 msgr2=0x7fa76c0777c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:35.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.692+0000 7fa773043640 1 -- 172.21.15.60:0/2327344828 shutdown_connections 2024-03-04T12:36:35.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.692+0000 7fa773043640 1 -- 172.21.15.60:0/2327344828 wait complete. 2024-03-04T12:36:35.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa773043640 1 Processor -- start 2024-03-04T12:36:35.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa773043640 1 -- start start 2024-03-04T12:36:35.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa773043640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa76c100450 0x7fa76c19d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:35.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa773043640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa76c105520 0x7fa76c19dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:35.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa773043640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c197960 0x7fa76c197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:35.702 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa773043640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c111710 con 0x7fa76c197960 2024-03-04T12:36:35.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa773043640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c111590 con 0x7fa76c105520 2024-03-04T12:36:35.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa773043640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c111890 con 0x7fa76c100450 2024-03-04T12:36:35.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa7715e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c197960 0x7fa76c197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:35.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa7715e0640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c197960 0x7fa76c197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58858/0 (socket says 172.21.15.60:58858) 2024-03-04T12:36:35.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa7715e0640 1 -- 172.21.15.60:0/2905691886 learned_addr learned my addr 172.21.15.60:0/2905691886 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:35.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa76bfff640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa76c105520 0x7fa76c19dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:35.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa770ddf640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa76c100450 0x7fa76c19d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:35.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa7715e0640 1 -- 172.21.15.60:0/2905691886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa76c100450 msgr2=0x7fa76c19d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:35.703 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa7715e0640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa76c100450 0x7fa76c19d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:35.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa7715e0640 1 -- 172.21.15.60:0/2905691886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa76c105520 msgr2=0x7fa76c19dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:35.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa7715e0640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa76c105520 0x7fa76c19dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:35.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.693+0000 7fa7715e0640 1 -- 172.21.15.60:0/2905691886 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c198480 con 0x7fa76c197960 2024-03-04T12:36:35.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.694+0000 7fa770ddf640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa76c100450 0x7fa76c19d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:36:35.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.694+0000 7fa7715e0640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c197960 0x7fa76c197e10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009230 tx=0x7fa75c02cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:35.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.694+0000 7fa769ffb640 1 -- 172.21.15.60:0/2905691886 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c041070 con 0x7fa76c197960 2024-03-04T12:36:35.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.694+0000 7fa769ffb640 1 -- 172.21.15.60:0/2905691886 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa75c004450 con 0x7fa76c197960 2024-03-04T12:36:35.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.694+0000 7fa773043640 1 -- 172.21.15.60:0/2905691886 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa76c198710 con 0x7fa76c197960 2024-03-04T12:36:35.704 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.694+0000 7fa769ffb640 1 -- 172.21.15.60:0/2905691886 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c034440 con 0x7fa76c197960 2024-03-04T12:36:35.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.694+0000 7fa773043640 1 -- 172.21.15.60:0/2905691886 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa76c1a4900 con 0x7fa76c197960 2024-03-04T12:36:35.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.695+0000 7fa773043640 1 -- 172.21.15.60:0/2905691886 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa76c0681b0 con 0x7fa76c197960 2024-03-04T12:36:35.705 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.696+0000 7fa769ffb640 1 -- 172.21.15.60:0/2905691886 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7fa75c044070 con 0x7fa76c197960 2024-03-04T12:36:35.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.697+0000 7fa769ffb640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa7580a1ef0 0x7fa7580a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:35.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.697+0000 7fa770ddf640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa7580a1ef0 0x7fa7580a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:35.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.697+0000 7fa769ffb640 1 -- 172.21.15.60:0/2905691886 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9539+0+0 (secure 0 0 0) 0x7fa75c0f0130 con 0x7fa76c197960 2024-03-04T12:36:35.706 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.697+0000 7fa770ddf640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa7580a1ef0 0x7fa7580a43b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa76c198aa0 tx=0x7fa764009520 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:35.707 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.701+0000 7fa769ffb640 1 -- 172.21.15.60:0/2905691886 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa75c0bced0 con 0x7fa76c197960 2024-03-04T12:36:35.889 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.883+0000 7fa773043640 1 -- 172.21.15.60:0/2905691886 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa76c100e00 con 0x7fa76c197960 2024-03-04T12:36:35.890 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.885+0000 7fa769ffb640 1 -- 172.21.15.60:0/2905691886 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+25187 (secure 0 0 0) 0x7fa75c0bc7d0 con 0x7fa76c197960 2024-03-04T12:36:35.891 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:36:35.891 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":105,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:36:15.046092+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2167195614":"2024-03-05T12:25:53.138536+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/1914863179":"2024-03-05T12:25:53.138536+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3302898178":"2024-03-05T12:25:53.138536+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/2604977412":"2024-03-05T12:25:53.138536+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/1914863179":"2024-03-05T12:25:53.138536+0000","172.21.15.60:0/366921369":"2024-03-05T12:25:53.138536+0000","172.21.15.60:0/1342937290":"2024-03-05T12:25:53.138536+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:36:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.887+0000 7fa773043640 1 -- 172.21.15.60:0/2905691886 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa7580a1ef0 msgr2=0x7fa7580a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.887+0000 7fa773043640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa7580a1ef0 0x7fa7580a43b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa76c198aa0 tx=0x7fa764009520 comp rx=0 tx=0).stop 2024-03-04T12:36:35.893 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.887+0000 7fa773043640 1 -- 172.21.15.60:0/2905691886 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c197960 msgr2=0x7fa76c197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.887+0000 7fa773043640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c197960 0x7fa76c197e10 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009230 tx=0x7fa75c02cbf0 comp rx=0 tx=0).stop 2024-03-04T12:36:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.887+0000 7fa773043640 1 -- 172.21.15.60:0/2905691886 shutdown_connections 2024-03-04T12:36:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.887+0000 7fa773043640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fa7580a1ef0 0x7fa7580a43b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.887+0000 7fa773043640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fa76c197960 0x7fa76c197e10 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.887+0000 7fa773043640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa76c105520 0x7fa76c19dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.887+0000 7fa773043640 1 --2- 172.21.15.60:0/2905691886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa76c100450 0x7fa76c19d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.887+0000 7fa773043640 1 -- 172.21.15.60:0/2905691886 >> 172.21.15.60:0/2905691886 conn(0x7fa76c0773b0 msgr2=0x7fa76c0fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.887+0000 7fa773043640 1 -- 172.21.15.60:0/2905691886 shutdown_connections 2024-03-04T12:36:35.894 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:35.887+0000 7fa773043640 1 -- 172.21.15.60:0/2905691886 wait complete. 2024-03-04T12:36:35.904 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T12:18:28.414510+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-04T12:23:10.438928+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-04T12:23:11.472998+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 104, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-04T12:23:15.162500+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 104, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-04T12:36:35.905 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-04T12:36:36.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 -- 172.21.15.60:0/3172761121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6278100df0 msgr2=0x7f6278104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 --2- 172.21.15.60:0/3172761121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6278100df0 0x7f6278104e30 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6264009900 tx=0x7f626402bc30 comp rx=0 tx=0).stop 2024-03-04T12:36:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 -- 172.21.15.60:0/3172761121 shutdown_connections 2024-03-04T12:36:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 --2- 172.21.15.60:0/3172761121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6278105510 0x7f627810d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 --2- 172.21.15.60:0/3172761121 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6278100df0 0x7f6278104e30 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 --2- 172.21.15.60:0/3172761121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6278100440 0x7f6278100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 -- 172.21.15.60:0/3172761121 >> 172.21.15.60:0/3172761121 conn(0x7f62780773e0 msgr2=0x7f62780777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 -- 172.21.15.60:0/3172761121 shutdown_connections 2024-03-04T12:36:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 -- 172.21.15.60:0/3172761121 wait complete. 2024-03-04T12:36:36.013 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 Processor -- start 2024-03-04T12:36:36.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 -- start start 2024-03-04T12:36:36.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6278100440 0x7f627819d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6278105510 0x7f627819dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62781979f0 0x7f6278197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62781105a0 con 0x7f6278105510 2024-03-04T12:36:36.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6278110420 con 0x7f6278100440 2024-03-04T12:36:36.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f627ec75640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6278110720 con 0x7f62781979f0 2024-03-04T12:36:36.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.005+0000 7f6277fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6278105510 0x7f627819dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f6277fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6278105510 0x7f627819dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58884/0 (socket says 172.21.15.60:58884) 2024-03-04T12:36:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f6277fff640 1 -- 172.21.15.60:0/3751653874 learned_addr learned my addr 172.21.15.60:0/3751653874 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f627d212640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62781979f0 0x7f6278197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f627ca11640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6278100440 0x7f627819d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f6277fff640 1 -- 172.21.15.60:0/3751653874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62781979f0 msgr2=0x7f6278197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f6277fff640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62781979f0 0x7f6278197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f6277fff640 1 -- 172.21.15.60:0/3751653874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6278100440 msgr2=0x7f627819d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f6277fff640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6278100440 0x7f627819d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.015 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f6277fff640 1 -- 172.21.15.60:0/3751653874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6278198510 con 0x7f6278105510 2024-03-04T12:36:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f6277fff640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6278105510 0x7f627819dd80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6264036040 tx=0x7f626402cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f6275ffb640 1 -- 172.21.15.60:0/3751653874 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6264041070 con 0x7f6278105510 2024-03-04T12:36:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f6275ffb640 1 -- 172.21.15.60:0/3751653874 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6264004450 con 0x7f6278105510 2024-03-04T12:36:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f627ec75640 1 -- 172.21.15.60:0/3751653874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62781987a0 con 0x7f6278105510 2024-03-04T12:36:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f6275ffb640 1 -- 172.21.15.60:0/3751653874 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6264003dc0 con 0x7f6278105510 2024-03-04T12:36:36.016 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.006+0000 7f627ec75640 1 -- 172.21.15.60:0/3751653874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62781a49f0 con 0x7f6278105510 2024-03-04T12:36:36.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.007+0000 7f627ec75640 1 -- 172.21.15.60:0/3751653874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62780681b0 con 0x7f6278105510 2024-03-04T12:36:36.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.008+0000 7f6275ffb640 1 -- 172.21.15.60:0/3751653874 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f6264044070 con 0x7f6278105510 2024-03-04T12:36:36.017 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.009+0000 7f6275ffb640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f62680a1f70 0x7f62680a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.009+0000 7f6275ffb640 1 -- 172.21.15.60:0/3751653874 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f62640f00b0 con 0x7f6278105510 2024-03-04T12:36:36.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.009+0000 7f627ca11640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f62680a1f70 0x7f62680a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.010+0000 7f627ca11640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f62680a1f70 0x7f62680a4430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6278198b30 tx=0x7f626c005f90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:36.018 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.013+0000 7f6275ffb640 1 -- 172.21.15.60:0/3751653874 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f62640bc860 con 0x7f6278105510 2024-03-04T12:36:36.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:36 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2905691886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:36:36.203 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.198+0000 7f627ec75640 1 -- 172.21.15.60:0/3751653874 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6278100fd0 con 0x7f6278105510 2024-03-04T12:36:36.204 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.198+0000 7f6275ffb640 1 -- 172.21.15.60:0/3751653874 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7f62640c0900 con 0x7f6278105510 2024-03-04T12:36:36.204 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2024-03-04T12:36:36.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.200+0000 7f627ec75640 1 -- 172.21.15.60:0/3751653874 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f62680a1f70 msgr2=0x7f62680a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.200+0000 7f627ec75640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f62680a1f70 0x7f62680a4430 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6278198b30 tx=0x7f626c005f90 comp rx=0 tx=0).stop 2024-03-04T12:36:36.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.200+0000 7f627ec75640 1 -- 172.21.15.60:0/3751653874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6278105510 msgr2=0x7f627819dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.206 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.200+0000 7f627ec75640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6278105510 0x7f627819dd80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f6264036040 tx=0x7f626402cb20 comp rx=0 tx=0).stop 2024-03-04T12:36:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.200+0000 7f627ec75640 1 -- 172.21.15.60:0/3751653874 shutdown_connections 2024-03-04T12:36:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.200+0000 7f627ec75640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62781979f0 0x7f6278197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.200+0000 7f627ec75640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f62680a1f70 0x7f62680a4430 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.200+0000 7f627ec75640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f6278105510 0x7f627819dd80 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.200+0000 7f627ec75640 1 --2- 172.21.15.60:0/3751653874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6278100440 0x7f627819d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.200+0000 7f627ec75640 1 -- 172.21.15.60:0/3751653874 >> 172.21.15.60:0/3751653874 conn(0x7f62780773e0 msgr2=0x7f62781066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.201+0000 7f627ec75640 1 -- 172.21.15.60:0/3751653874 shutdown_connections 2024-03-04T12:36:36.207 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.201+0000 7f627ec75640 1 -- 172.21.15.60:0/3751653874 wait complete. 2024-03-04T12:36:36.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-04T12:36:36.325 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.317+0000 7f7160803640 1 -- 172.21.15.60:0/2566368461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f715c102be0 msgr2=0x7f715c10b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.317+0000 7f7160803640 1 --2- 172.21.15.60:0/2566368461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f715c102be0 0x7f715c10b0c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7144009900 tx=0x7f714402bcd0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.317+0000 7f7160803640 1 -- 172.21.15.60:0/2566368461 shutdown_connections 2024-03-04T12:36:36.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.317+0000 7f7160803640 1 --2- 172.21.15.60:0/2566368461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f715c10b6e0 0x7f715c10db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.317+0000 7f7160803640 1 --2- 172.21.15.60:0/2566368461 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f715c102be0 0x7f715c10b0c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.317+0000 7f7160803640 1 --2- 172.21.15.60:0/2566368461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f715c1022c0 0x7f715c1026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.317+0000 7f7160803640 1 -- 172.21.15.60:0/2566368461 >> 172.21.15.60:0/2566368461 conn(0x7f715c076dc0 msgr2=0x7f715c0fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:36.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.318+0000 7f7160803640 1 -- 172.21.15.60:0/2566368461 shutdown_connections 2024-03-04T12:36:36.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.318+0000 7f7160803640 1 -- 172.21.15.60:0/2566368461 wait complete. 2024-03-04T12:36:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.318+0000 7f7160803640 1 Processor -- start 2024-03-04T12:36:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.318+0000 7f7160803640 1 -- start start 2024-03-04T12:36:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f7160803640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f715c1022c0 0x7f715c19d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f7160803640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f715c102be0 0x7f715c19dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f7160803640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f715c10b6e0 0x7f715c197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f7160803640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c1114b0 con 0x7f715c1022c0 2024-03-04T12:36:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f7160803640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c111330 con 0x7f715c102be0 2024-03-04T12:36:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f7160803640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c111630 con 0x7f715c10b6e0 2024-03-04T12:36:36.327 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f715ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f715c1022c0 0x7f715c19d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f715ad9d640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f715c1022c0 0x7f715c19d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58912/0 (socket says 172.21.15.60:58912) 2024-03-04T12:36:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f715ad9d640 1 -- 172.21.15.60:0/3244840780 learned_addr learned my addr 172.21.15.60:0/3244840780 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f715b59e640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f715c10b6e0 0x7f715c197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f715a59c640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f715c102be0 0x7f715c19dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f715ad9d640 1 -- 172.21.15.60:0/3244840780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f715c10b6e0 msgr2=0x7f715c197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f715ad9d640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f715c10b6e0 0x7f715c197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f715ad9d640 1 -- 172.21.15.60:0/3244840780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f715c102be0 msgr2=0x7f715c19dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f715ad9d640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f715c102be0 0x7f715c19dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f715ad9d640 1 -- 172.21.15.60:0/3244840780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c198010 con 0x7f715c1022c0 2024-03-04T12:36:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f715a59c640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f715c102be0 0x7f715c19dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:36:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f715ad9d640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f715c1022c0 0x7f715c19d690 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f715400c920 tx=0x7f715400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f7153fff640 1 -- 172.21.15.60:0/3244840780 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154007c70 con 0x7f715c1022c0 2024-03-04T12:36:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f7153fff640 1 -- 172.21.15.60:0/3244840780 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7154007e10 con 0x7f715c1022c0 2024-03-04T12:36:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f7160803640 1 -- 172.21.15.60:0/3244840780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715c198300 con 0x7f715c1022c0 2024-03-04T12:36:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f7153fff640 1 -- 172.21.15.60:0/3244840780 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154018890 con 0x7f715c1022c0 2024-03-04T12:36:36.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.319+0000 7f7160803640 1 -- 172.21.15.60:0/3244840780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715c1a4610 con 0x7f715c1022c0 2024-03-04T12:36:36.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.320+0000 7f7160803640 1 -- 172.21.15.60:0/3244840780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715c068ed0 con 0x7f715c1022c0 2024-03-04T12:36:36.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.322+0000 7f7153fff640 1 -- 172.21.15.60:0/3244840780 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f7154018a50 con 0x7f715c1022c0 2024-03-04T12:36:36.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.322+0000 7f7153fff640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f71480a1ef0 0x7f71480a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.322+0000 7f715a59c640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f71480a1ef0 0x7f71480a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.322+0000 7f7153fff640 1 -- 172.21.15.60:0/3244840780 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f71540ce970 con 0x7f715c1022c0 2024-03-04T12:36:36.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.323+0000 7f715a59c640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f71480a1ef0 0x7f71480a43b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f714402efd0 tx=0x7f7144000bc0 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:36.331 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.326+0000 7f7153fff640 1 -- 172.21.15.60:0/3244840780 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7154014030 con 0x7f715c1022c0 2024-03-04T12:36:36.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:36 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2905691886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:36:36.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:36 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2905691886' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:36:36.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.508+0000 7f7160803640 1 -- 172.21.15.60:0/3244840780 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f715c1a47e0 con 0x7f715c1022c0 2024-03-04T12:36:36.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.509+0000 7f7153fff640 1 -- 172.21.15.60:0/3244840780 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v105) v1 ==== 104+0+11 (secure 0 0 0) 0x7f715409b6f0 con 0x7f715c1022c0 2024-03-04T12:36:36.515 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:36:36.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.511+0000 7f7160803640 1 -- 172.21.15.60:0/3244840780 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f71480a1ef0 msgr2=0x7f71480a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.511+0000 7f7160803640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f71480a1ef0 0x7f71480a43b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f714402efd0 tx=0x7f7144000bc0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.511+0000 7f7160803640 1 -- 172.21.15.60:0/3244840780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f715c1022c0 msgr2=0x7f715c19d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.511+0000 7f7160803640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f715c1022c0 0x7f715c19d690 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f715400c920 tx=0x7f715400cde0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.511+0000 7f7160803640 1 -- 172.21.15.60:0/3244840780 shutdown_connections 2024-03-04T12:36:36.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.511+0000 7f7160803640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f715c10b6e0 0x7f715c197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.517 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.511+0000 7f7160803640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f715c102be0 0x7f715c19dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.511+0000 7f7160803640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f71480a1ef0 0x7f71480a43b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.511+0000 7f7160803640 1 --2- 172.21.15.60:0/3244840780 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f715c1022c0 0x7f715c19d690 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.511+0000 7f7160803640 1 -- 172.21.15.60:0/3244840780 >> 172.21.15.60:0/3244840780 conn(0x7f715c076dc0 msgr2=0x7f715c0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:36.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.511+0000 7f7160803640 1 -- 172.21.15.60:0/3244840780 shutdown_connections 2024-03-04T12:36:36.518 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.511+0000 7f7160803640 1 -- 172.21.15.60:0/3244840780 wait complete. 2024-03-04T12:36:36.527 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-04T12:36:36.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.627+0000 7f35b3fa9640 1 -- 172.21.15.60:0/3338281415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35ac105520 msgr2=0x7f35ac10d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.638 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.627+0000 7f35b3fa9640 1 --2- 172.21.15.60:0/3338281415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35ac105520 0x7f35ac10d0f0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f359c009ca0 tx=0x7f359c02c0f0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.627+0000 7f35b3fa9640 1 -- 172.21.15.60:0/3338281415 shutdown_connections 2024-03-04T12:36:36.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.627+0000 7f35b3fa9640 1 --2- 172.21.15.60:0/3338281415 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35ac105520 0x7f35ac10d0f0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.627+0000 7f35b3fa9640 1 --2- 172.21.15.60:0/3338281415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35ac100e50 0x7f35ac104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.639 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.627+0000 7f35b3fa9640 1 --2- 172.21.15.60:0/3338281415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35ac1004a0 0x7f35ac100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.627+0000 7f35b3fa9640 1 -- 172.21.15.60:0/3338281415 >> 172.21.15.60:0/3338281415 conn(0x7f35ac077400 msgr2=0x7f35ac077810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:36.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.627+0000 7f35b3fa9640 1 -- 172.21.15.60:0/3338281415 shutdown_connections 2024-03-04T12:36:36.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.627+0000 7f35b3fa9640 1 -- 172.21.15.60:0/3338281415 wait complete. 2024-03-04T12:36:36.640 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.628+0000 7f35b3fa9640 1 Processor -- start 2024-03-04T12:36:36.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.628+0000 7f35b3fa9640 1 -- start start 2024-03-04T12:36:36.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.628+0000 7f35b3fa9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35ac1004a0 0x7f35ac19c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.628+0000 7f35b3fa9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35ac105520 0x7f35ac19cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.628+0000 7f35b3fa9640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35ac196840 0x7f35ac196cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.628+0000 7f35b3fa9640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac1105c0 con 0x7f35ac196840 2024-03-04T12:36:36.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.628+0000 7f35b3fa9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac110440 con 0x7f35ac105520 2024-03-04T12:36:36.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.628+0000 7f35b3fa9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ac110740 con 0x7f35ac1004a0 2024-03-04T12:36:36.642 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.628+0000 7f35b2546640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35ac196840 0x7f35ac196cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.628+0000 7f35b2546640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35ac196840 0x7f35ac196cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58934/0 (socket says 172.21.15.60:58934) 2024-03-04T12:36:36.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.628+0000 7f35b2546640 1 -- 172.21.15.60:0/2820714611 learned_addr learned my addr 172.21.15.60:0/2820714611 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:36.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.628+0000 7f35b1544640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35ac105520 0x7f35ac19cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.629+0000 7f35b1d45640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35ac1004a0 0x7f35ac19c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.629+0000 7f35b2546640 1 -- 172.21.15.60:0/2820714611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35ac1004a0 msgr2=0x7f35ac19c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.629+0000 7f35b2546640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35ac1004a0 0x7f35ac19c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.629+0000 7f35b2546640 1 -- 172.21.15.60:0/2820714611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35ac105520 msgr2=0x7f35ac19cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.644 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.629+0000 7f35b2546640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35ac105520 0x7f35ac19cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.629+0000 7f35b2546640 1 -- 172.21.15.60:0/2820714611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac1973d0 con 0x7f35ac196840 2024-03-04T12:36:36.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.629+0000 7f35b2546640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35ac196840 0x7f35ac196cf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f359c02c700 tx=0x7f359c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:36.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.629+0000 7f35aaffd640 1 -- 172.21.15.60:0/2820714611 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c041020 con 0x7f35ac196840 2024-03-04T12:36:36.645 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.629+0000 7f35aaffd640 1 -- 172.21.15.60:0/2820714611 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f359c034cd0 con 0x7f35ac196840 2024-03-04T12:36:36.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.629+0000 7f35b3fa9640 1 -- 172.21.15.60:0/2820714611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ac197660 con 0x7f35ac196840 2024-03-04T12:36:36.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.629+0000 7f35aaffd640 1 -- 172.21.15.60:0/2820714611 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c039040 con 0x7f35ac196840 2024-03-04T12:36:36.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.629+0000 7f35b3fa9640 1 -- 172.21.15.60:0/2820714611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ac1a3810 con 0x7f35ac196840 2024-03-04T12:36:36.646 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.630+0000 7f35b3fa9640 1 -- 172.21.15.60:0/2820714611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ac101920 con 0x7f35ac196840 2024-03-04T12:36:36.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.631+0000 7f35aaffd640 1 -- 172.21.15.60:0/2820714611 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f359c044020 con 0x7f35ac196840 2024-03-04T12:36:36.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.632+0000 7f35aaffd640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f35940a1ef0 0x7f35940a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.632+0000 7f35b1d45640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f35940a1ef0 0x7f35940a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.647 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.632+0000 7f35aaffd640 1 -- 172.21.15.60:0/2820714611 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f359c0f05e0 con 0x7f35ac196840 2024-03-04T12:36:36.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.632+0000 7f35b1d45640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f35940a1ef0 0x7f35940a43b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f35a4005fd0 tx=0x7f35a4005850 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:36.648 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.635+0000 7f35aaffd640 1 -- 172.21.15.60:0/2820714611 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f359c04b020 con 0x7f35ac196840 2024-03-04T12:36:36.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.817+0000 7f35b3fa9640 1 -- 172.21.15.60:0/2820714611 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f35ac100e50 con 0x7f35ac196840 2024-03-04T12:36:36.823 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.817+0000 7f35aaffd640 1 -- 172.21.15.60:0/2820714611 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v105) v1 ==== 100+0+11 (secure 0 0 0) 0x7f359c0bd500 con 0x7f35ac196840 2024-03-04T12:36:36.824 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 64 2024-03-04T12:36:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.819+0000 7f35b3fa9640 1 -- 172.21.15.60:0/2820714611 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f35940a1ef0 msgr2=0x7f35940a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.819+0000 7f35b3fa9640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f35940a1ef0 0x7f35940a43b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f35a4005fd0 tx=0x7f35a4005850 comp rx=0 tx=0).stop 2024-03-04T12:36:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.819+0000 7f35b3fa9640 1 -- 172.21.15.60:0/2820714611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35ac196840 msgr2=0x7f35ac196cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.825 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.819+0000 7f35b3fa9640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35ac196840 0x7f35ac196cf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f359c02c700 tx=0x7f359c032f20 comp rx=0 tx=0).stop 2024-03-04T12:36:36.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.819+0000 7f35b3fa9640 1 -- 172.21.15.60:0/2820714611 shutdown_connections 2024-03-04T12:36:36.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.819+0000 7f35b3fa9640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f35940a1ef0 0x7f35940a43b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.819+0000 7f35b3fa9640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f35ac196840 0x7f35ac196cf0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.819+0000 7f35b3fa9640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35ac105520 0x7f35ac19cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.819+0000 7f35b3fa9640 1 --2- 172.21.15.60:0/2820714611 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35ac1004a0 0x7f35ac19c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.819+0000 7f35b3fa9640 1 -- 172.21.15.60:0/2820714611 >> 172.21.15.60:0/2820714611 conn(0x7f35ac077400 msgr2=0x7f35ac0fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:36.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.819+0000 7f35b3fa9640 1 -- 172.21.15.60:0/2820714611 shutdown_connections 2024-03-04T12:36:36.826 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.819+0000 7f35b3fa9640 1 -- 172.21.15.60:0/2820714611 wait complete. 2024-03-04T12:36:36.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-04T12:36:36.943 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.936+0000 7f67eec97640 1 -- 172.21.15.60:0/3464058430 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67e8105500 msgr2=0x7f67e810d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.936+0000 7f67eec97640 1 --2- 172.21.15.60:0/3464058430 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67e8105500 0x7f67e810d0d0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009ca0 tx=0x7f67e002c0f0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.936+0000 7f67eec97640 1 -- 172.21.15.60:0/3464058430 shutdown_connections 2024-03-04T12:36:36.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.936+0000 7f67eec97640 1 --2- 172.21.15.60:0/3464058430 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67e8105500 0x7f67e810d0d0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.936+0000 7f67eec97640 1 --2- 172.21.15.60:0/3464058430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e8100de0 0x7f67e8104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.936+0000 7f67eec97640 1 --2- 172.21.15.60:0/3464058430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67e8100430 0x7f67e8100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.944 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.936+0000 7f67eec97640 1 -- 172.21.15.60:0/3464058430 >> 172.21.15.60:0/3464058430 conn(0x7f67e8077350 msgr2=0x7f67e8077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:36.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.936+0000 7f67eec97640 1 -- 172.21.15.60:0/3464058430 shutdown_connections 2024-03-04T12:36:36.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.936+0000 7f67eec97640 1 -- 172.21.15.60:0/3464058430 wait complete. 2024-03-04T12:36:36.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.936+0000 7f67eec97640 1 Processor -- start 2024-03-04T12:36:36.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.936+0000 7f67eec97640 1 -- start start 2024-03-04T12:36:36.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67eec97640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67e8100430 0x7f67e819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67eec97640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67e8105500 0x7f67e819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67eec97640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e81968a0 0x7f67e8196d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.945 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67eec97640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e81105c0 con 0x7f67e8105500 2024-03-04T12:36:36.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67eec97640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e8110440 con 0x7f67e8100430 2024-03-04T12:36:36.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67eec97640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e8110740 con 0x7f67e81968a0 2024-03-04T12:36:36.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67e7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67e8105500 0x7f67e819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67e7fff640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67e8105500 0x7f67e819cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58964/0 (socket says 172.21.15.60:58964) 2024-03-04T12:36:36.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67e7fff640 1 -- 172.21.15.60:0/2319603751 learned_addr learned my addr 172.21.15.60:0/2319603751 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:36.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67eca33640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67e8100430 0x7f67e819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67e7fff640 1 -- 172.21.15.60:0/2319603751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e81968a0 msgr2=0x7f67e8196d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.946 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67e7fff640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e81968a0 0x7f67e8196d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67e7fff640 1 -- 172.21.15.60:0/2319603751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67e8100430 msgr2=0x7f67e819c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:36.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67e7fff640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67e8100430 0x7f67e819c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:36.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67e7fff640 1 -- 172.21.15.60:0/2319603751 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e81973c0 con 0x7f67e8105500 2024-03-04T12:36:36.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67e7fff640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67e8105500 0x7f67e819cc30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f67d400d740 tx=0x7f67d400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:36.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67e5ffb640 1 -- 172.21.15.60:0/2319603751 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d4017070 con 0x7f67e8105500 2024-03-04T12:36:36.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67e5ffb640 1 -- 172.21.15.60:0/2319603751 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67d400de50 con 0x7f67e8105500 2024-03-04T12:36:36.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67eec97640 1 -- 172.21.15.60:0/2319603751 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e81976b0 con 0x7f67e8105500 2024-03-04T12:36:36.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.937+0000 7f67e5ffb640 1 -- 172.21.15.60:0/2319603751 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d4004e00 con 0x7f67e8105500 2024-03-04T12:36:36.947 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.938+0000 7f67eec97640 1 -- 172.21.15.60:0/2319603751 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e81a38b0 con 0x7f67e8105500 2024-03-04T12:36:36.948 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.939+0000 7f67eec97640 1 -- 172.21.15.60:0/2319603751 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e8101900 con 0x7f67e8105500 2024-03-04T12:36:36.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.940+0000 7f67e5ffb640 1 -- 172.21.15.60:0/2319603751 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f67d4004fa0 con 0x7f67e8105500 2024-03-04T12:36:36.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.940+0000 7f67e5ffb640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f67d80a1ef0 0x7f67d80a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:36.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.940+0000 7f67eca33640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f67d80a1ef0 0x7f67d80a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:36.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.940+0000 7f67e5ffb640 1 -- 172.21.15.60:0/2319603751 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f67d40cded0 con 0x7f67e8105500 2024-03-04T12:36:36.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.941+0000 7f67eca33640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f67d80a1ef0 0x7f67d80a43b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f67e81979e0 tx=0x7f67dc00b410 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:36.950 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:36.945+0000 7f67e5ffb640 1 -- 172.21.15.60:0/2319603751 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f67d409abc0 con 0x7f67e8105500 2024-03-04T12:36:37.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.128+0000 7f67eec97640 1 -- 172.21.15.60:0/2319603751 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f67e8100de0 con 0x7f67e8105500 2024-03-04T12:36:37.134 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.129+0000 7f67e5ffb640 1 -- 172.21.15.60:0/2319603751 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v105) v1 ==== 103+0+11 (secure 0 0 0) 0x7f67d409a4c0 con 0x7f67e8105500 2024-03-04T12:36:37.135 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 32 2024-03-04T12:36:37.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.131+0000 7f67eec97640 1 -- 172.21.15.60:0/2319603751 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f67d80a1ef0 msgr2=0x7f67d80a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:37.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.131+0000 7f67eec97640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f67d80a1ef0 0x7f67d80a43b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f67e81979e0 tx=0x7f67dc00b410 comp rx=0 tx=0).stop 2024-03-04T12:36:37.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.131+0000 7f67eec97640 1 -- 172.21.15.60:0/2319603751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67e8105500 msgr2=0x7f67e819cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:37.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.131+0000 7f67eec97640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67e8105500 0x7f67e819cc30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f67d400d740 tx=0x7f67d400dc00 comp rx=0 tx=0).stop 2024-03-04T12:36:37.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.131+0000 7f67eec97640 1 -- 172.21.15.60:0/2319603751 shutdown_connections 2024-03-04T12:36:37.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.131+0000 7f67eec97640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e81968a0 0x7f67e8196d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.131+0000 7f67eec97640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f67d80a1ef0 0x7f67d80a43b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.131+0000 7f67eec97640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f67e8105500 0x7f67e819cc30 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.131+0000 7f67eec97640 1 --2- 172.21.15.60:0/2319603751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67e8100430 0x7f67e819c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.131+0000 7f67eec97640 1 -- 172.21.15.60:0/2319603751 >> 172.21.15.60:0/2319603751 conn(0x7f67e8077350 msgr2=0x7f67e80fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:37.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.131+0000 7f67eec97640 1 -- 172.21.15.60:0/2319603751 shutdown_connections 2024-03-04T12:36:37.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.131+0000 7f67eec97640 1 -- 172.21.15.60:0/2319603751 wait complete. 2024-03-04T12:36:37.148 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-04T12:36:37.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.246+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/2133225648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb0101620 msgr2=0x7f5fb0101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:37.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.246+0000 7f5fb4cdc640 1 --2- 172.21.15.60:0/2133225648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb0101620 0x7f5fb0101a00 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009930 tx=0x7f5fa002bc30 comp rx=0 tx=0).stop 2024-03-04T12:36:37.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.246+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/2133225648 shutdown_connections 2024-03-04T12:36:37.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.246+0000 7f5fb4cdc640 1 --2- 172.21.15.60:0/2133225648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fb01038a0 0x7f5fb010e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.246+0000 7f5fb4cdc640 1 --2- 172.21.15.60:0/2133225648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fb0102f00 0x7f5fb0103360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.252 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.246+0000 7f5fb4cdc640 1 --2- 172.21.15.60:0/2133225648 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb0101620 0x7f5fb0101a00 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.246+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/2133225648 >> 172.21.15.60:0/2133225648 conn(0x7f5fb00fcdc0 msgr2=0x7f5fb00ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:37.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.246+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/2133225648 shutdown_connections 2024-03-04T12:36:37.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.246+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/2133225648 wait complete. 2024-03-04T12:36:37.253 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.246+0000 7f5fb4cdc640 1 Processor -- start 2024-03-04T12:36:37.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fb4cdc640 1 -- start start 2024-03-04T12:36:37.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fb4cdc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb0101620 0x7f5fb019d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:37.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fb4cdc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fb0102f00 0x7f5fb019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:37.255 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fb4cdc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fb01038a0 0x7f5fb0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:37.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fb4cdc640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb0111790 con 0x7f5fb0101620 2024-03-04T12:36:37.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fb4cdc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb0111610 con 0x7f5fb0102f00 2024-03-04T12:36:37.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fb4cdc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb0111910 con 0x7f5fb01038a0 2024-03-04T12:36:37.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fae59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb0101620 0x7f5fb019d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:37.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fae59c640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb0101620 0x7f5fb019d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:58980/0 (socket says 172.21.15.60:58980) 2024-03-04T12:36:37.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fae59c640 1 -- 172.21.15.60:0/3303080264 learned_addr learned my addr 172.21.15.60:0/3303080264 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:37.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fa5d9b640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fb0102f00 0x7f5fb019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:37.256 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5faed9d640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fb01038a0 0x7f5fb0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:37.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fae59c640 1 -- 172.21.15.60:0/3303080264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fb01038a0 msgr2=0x7f5fb0197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:37.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fae59c640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fb01038a0 0x7f5fb0197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fae59c640 1 -- 172.21.15.60:0/3303080264 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fb0102f00 msgr2=0x7f5fb019de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:37.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fae59c640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fb0102f00 0x7f5fb019de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fae59c640 1 -- 172.21.15.60:0/3303080264 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb0198270 con 0x7f5fb0101620 2024-03-04T12:36:37.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.247+0000 7f5fae59c640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb0101620 0x7f5fb019d940 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009230 tx=0x7f5fa002cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:37.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.248+0000 7f5fa7fff640 1 -- 172.21.15.60:0/3303080264 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0041070 con 0x7f5fb0101620 2024-03-04T12:36:37.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.248+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/3303080264 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb0198500 con 0x7f5fb0101620 2024-03-04T12:36:37.257 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.248+0000 7f5fa7fff640 1 -- 172.21.15.60:0/3303080264 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fa0004450 con 0x7f5fb0101620 2024-03-04T12:36:37.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.248+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/3303080264 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb01a4800 con 0x7f5fb0101620 2024-03-04T12:36:37.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.248+0000 7f5fa7fff640 1 -- 172.21.15.60:0/3303080264 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0034440 con 0x7f5fb0101620 2024-03-04T12:36:37.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.249+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/3303080264 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb0198fe0 con 0x7f5fb0101620 2024-03-04T12:36:37.258 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.250+0000 7f5fa7fff640 1 -- 172.21.15.60:0/3303080264 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f5fa0044070 con 0x7f5fb0101620 2024-03-04T12:36:37.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.250+0000 7f5fa7fff640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f5f9c0a1ef0 0x7f5f9c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:37.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.250+0000 7f5fa5d9b640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f5f9c0a1ef0 0x7f5f9c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:37.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.250+0000 7f5fa7fff640 1 -- 172.21.15.60:0/3303080264 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f5fa00effb0 con 0x7f5fb0101620 2024-03-04T12:36:37.259 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.251+0000 7f5fa5d9b640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f5f9c0a1ef0 0x7f5f9c0a43b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00ff4f0 tx=0x7f5f9800b390 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:37.260 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.255+0000 7f5fa7fff640 1 -- 172.21.15.60:0/3303080264 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5fa00bcdb0 con 0x7f5fb0101620 2024-03-04T12:36:37.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:37 smithi086 ceph-mon[36851]: pgmap v335: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-03-04T12:36:37.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:37 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3751653874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:36:37.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:37 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3244840780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:36:37.349 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:37 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2820714611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:36:37.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:37 smithi063 ceph-mon[36717]: pgmap v335: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-03-04T12:36:37.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:37 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3751653874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:36:37.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:37 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3244840780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:36:37.352 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:37 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2820714611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:36:37.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:37 smithi060 ceph-mon[32745]: pgmap v335: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-03-04T12:36:37.428 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:37 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3751653874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T12:36:37.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:37 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3244840780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-04T12:36:37.429 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:37 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2820714611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-04T12:36:37.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.448+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/3303080264 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5fb00685e0 con 0x7f5fb0101620 2024-03-04T12:36:37.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.448+0000 7f5fa7fff640 1 -- 172.21.15.60:0/3303080264 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v105) v1 ==== 81+0+1002 (secure 0 0 0) 0x7f5fb00685e0 con 0x7f5fb0101620 2024-03-04T12:36:37.454 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/2167195614 2024-03-05T12:25:53.138536+0000 2024-03-04T12:36:37.455 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/2028967685 2024-03-05T12:16:04.073665+0000 2024-03-04T12:36:37.455 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6800/2061680026 2024-03-05T12:15:37.282929+0000 2024-03-04T12:36:37.455 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/3401854556 2024-03-05T12:15:37.282929+0000 2024-03-04T12:36:37.455 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6801/1914863179 2024-03-05T12:25:53.138536+0000 2024-03-04T12:36:37.455 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/758894979 2024-03-05T12:15:37.282929+0000 2024-03-04T12:36:37.455 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6800/2246601804 2024-03-05T12:16:04.073665+0000 2024-03-04T12:36:37.455 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/3539227088 2024-03-05T12:15:37.282929+0000 2024-03-04T12:36:37.455 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/3302898178 2024-03-05T12:25:53.138536+0000 2024-03-04T12:36:37.456 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6801/2246601804 2024-03-05T12:16:04.073665+0000 2024-03-04T12:36:37.456 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/2604977412 2024-03-05T12:25:53.138536+0000 2024-03-04T12:36:37.456 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6801/2061680026 2024-03-05T12:15:37.282929+0000 2024-03-04T12:36:37.456 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/3868523955 2024-03-05T12:16:04.073665+0000 2024-03-04T12:36:37.456 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/2852203003 2024-03-05T12:16:04.073665+0000 2024-03-04T12:36:37.456 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:6800/1914863179 2024-03-05T12:25:53.138536+0000 2024-03-04T12:36:37.457 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/366921369 2024-03-05T12:25:53.138536+0000 2024-03-04T12:36:37.457 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60:0/1342937290 2024-03-05T12:25:53.138536+0000 2024-03-04T12:36:37.458 INFO:teuthology.orchestra.run.smithi060.stderr:listed 17 entries 2024-03-04T12:36:37.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.450+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/3303080264 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f5f9c0a1ef0 msgr2=0x7f5f9c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:37.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.450+0000 7f5fb4cdc640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f5f9c0a1ef0 0x7f5f9c0a43b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00ff4f0 tx=0x7f5f9800b390 comp rx=0 tx=0).stop 2024-03-04T12:36:37.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.450+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/3303080264 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb0101620 msgr2=0x7f5fb019d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:37.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.450+0000 7f5fb4cdc640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb0101620 0x7f5fb019d940 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009230 tx=0x7f5fa002cc70 comp rx=0 tx=0).stop 2024-03-04T12:36:37.458 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.450+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/3303080264 shutdown_connections 2024-03-04T12:36:37.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.450+0000 7f5fb4cdc640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5fb01038a0 0x7f5fb0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.450+0000 7f5fb4cdc640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5fb0102f00 0x7f5fb019de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.450+0000 7f5fb4cdc640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f5f9c0a1ef0 0x7f5f9c0a43b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.450+0000 7f5fb4cdc640 1 --2- 172.21.15.60:0/3303080264 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f5fb0101620 0x7f5fb019d940 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:37.459 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.450+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/3303080264 >> 172.21.15.60:0/3303080264 conn(0x7f5fb00fcdc0 msgr2=0x7f5fb00fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:37.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.450+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/3303080264 shutdown_connections 2024-03-04T12:36:37.460 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:37.450+0000 7f5fb4cdc640 1 -- 172.21.15.60:0/3303080264 wait complete. 2024-03-04T12:36:37.467 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-04T12:36:37.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:36:37.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-04T12:36:37.545 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi060.front.sepia.ceph.com 2024-03-04T12:36:37.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:36:37.545 DEBUG:teuthology.orchestra.run.smithi060:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-04T12:36:37.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:36:37.569 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-03-04T12:36:37.627 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-04T12:36:37.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:36:37.627 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-04T12:36:37.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:36:37.696 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2024-03-04T12:36:37.806 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-04T12:36:37.807 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-04T12:36:37.807 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-04T12:36:37.823 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1356 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/c4a102f7a095da0fde90b9779e9b871022f81313d36f134bf0674245bc178f3a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YP4LWGZ5AZBUR2IOBHZUOGYRS7:/var/lib/containers/storage/overlay/l/JKXYVLCI3O7G6NWX2TTZNTJBTR,upperdir=/var/lib/containers/storage/overlay/c4a102f7a095da0fde90b9779e9b871022f81313d36f134bf0674245bc178f3a/diff,workdir=/var/lib/containers/storage/overlay/c4a102f7a095da0fde90b9779e9b871022f81313d36f134bf0674245bc178f3a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/ad1d20e4fc7ef75db5a0c328b8375a6f0f6dc9d5c7e0262cb4f57f833af588a4/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/YP4LWGZ5AZBUR2IOBHZUOGYRS7:/var/lib/containers/storage/overlay/l/JKXYVLCI3O7G6NWX2TTZNTJBTR,upperdir=/var/lib/containers/storage/overlay/ad1d20e4fc7ef75db5a0c328b8375a6f0f6dc9d5c7e0262cb4f57f833af588a4/diff,workdir=/var/lib/containers/storage/overlay/ad1d20e4fc7ef75db5a0c328b8375a6f0f6dc9d5c7e0262cb4f57f833af588a4/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/f1672df3d5c296519b3fddac45de52ea36246890e702e22bc630df6935ee0053/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YP4LWGZ5AZBUR2IOBHZUOGYRS7:/var/lib/containers/storage/overlay/l/JKXYVLCI3O7G6NWX2TTZNTJBTR,upperdir=/var/lib/containers/storage/overlay/f1672df3d5c296519b3fddac45de52ea36246890e702e22bc630df6935ee0053/diff,workdir=/var/lib/containers/storage/overlay/f1672df3d5c296519b3fddac45de52ea36246890e702e22bc630df6935ee0053/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4823d1f2b2463e8cb8b607d307c56ebb2a70f0049b89af0e71e1c3dc3fe70bcd/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YP4LWGZ5AZBUR2IOBHZUOGYRS7:/var/lib/containers/storage/overlay/l/JKXYVLCI3O7G6NWX2TTZNTJBTR,upperdir=/var/lib/containers/storage/overlay/4823d1f2b2463e8cb8b607d307c56ebb2a70f0049b89af0e71e1c3dc3fe70bcd/diff,workdir=/var/lib/containers/storage/overlay/4823d1f2b2463e8cb8b607d307c56ebb2a70f0049b89af0e71e1c3dc3fe70bcd/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/38f7ea4fd5c8e700ffaf733653e1db7571b477284edb93622a0d6228b3d79d93/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YP4LWGZ5AZBUR2IOBHZUOGYRS7:/var/lib/containers/storage/overlay/l/JKXYVLCI3O7G6NWX2TTZNTJBTR,upperdir=/var/lib/containers/storage/overlay/38f7ea4fd5c8e700ffaf733653e1db7571b477284edb93622a0d6228b3d79d93/diff,workdir=/var/lib/containers/storage/overlay/38f7ea4fd5c8e700ffaf733653e1db7571b477284edb93622a0d6228b3d79d93/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/7fd89f1d2c09f40a834d2f3b2c468f7eb45921aa58710b1c9b95319dfa923d92/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/YP4LWGZ5AZBUR2IOBHZUOGYRS7:/var/lib/containers/storage/overlay/l/JKXYVLCI3O7G6NWX2TTZNTJBTR,upperdir=/var/lib/containers/storage/overlay/7fd89f1d2c09f40a834d2f3b2c468f7eb45921aa58710b1c9b95319dfa923d92/diff,workdir=/var/lib/containers/storage/overlay/7fd89f1d2c09f40a834d2f3b2c468f7eb45921aa58710b1c9b95319dfa923d92/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0d6dd2b1404937a83a7be0fd54514cf4adb139ee463e96e5096ceb0c939e95d2/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/YP4LWGZ5AZBUR2IOBHZUOGYRS7:/var/lib/containers/storage/overlay/l/JKXYVLCI3O7G6NWX2TTZNTJBTR,upperdir=/var/lib/containers/storage/overlay/0d6dd2b1404937a83a7be0fd54514cf4adb139ee463e96e5096ceb0c939e95d2/diff,workdir=/var/lib/containers/storage/overlay/0d6dd2b1404937a83a7be0fd54514cf4adb139ee463e96e5096ceb0c939e95d2/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9fa9c3f9a819e7061d67b66f4db2952ca0881eb29aa054ecfa66d1e2336d9e40/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/YP4LWGZ5AZBUR2IOBHZUOGYRS7:/var/lib/containers/storage/overlay/l/JKXYVLCI3O7G6NWX2TTZNTJBTR,upperdir=/var/lib/containers/storage/overlay/9fa9c3f9a819e7061d67b66f4db2952ca0881eb29aa054ecfa66d1e2336d9e40/diff,workdir=/var/lib/containers/storage/overlay/9fa9c3f9a819e7061d67b66f4db2952ca0881eb29aa054ecfa66d1e2336d9e40/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e3e9b9657de03e125fb9ed16494a2dab386335c220b227376d70c8045371630b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/YP4LWGZ5AZBUR2IOBHZUOGYRS7:/var/lib/containers/storage/overlay/l/JKXYVLCI3O7G6NWX2TTZNTJBTR,upperdir=/var/lib/containers/storage/overlay/e3e9b9657de03e125fb9ed16494a2dab386335c220b227376d70c8045371630b/diff,workdir=/var/lib/containers/storage/overlay/e3e9b9657de03e125fb9ed16494a2dab386335c220b227376d70c8045371630b/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4a5bee2b063dd88be60df02286a771ad4ed9d56f3f562171b39abfdd882ffdb8/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/YP4LWGZ5AZBUR2IOBHZUOGYRS7:/var/lib/containers/storage/overlay/l/JKXYVLCI3O7G6NWX2TTZNTJBTR,upperdir=/var/lib/containers/storage/overlay/4a5bee2b063dd88be60df02286a771ad4ed9d56f3f562171b39abfdd882ffdb8/diff,workdir=/var/lib/containers/storage/overlay/4a5bee2b063dd88be60df02286a771ad4ed9d56f3f562171b39abfdd882ffdb8/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 2024-03-04T12:36:37.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:36:37.823 DEBUG:teuthology.orchestra.run.smithi060:> ip netns list 2024-03-04T12:36:37.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-04T12:36:37.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo ip link delete ceph-brx 2024-03-04T12:36:37.948 INFO:teuthology.orchestra.run.smithi060.stderr:Cannot find device "ceph-brx" 2024-03-04T12:36:37.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:36:37.950 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-04T12:36:37.962 INFO:tasks.cephadm:Teardown begin 2024-03-04T12:36:37.962 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-04T12:36:37.962 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:36:38.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.091+0000 7f4fb9aea640 1 -- 172.21.15.60:0/1559602745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb4105500 msgr2=0x7f4fb410d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.091+0000 7f4fb9aea640 1 --2- 172.21.15.60:0/1559602745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb4105500 0x7f4fb410d0d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009ca0 tx=0x7f4fac02c0f0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.091+0000 7f4fb9aea640 1 -- 172.21.15.60:0/1559602745 shutdown_connections 2024-03-04T12:36:38.097 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.091+0000 7f4fb9aea640 1 --2- 172.21.15.60:0/1559602745 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb4105500 0x7f4fb410d0d0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.091+0000 7f4fb9aea640 1 --2- 172.21.15.60:0/1559602745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb4100de0 0x7f4fb4104e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.091+0000 7f4fb9aea640 1 --2- 172.21.15.60:0/1559602745 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb4100430 0x7f4fb4100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.091+0000 7f4fb9aea640 1 -- 172.21.15.60:0/1559602745 >> 172.21.15.60:0/1559602745 conn(0x7f4fb40fbcb0 msgr2=0x7f4fb40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:38.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.091+0000 7f4fb9aea640 1 -- 172.21.15.60:0/1559602745 shutdown_connections 2024-03-04T12:36:38.098 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.091+0000 7f4fb9aea640 1 -- 172.21.15.60:0/1559602745 wait complete. 2024-03-04T12:36:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.092+0000 7f4fb9aea640 1 Processor -- start 2024-03-04T12:36:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.092+0000 7f4fb9aea640 1 -- start start 2024-03-04T12:36:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.092+0000 7f4fb9aea640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb4100430 0x7f4fb419d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.092+0000 7f4fb9aea640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb4105500 0x7f4fb419dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.092+0000 7f4fb9aea640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb41979d0 0x7f4fb4197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:38.100 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.092+0000 7f4fb9aea640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb4110590 con 0x7f4fb4100430 2024-03-04T12:36:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.092+0000 7f4fb9aea640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb4110410 con 0x7f4fb41979d0 2024-03-04T12:36:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.092+0000 7f4fb9aea640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb4110710 con 0x7f4fb4105500 2024-03-04T12:36:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.092+0000 7f4fb37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb4100430 0x7f4fb419d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.093+0000 7f4fb37fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb4100430 0x7f4fb419d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59006/0 (socket says 172.21.15.60:59006) 2024-03-04T12:36:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.093+0000 7f4fb37fe640 1 -- 172.21.15.60:0/2262392797 learned_addr learned my addr 172.21.15.60:0/2262392797 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:38.101 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.093+0000 7f4fb2ffd640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb4105500 0x7f4fb419dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:38.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.093+0000 7f4fb3fff640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb41979d0 0x7f4fb4197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:38.102 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.093+0000 7f4fb37fe640 1 -- 172.21.15.60:0/2262392797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb4105500 msgr2=0x7f4fb419dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.093+0000 7f4fb37fe640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb4105500 0x7f4fb419dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.093+0000 7f4fb37fe640 1 -- 172.21.15.60:0/2262392797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb41979d0 msgr2=0x7f4fb4197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.093+0000 7f4fb37fe640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb41979d0 0x7f4fb4197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.093+0000 7f4fb37fe640 1 -- 172.21.15.60:0/2262392797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb41984f0 con 0x7f4fb4100430 2024-03-04T12:36:38.103 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.093+0000 7f4fb37fe640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb4100430 0x7f4fb419d820 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400e9f0 tx=0x7f4fa400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:38.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.095+0000 7f4fb0ff9640 1 -- 172.21.15.60:0/2262392797 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa400cd80 con 0x7f4fb4100430 2024-03-04T12:36:38.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.095+0000 7f4fb0ff9640 1 -- 172.21.15.60:0/2262392797 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4fa4010910 con 0x7f4fb4100430 2024-03-04T12:36:38.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.095+0000 7f4fb9aea640 1 -- 172.21.15.60:0/2262392797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb41987e0 con 0x7f4fb4100430 2024-03-04T12:36:38.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.095+0000 7f4fb0ff9640 1 -- 172.21.15.60:0/2262392797 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa4018990 con 0x7f4fb4100430 2024-03-04T12:36:38.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.095+0000 7f4fb9aea640 1 -- 172.21.15.60:0/2262392797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb41a4a80 con 0x7f4fb4100430 2024-03-04T12:36:38.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.096+0000 7f4fb9aea640 1 -- 172.21.15.60:0/2262392797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb4101900 con 0x7f4fb4100430 2024-03-04T12:36:38.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.097+0000 7f4fb0ff9640 1 -- 172.21.15.60:0/2262392797 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f4fa4010ac0 con 0x7f4fb4100430 2024-03-04T12:36:38.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.097+0000 7f4fb0ff9640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4f9c0a1f70 0x7f4f9c0a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:38.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.098+0000 7f4fb0ff9640 1 -- 172.21.15.60:0/2262392797 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f4fa40ce8c0 con 0x7f4fb4100430 2024-03-04T12:36:38.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.098+0000 7f4fb2ffd640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4f9c0a1f70 0x7f4f9c0a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:38.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.098+0000 7f4fb2ffd640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4f9c0a1f70 0x7f4f9c0a4430 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40ff540 tx=0x7f4fa8005f90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:38.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.102+0000 7f4fb0ff9640 1 -- 172.21.15.60:0/2262392797 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4fa409af00 con 0x7f4fb4100430 2024-03-04T12:36:38.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.323+0000 7f4fb9aea640 1 -- 172.21.15.60:0/2262392797 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fb4068600 con 0x7f4fb4100430 2024-03-04T12:36:38.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.324+0000 7f4fb0ff9640 1 -- 172.21.15.60:0/2262392797 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+11376 (secure 0 0 0) 0x7f4fa409efa0 con 0x7f4fb4100430 2024-03-04T12:36:38.330 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:36:38.331 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:30:40.078275+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":69,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_25741":{"gid":25741,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/2924944116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2924944116},{"type":"v1","addr":"172.21.15.86:6837","nonce":2924944116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:36:38.333 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 40 2024-03-04T12:36:38.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.326+0000 7f4fb9aea640 1 -- 172.21.15.60:0/2262392797 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4f9c0a1f70 msgr2=0x7f4f9c0a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.326+0000 7f4fb9aea640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4f9c0a1f70 0x7f4f9c0a4430 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4fb40ff540 tx=0x7f4fa8005f90 comp rx=0 tx=0).stop 2024-03-04T12:36:38.333 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.326+0000 7f4fb9aea640 1 -- 172.21.15.60:0/2262392797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb4100430 msgr2=0x7f4fb419d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.326+0000 7f4fb9aea640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb4100430 0x7f4fb419d820 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4fa400e9f0 tx=0x7f4fa400eeb0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.326+0000 7f4fb9aea640 1 -- 172.21.15.60:0/2262392797 shutdown_connections 2024-03-04T12:36:38.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.326+0000 7f4fb9aea640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb41979d0 0x7f4fb4197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.334 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.326+0000 7f4fb9aea640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb4105500 0x7f4fb419dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.326+0000 7f4fb9aea640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f4f9c0a1f70 0x7f4f9c0a4430 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.326+0000 7f4fb9aea640 1 --2- 172.21.15.60:0/2262392797 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f4fb4100430 0x7f4fb419d820 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.326+0000 7f4fb9aea640 1 -- 172.21.15.60:0/2262392797 >> 172.21.15.60:0/2262392797 conn(0x7f4fb40fbcb0 msgr2=0x7f4fb4069d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.326+0000 7f4fb9aea640 1 -- 172.21.15.60:0/2262392797 shutdown_connections 2024-03-04T12:36:38.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.326+0000 7f4fb9aea640 1 -- 172.21.15.60:0/2262392797 wait complete. 2024-03-04T12:36:38.341 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:38 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2319603751' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:36:38.342 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:38 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3303080264' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T12:36:38.344 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-04T12:36:38.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:38 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2319603751' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:36:38.348 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:38 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3303080264' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T12:36:38.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:38 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2319603751' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-04T12:36:38.351 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:38 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3303080264' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T12:36:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.441+0000 7f4143174640 1 -- 172.21.15.60:0/3874843174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f413c100450 msgr2=0x7f413c100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.441+0000 7f4143174640 1 --2- 172.21.15.60:0/3874843174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f413c100450 0x7f413c100830 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f4130009900 tx=0x7f413002bc30 comp rx=0 tx=0).stop 2024-03-04T12:36:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.441+0000 7f4143174640 1 -- 172.21.15.60:0/3874843174 shutdown_connections 2024-03-04T12:36:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.441+0000 7f4143174640 1 --2- 172.21.15.60:0/3874843174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f413c105520 0x7f413c10d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.441+0000 7f4143174640 1 --2- 172.21.15.60:0/3874843174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f413c100e00 0x7f413c104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.449 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.441+0000 7f4143174640 1 --2- 172.21.15.60:0/3874843174 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f413c100450 0x7f413c100830 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.441+0000 7f4143174640 1 -- 172.21.15.60:0/3874843174 >> 172.21.15.60:0/3874843174 conn(0x7f413c0773b0 msgr2=0x7f413c0777c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.441+0000 7f4143174640 1 -- 172.21.15.60:0/3874843174 shutdown_connections 2024-03-04T12:36:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.441+0000 7f4143174640 1 -- 172.21.15.60:0/3874843174 wait complete. 2024-03-04T12:36:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.441+0000 7f4143174640 1 Processor -- start 2024-03-04T12:36:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.441+0000 7f4143174640 1 -- start start 2024-03-04T12:36:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4143174640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f413c100450 0x7f413c198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4143174640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f413c105520 0x7f413c198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4143174640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f413c199520 0x7f413c1a3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:38.450 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4143174640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c1105d0 con 0x7f413c100450 2024-03-04T12:36:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4143174640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c110450 con 0x7f413c199520 2024-03-04T12:36:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4143174640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c110750 con 0x7f413c105520 2024-03-04T12:36:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4140f10640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f413c100450 0x7f413c198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4140f10640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f413c100450 0x7f413c198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59028/0 (socket says 172.21.15.60:59028) 2024-03-04T12:36:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4140f10640 1 -- 172.21.15.60:0/3985321725 learned_addr learned my addr 172.21.15.60:0/3985321725 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4141711640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f413c199520 0x7f413c1a3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:38.451 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f413bfff640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f413c105520 0x7f413c198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4140f10640 1 -- 172.21.15.60:0/3985321725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f413c105520 msgr2=0x7f413c198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4140f10640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f413c105520 0x7f413c198f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4140f10640 1 -- 172.21.15.60:0/3985321725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f413c199520 msgr2=0x7f413c1a3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4140f10640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f413c199520 0x7f413c1a3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4140f10640 1 -- 172.21.15.60:0/3985321725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413c1a42d0 con 0x7f413c100450 2024-03-04T12:36:38.452 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f413bfff640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f413c105520 0x7f413c198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:36:38.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4140f10640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f413c100450 0x7f413c198a50 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4130006010 tx=0x7f413002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:38.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4139ffb640 1 -- 172.21.15.60:0/3985321725 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130041070 con 0x7f413c100450 2024-03-04T12:36:38.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4139ffb640 1 -- 172.21.15.60:0/3985321725 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4130004450 con 0x7f413c100450 2024-03-04T12:36:38.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4143174640 1 -- 172.21.15.60:0/3985321725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413c1a4560 con 0x7f413c100450 2024-03-04T12:36:38.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.442+0000 7f4139ffb640 1 -- 172.21.15.60:0/3985321725 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130003e00 con 0x7f413c100450 2024-03-04T12:36:38.453 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.443+0000 7f4143174640 1 -- 172.21.15.60:0/3985321725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413c1a4a50 con 0x7f413c100450 2024-03-04T12:36:38.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.444+0000 7f4143174640 1 -- 172.21.15.60:0/3985321725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413c101920 con 0x7f413c100450 2024-03-04T12:36:38.454 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.445+0000 7f4139ffb640 1 -- 172.21.15.60:0/3985321725 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f4130044070 con 0x7f413c100450 2024-03-04T12:36:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.445+0000 7f4139ffb640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f412c0a1ef0 0x7f412c0a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.445+0000 7f4139ffb640 1 -- 172.21.15.60:0/3985321725 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f41300f0050 con 0x7f413c100450 2024-03-04T12:36:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.445+0000 7f413bfff640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f412c0a1ef0 0x7f412c0a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.446+0000 7f413bfff640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f412c0a1ef0 0x7f412c0a43b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f413c06a580 tx=0x7f4128005f90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:38.455 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.450+0000 7f4139ffb640 1 -- 172.21.15.60:0/3985321725 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f41300bce50 con 0x7f413c100450 2024-03-04T12:36:38.672 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.667+0000 7f4143174640 1 -- 172.21.15.60:0/3985321725 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f413c108750 con 0x7f413c100450 2024-03-04T12:36:38.673 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.668+0000 7f4139ffb640 1 -- 172.21.15.60:0/3985321725 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+11376 (secure 0 0 0) 0x7f41300bc750 con 0x7f413c100450 2024-03-04T12:36:38.674 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:36:38.674 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6835/49412012","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6834","nonce":49412012},{"type":"v1","addr":"172.21.15.60:6835","nonce":49412012}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.60:6837/3896783798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6836","nonce":3896783798},{"type":"v1","addr":"172.21.15.60:6837","nonce":3896783798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-04T12:23:12.241967+0000","modified":"2024-03-04T12:30:40.078275+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24524,"mds_1":24487,"mds_2":24479,"mds_3":24407,"mds_4":24394},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6839/2180757903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6838","nonce":2180757903},{"type":"v1","addr":"172.21.15.60:6839","nonce":2180757903}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14643":{"gid":14643,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.60:6841/4175833227","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6840","nonce":4175833227},{"type":"v1","addr":"172.21.15.60:6841","nonce":4175833227}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24394":{"gid":24394,"name":"l","rank":4,"incarnation":26,"state":"up:active","state_seq":18,"addr":"172.21.15.86:6833/777854669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":777854669},{"type":"v1","addr":"172.21.15.86:6833","nonce":777854669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24407":{"gid":24407,"name":"k","rank":3,"incarnation":24,"state":"up:active","state_seq":17,"addr":"172.21.15.63:6833/2270964219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":2270964219},{"type":"v1","addr":"172.21.15.63:6833","nonce":2270964219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/2453356505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":2453356505},{"type":"v1","addr":"172.21.15.63:6835","nonce":2453356505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24451":{"gid":24451,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6835/2934725061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2934725061},{"type":"v1","addr":"172.21.15.86:6835","nonce":2934725061}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":69,"addr":"172.21.15.63:6837/3974574804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":3974574804},{"type":"v1","addr":"172.21.15.63:6837","nonce":3974574804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6839/3347800322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6838","nonce":3347800322},{"type":"v1","addr":"172.21.15.86:6839","nonce":3347800322}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/2728229139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":2728229139},{"type":"v1","addr":"172.21.15.63:6839","nonce":2728229139}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_25741":{"gid":25741,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.86:6837/2924944116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":2924944116},{"type":"v1","addr":"172.21.15.86:6837","nonce":2924944116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-04T12:36:38.674 INFO:teuthology.orchestra.run.smithi060.stderr:dumped fsmap epoch 40 2024-03-04T12:36:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.670+0000 7f4143174640 1 -- 172.21.15.60:0/3985321725 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f412c0a1ef0 msgr2=0x7f412c0a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.670+0000 7f4143174640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f412c0a1ef0 0x7f412c0a43b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f413c06a580 tx=0x7f4128005f90 comp rx=0 tx=0).stop 2024-03-04T12:36:38.676 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.670+0000 7f4143174640 1 -- 172.21.15.60:0/3985321725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f413c100450 msgr2=0x7f413c198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.670+0000 7f4143174640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f413c100450 0x7f413c198a50 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4130006010 tx=0x7f413002cb30 comp rx=0 tx=0).stop 2024-03-04T12:36:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.670+0000 7f4143174640 1 -- 172.21.15.60:0/3985321725 shutdown_connections 2024-03-04T12:36:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.670+0000 7f4143174640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f413c199520 0x7f413c1a3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.670+0000 7f4143174640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f413c105520 0x7f413c198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.670+0000 7f4143174640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f412c0a1ef0 0x7f412c0a43b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.670+0000 7f4143174640 1 --2- 172.21.15.60:0/3985321725 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f413c100450 0x7f413c198a50 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.677 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.670+0000 7f4143174640 1 -- 172.21.15.60:0/3985321725 >> 172.21.15.60:0/3985321725 conn(0x7f413c0773b0 msgr2=0x7f413c0fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:38.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.670+0000 7f4143174640 1 -- 172.21.15.60:0/3985321725 shutdown_connections 2024-03-04T12:36:38.678 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.670+0000 7f4143174640 1 -- 172.21.15.60:0/3985321725 wait complete. 2024-03-04T12:36:38.687 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-04T12:36:38.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.785+0000 7f2dd1be6640 1 -- 172.21.15.60:0/468757350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dcc10b920 msgr2=0x7f2dcc10dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.785+0000 7f2dd1be6640 1 --2- 172.21.15.60:0/468757350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dcc10b920 0x7f2dcc10dd70 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009ca0 tx=0x7f2dc402c180 comp rx=0 tx=0).stop 2024-03-04T12:36:38.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.785+0000 7f2dd1be6640 1 -- 172.21.15.60:0/468757350 shutdown_connections 2024-03-04T12:36:38.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.785+0000 7f2dd1be6640 1 --2- 172.21.15.60:0/468757350 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dcc10b920 0x7f2dcc10dd70 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.793 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.785+0000 7f2dd1be6640 1 --2- 172.21.15.60:0/468757350 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2dcc102e70 0x7f2dcc10b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.785+0000 7f2dd1be6640 1 --2- 172.21.15.60:0/468757350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dcc102550 0x7f2dcc102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.785+0000 7f2dd1be6640 1 -- 172.21.15.60:0/468757350 >> 172.21.15.60:0/468757350 conn(0x7f2dcc0773b0 msgr2=0x7f2dcc0777c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:38.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.785+0000 7f2dd1be6640 1 -- 172.21.15.60:0/468757350 shutdown_connections 2024-03-04T12:36:38.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.785+0000 7f2dd1be6640 1 -- 172.21.15.60:0/468757350 wait complete. 2024-03-04T12:36:38.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dd1be6640 1 Processor -- start 2024-03-04T12:36:38.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dd1be6640 1 -- start start 2024-03-04T12:36:38.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dd1be6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2dcc102550 0x7f2dcc19d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:38.794 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dd1be6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dcc102e70 0x7f2dcc19dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:38.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dd1be6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dcc10b920 0x7f2dcc197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:38.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dd1be6640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc111680 con 0x7f2dcc102e70 2024-03-04T12:36:38.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dd1be6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc111500 con 0x7f2dcc102550 2024-03-04T12:36:38.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dd1be6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc111800 con 0x7f2dcc10b920 2024-03-04T12:36:38.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dcaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dcc102e70 0x7f2dcc19dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:38.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dcaffd640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dcc102e70 0x7f2dcc19dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59044/0 (socket says 172.21.15.60:59044) 2024-03-04T12:36:38.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dcaffd640 1 -- 172.21.15.60:0/2076250156 learned_addr learned my addr 172.21.15.60:0/2076250156 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:38.795 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dcbfff640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dcc10b920 0x7f2dcc197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:38.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dcb7fe640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2dcc102550 0x7f2dcc19d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:38.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dcaffd640 1 -- 172.21.15.60:0/2076250156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dcc10b920 msgr2=0x7f2dcc197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dcaffd640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dcc10b920 0x7f2dcc197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dcaffd640 1 -- 172.21.15.60:0/2076250156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2dcc102550 msgr2=0x7f2dcc19d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dcaffd640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2dcc102550 0x7f2dcc19d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.786+0000 7f2dcaffd640 1 -- 172.21.15.60:0/2076250156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc1980e0 con 0x7f2dcc102e70 2024-03-04T12:36:38.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.787+0000 7f2dcaffd640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dcc102e70 0x7f2dcc19dcf0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000d950 tx=0x7f2dc000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:38.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.787+0000 7f2dc8ff9640 1 -- 172.21.15.60:0/2076250156 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0017070 con 0x7f2dcc102e70 2024-03-04T12:36:38.796 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.787+0000 7f2dc8ff9640 1 -- 172.21.15.60:0/2076250156 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dc0009c30 con 0x7f2dcc102e70 2024-03-04T12:36:38.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.787+0000 7f2dc8ff9640 1 -- 172.21.15.60:0/2076250156 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc000f6b0 con 0x7f2dcc102e70 2024-03-04T12:36:38.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.787+0000 7f2dd1be6640 1 -- 172.21.15.60:0/2076250156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dcc1983d0 con 0x7f2dcc102e70 2024-03-04T12:36:38.797 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.787+0000 7f2dd1be6640 1 -- 172.21.15.60:0/2076250156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dcc1a4670 con 0x7f2dcc102e70 2024-03-04T12:36:38.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.788+0000 7f2dd1be6640 1 -- 172.21.15.60:0/2076250156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dcc0681b0 con 0x7f2dcc102e70 2024-03-04T12:36:38.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.789+0000 7f2dc8ff9640 1 -- 172.21.15.60:0/2076250156 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f2dc00109d0 con 0x7f2dcc102e70 2024-03-04T12:36:38.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.789+0000 7f2dc8ff9640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2db40a1ef0 0x7f2db40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:38.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.789+0000 7f2dcb7fe640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2db40a1ef0 0x7f2db40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:38.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.790+0000 7f2dc8ff9640 1 -- 172.21.15.60:0/2076250156 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9539+0+0 (secure 0 0 0) 0x7f2dc00ce120 con 0x7f2dcc102e70 2024-03-04T12:36:38.798 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.790+0000 7f2dcb7fe640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2db40a1ef0 0x7f2db40a43b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc005fd0 tx=0x7f2dbc005850 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:38.799 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.793+0000 7f2dc8ff9640 1 -- 172.21.15.60:0/2076250156 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2dc0009de0 con 0x7f2dcc102e70 2024-03-04T12:36:38.980 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.975+0000 7f2dd1be6640 1 -- 172.21.15.60:0/2076250156 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dcc1a4970 con 0x7f2dcc102e70 2024-03-04T12:36:38.982 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.976+0000 7f2dc8ff9640 1 -- 172.21.15.60:0/2076250156 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+25187 (secure 0 0 0) 0x7f2dc009ae10 con 0x7f2dcc102e70 2024-03-04T12:36:38.983 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-04T12:36:38.983 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":105,"fsid":"a1c49686-da20-11ee-95c6-87774f69a715","created":"2024-03-04T12:15:08.418686+0000","modified":"2024-03-04T12:36:15.046092+0000","last_up_change":"2024-03-04T12:22:18.769057+0000","last_in_change":"2024-03-04T12:21:57.729731+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T12:18:28.414510+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-04T12:23:10.438928+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-04T12:23:11.472998+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-04T12:23:15.162500+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"c1ecbdba-fb7e-4e00-ad43-0ad0dfe1e14f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6803","nonce":4258653647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6805","nonce":4258653647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6809","nonce":4258653647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":4258653647},{"type":"v1","addr":"172.21.15.60:6807","nonce":4258653647}]},"public_addr":"172.21.15.60:6803/4258653647","cluster_addr":"172.21.15.60:6805/4258653647","heartbeat_back_addr":"172.21.15.60:6809/4258653647","heartbeat_front_addr":"172.21.15.60:6807/4258653647","state":["exists","up"]},{"osd":1,"uuid":"59fe4466-e274-4543-a2a4-f95588613bf4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6801","nonce":1936533350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6803","nonce":1936533350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6807","nonce":1936533350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1936533350},{"type":"v1","addr":"172.21.15.63:6805","nonce":1936533350}]},"public_addr":"172.21.15.63:6801/1936533350","cluster_addr":"172.21.15.63:6803/1936533350","heartbeat_back_addr":"172.21.15.63:6807/1936533350","heartbeat_front_addr":"172.21.15.63:6805/1936533350","state":["exists","up"]},{"osd":2,"uuid":"d05a35c6-2586-47fa-8172-6597e7925861","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6801","nonce":3773043222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6803","nonce":3773043222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6807","nonce":3773043222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3773043222},{"type":"v1","addr":"172.21.15.86:6805","nonce":3773043222}]},"public_addr":"172.21.15.86:6801/3773043222","cluster_addr":"172.21.15.86:6803/3773043222","heartbeat_back_addr":"172.21.15.86:6807/3773043222","heartbeat_front_addr":"172.21.15.86:6805/3773043222","state":["exists","up"]},{"osd":3,"uuid":"a823b593-a7b5-45ff-8765-cac26276dedd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6811","nonce":1173200157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6813","nonce":1173200157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6817","nonce":1173200157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":1173200157},{"type":"v1","addr":"172.21.15.60:6815","nonce":1173200157}]},"public_addr":"172.21.15.60:6811/1173200157","cluster_addr":"172.21.15.60:6813/1173200157","heartbeat_back_addr":"172.21.15.60:6817/1173200157","heartbeat_front_addr":"172.21.15.60:6815/1173200157","state":["exists","up"]},{"osd":4,"uuid":"75da5356-1007-4b12-89dc-5e91ec454df8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":793955},{"type":"v1","addr":"172.21.15.63:6809","nonce":793955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":793955},{"type":"v1","addr":"172.21.15.63:6811","nonce":793955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":793955},{"type":"v1","addr":"172.21.15.63:6815","nonce":793955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":793955},{"type":"v1","addr":"172.21.15.63:6813","nonce":793955}]},"public_addr":"172.21.15.63:6809/793955","cluster_addr":"172.21.15.63:6811/793955","heartbeat_back_addr":"172.21.15.63:6815/793955","heartbeat_front_addr":"172.21.15.63:6813/793955","state":["exists","up"]},{"osd":5,"uuid":"6d29eae4-5669-4e1e-a94f-ddb37bcc1a72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6809","nonce":1744734157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6811","nonce":1744734157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6815","nonce":1744734157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1744734157},{"type":"v1","addr":"172.21.15.86:6813","nonce":1744734157}]},"public_addr":"172.21.15.86:6809/1744734157","cluster_addr":"172.21.15.86:6811/1744734157","heartbeat_back_addr":"172.21.15.86:6815/1744734157","heartbeat_front_addr":"172.21.15.86:6813/1744734157","state":["exists","up"]},{"osd":6,"uuid":"4794189a-f45f-4859-99de-d7578ff20141","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6819","nonce":285200459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6820","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6821","nonce":285200459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6825","nonce":285200459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":285200459},{"type":"v1","addr":"172.21.15.60:6823","nonce":285200459}]},"public_addr":"172.21.15.60:6819/285200459","cluster_addr":"172.21.15.60:6821/285200459","heartbeat_back_addr":"172.21.15.60:6825/285200459","heartbeat_front_addr":"172.21.15.60:6823/285200459","state":["exists","up"]},{"osd":7,"uuid":"86f9c999-69b0-4c33-b39c-5c307ec78e58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6817","nonce":942963290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6819","nonce":942963290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6823","nonce":942963290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":942963290},{"type":"v1","addr":"172.21.15.63:6821","nonce":942963290}]},"public_addr":"172.21.15.63:6817/942963290","cluster_addr":"172.21.15.63:6819/942963290","heartbeat_back_addr":"172.21.15.63:6823/942963290","heartbeat_front_addr":"172.21.15.63:6821/942963290","state":["exists","up"]},{"osd":8,"uuid":"5848de7e-e01b-4ae8-a060-ead59bf03d90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6817","nonce":1013701944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6819","nonce":1013701944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6823","nonce":1013701944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1013701944},{"type":"v1","addr":"172.21.15.86:6821","nonce":1013701944}]},"public_addr":"172.21.15.86:6817/1013701944","cluster_addr":"172.21.15.86:6819/1013701944","heartbeat_back_addr":"172.21.15.86:6823/1013701944","heartbeat_front_addr":"172.21.15.86:6821/1013701944","state":["exists","up"]},{"osd":9,"uuid":"92ab3932-3338-42a8-ae91-6d348af0c507","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6826","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6827","nonce":1552052384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6828","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6829","nonce":1552052384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6832","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6833","nonce":1552052384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6830","nonce":1552052384},{"type":"v1","addr":"172.21.15.60:6831","nonce":1552052384}]},"public_addr":"172.21.15.60:6827/1552052384","cluster_addr":"172.21.15.60:6829/1552052384","heartbeat_back_addr":"172.21.15.60:6833/1552052384","heartbeat_front_addr":"172.21.15.60:6831/1552052384","state":["exists","up"]},{"osd":10,"uuid":"9fa85ae1-3566-4486-8586-05edcecc7848","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6825","nonce":4239140227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6827","nonce":4239140227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6831","nonce":4239140227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":4239140227},{"type":"v1","addr":"172.21.15.63:6829","nonce":4239140227}]},"public_addr":"172.21.15.63:6825/4239140227","cluster_addr":"172.21.15.63:6827/4239140227","heartbeat_back_addr":"172.21.15.63:6831/4239140227","heartbeat_front_addr":"172.21.15.63:6829/4239140227","state":["exists","up"]},{"osd":11,"uuid":"344b7daf-897d-4217-8cba-192c507c09e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6825","nonce":2397477201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6827","nonce":2397477201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6831","nonce":2397477201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":2397477201},{"type":"v1","addr":"172.21.15.86:6829","nonce":2397477201}]},"public_addr":"172.21.15.86:6825/2397477201","cluster_addr":"172.21.15.86:6827/2397477201","heartbeat_back_addr":"172.21.15.86:6831/2397477201","heartbeat_front_addr":"172.21.15.86:6829/2397477201","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:17:36.863263+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:01.220340+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:24.695503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:18:49.948228+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:15.003849+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:19:40.002729+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:05.414813+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:31.827234+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:20:57.068847+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:23.504644+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:21:50.162686+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T12:22:16.323017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.27","mappings":[{"from":4,"to":7}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.60:0/2167195614":"2024-03-05T12:25:53.138536+0000","172.21.15.60:0/2028967685":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3401854556":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6801/1914863179":"2024-03-05T12:25:53.138536+0000","172.21.15.60:0/758894979":"2024-03-05T12:15:37.282929+0000","172.21.15.60:6800/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/3539227088":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3302898178":"2024-03-05T12:25:53.138536+0000","172.21.15.60:6801/2246601804":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/2604977412":"2024-03-05T12:25:53.138536+0000","172.21.15.60:6801/2061680026":"2024-03-05T12:15:37.282929+0000","172.21.15.60:0/3868523955":"2024-03-05T12:16:04.073665+0000","172.21.15.60:0/2852203003":"2024-03-05T12:16:04.073665+0000","172.21.15.60:6800/1914863179":"2024-03-05T12:25:53.138536+0000","172.21.15.60:0/366921369":"2024-03-05T12:25:53.138536+0000","172.21.15.60:0/1342937290":"2024-03-05T12:25:53.138536+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T12:36:38.985 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.978+0000 7f2dd1be6640 1 -- 172.21.15.60:0/2076250156 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2db40a1ef0 msgr2=0x7f2db40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.978+0000 7f2dd1be6640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2db40a1ef0 0x7f2db40a43b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc005fd0 tx=0x7f2dbc005850 comp rx=0 tx=0).stop 2024-03-04T12:36:38.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.978+0000 7f2dd1be6640 1 -- 172.21.15.60:0/2076250156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dcc102e70 msgr2=0x7f2dcc19dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:38.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.978+0000 7f2dd1be6640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dcc102e70 0x7f2dcc19dcf0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000d950 tx=0x7f2dc000de10 comp rx=0 tx=0).stop 2024-03-04T12:36:38.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.979+0000 7f2dd1be6640 1 -- 172.21.15.60:0/2076250156 shutdown_connections 2024-03-04T12:36:38.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.979+0000 7f2dd1be6640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dcc10b920 0x7f2dcc197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.979+0000 7f2dd1be6640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f2db40a1ef0 0x7f2db40a43b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.979+0000 7f2dd1be6640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f2dcc102e70 0x7f2dcc19dcf0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.986 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.979+0000 7f2dd1be6640 1 --2- 172.21.15.60:0/2076250156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2dcc102550 0x7f2dcc19d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:38.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.979+0000 7f2dd1be6640 1 -- 172.21.15.60:0/2076250156 >> 172.21.15.60:0/2076250156 conn(0x7f2dcc0773b0 msgr2=0x7f2dcc0fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:38.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.979+0000 7f2dd1be6640 1 -- 172.21.15.60:0/2076250156 shutdown_connections 2024-03-04T12:36:38.987 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:38.979+0000 7f2dd1be6640 1 -- 172.21.15.60:0/2076250156 wait complete. 2024-03-04T12:36:38.996 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-04T12:36:39.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.097+0000 7fdd2c212640 1 -- 172.21.15.60:0/1735266042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd24101e20 msgr2=0x7fdd24102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:39.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.097+0000 7fdd2c212640 1 --2- 172.21.15.60:0/1735266042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd24101e20 0x7fdd24102280 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009930 tx=0x7fdd1402bc30 comp rx=0 tx=0).stop 2024-03-04T12:36:39.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.097+0000 7fdd2c212640 1 -- 172.21.15.60:0/1735266042 shutdown_connections 2024-03-04T12:36:39.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.097+0000 7fdd2c212640 1 --2- 172.21.15.60:0/1735266042 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd241027c0 0x7fdd2410d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.097+0000 7fdd2c212640 1 --2- 172.21.15.60:0/1735266042 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd24101e20 0x7fdd24102280 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.105 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.097+0000 7fdd2c212640 1 --2- 172.21.15.60:0/1735266042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd24100540 0x7fdd24100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.097+0000 7fdd2c212640 1 -- 172.21.15.60:0/1735266042 >> 172.21.15.60:0/1735266042 conn(0x7fdd24077410 msgr2=0x7fdd24077820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.097+0000 7fdd2c212640 1 -- 172.21.15.60:0/1735266042 shutdown_connections 2024-03-04T12:36:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.097+0000 7fdd2c212640 1 -- 172.21.15.60:0/1735266042 wait complete. 2024-03-04T12:36:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.097+0000 7fdd2c212640 1 Processor -- start 2024-03-04T12:36:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.097+0000 7fdd2c212640 1 -- start start 2024-03-04T12:36:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2c212640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd24100540 0x7fdd2419d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2c212640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd24101e20 0x7fdd2419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2c212640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd241027c0 0x7fdd24197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:39.106 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2c212640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd24110700 con 0x7fdd241027c0 2024-03-04T12:36:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2c212640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd24110580 con 0x7fdd24101e20 2024-03-04T12:36:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2c212640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd24110880 con 0x7fdd24100540 2024-03-04T12:36:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2a7af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd241027c0 0x7fdd24197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2a7af640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd241027c0 0x7fdd24197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59060/0 (socket says 172.21.15.60:59060) 2024-03-04T12:36:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2a7af640 1 -- 172.21.15.60:0/3109237465 learned_addr learned my addr 172.21.15.60:0/3109237465 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd29fae640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd24100540 0x7fdd2419d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd297ad640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd24101e20 0x7fdd2419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:39.107 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2a7af640 1 -- 172.21.15.60:0/3109237465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd24100540 msgr2=0x7fdd2419d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2a7af640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd24100540 0x7fdd2419d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2a7af640 1 -- 172.21.15.60:0/3109237465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd24101e20 msgr2=0x7fdd2419de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2a7af640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd24101e20 0x7fdd2419de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2a7af640 1 -- 172.21.15.60:0/3109237465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd24198200 con 0x7fdd241027c0 2024-03-04T12:36:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2a7af640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd241027c0 0x7fdd24197a80 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00e960 tx=0x7fdd1c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd1affd640 1 -- 172.21.15.60:0/3109237465 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c004d20 con 0x7fdd241027c0 2024-03-04T12:36:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd1affd640 1 -- 172.21.15.60:0/3109237465 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd1c005020 con 0x7fdd241027c0 2024-03-04T12:36:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2c212640 1 -- 172.21.15.60:0/3109237465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd241984f0 con 0x7fdd241027c0 2024-03-04T12:36:39.108 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd1affd640 1 -- 172.21.15.60:0/3109237465 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c0116d0 con 0x7fdd241027c0 2024-03-04T12:36:39.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.098+0000 7fdd2c212640 1 -- 172.21.15.60:0/3109237465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd241a4790 con 0x7fdd241027c0 2024-03-04T12:36:39.109 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.099+0000 7fdd2c212640 1 -- 172.21.15.60:0/3109237465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd24068600 con 0x7fdd241027c0 2024-03-04T12:36:39.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.100+0000 7fdd1affd640 1 -- 172.21.15.60:0/3109237465 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7fdd1c005560 con 0x7fdd241027c0 2024-03-04T12:36:39.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.101+0000 7fdd1affd640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fdd100a1f70 0x7fdd100a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:39.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.101+0000 7fdd29fae640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fdd100a1f70 0x7fdd100a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:39.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.101+0000 7fdd1affd640 1 -- 172.21.15.60:0/3109237465 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9539+0+0 (secure 0 0 0) 0x7fdd1c0ce980 con 0x7fdd241027c0 2024-03-04T12:36:39.110 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.102+0000 7fdd29fae640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fdd100a1f70 0x7fdd100a4430 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdd24198bc0 tx=0x7fdd20005f90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:39.111 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.106+0000 7fdd1affd640 1 -- 172.21.15.60:0/3109237465 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdd1c016030 con 0x7fdd241027c0 2024-03-04T12:36:39.332 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.327+0000 7fdd2c212640 1 -- 172.21.15.60:0/3109237465 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdd24109820 con 0x7fdd241027c0 2024-03-04T12:36:39.334 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-mon[32745]: pgmap v336: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:36:39.334 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2262392797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:36:39.334 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3985321725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:36:39.334 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2076250156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:36:39.335 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.329+0000 7fdd1affd640 1 -- 172.21.15.60:0/3109237465 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v41) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdd1c09b020 con 0x7fdd241027c0 2024-03-04T12:36:39.335 INFO:teuthology.orchestra.run.smithi060.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-04T12:36:39.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.332+0000 7fdd2c212640 1 -- 172.21.15.60:0/3109237465 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fdd100a1f70 msgr2=0x7fdd100a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:39.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.332+0000 7fdd2c212640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fdd100a1f70 0x7fdd100a4430 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdd24198bc0 tx=0x7fdd20005f90 comp rx=0 tx=0).stop 2024-03-04T12:36:39.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.332+0000 7fdd2c212640 1 -- 172.21.15.60:0/3109237465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd241027c0 msgr2=0x7fdd24197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:39.338 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.332+0000 7fdd2c212640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd241027c0 0x7fdd24197a80 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00e960 tx=0x7fdd1c00ee20 comp rx=0 tx=0).stop 2024-03-04T12:36:39.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.333+0000 7fdd2c212640 1 -- 172.21.15.60:0/3109237465 shutdown_connections 2024-03-04T12:36:39.339 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.333+0000 7fdd2c212640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7fdd100a1f70 0x7fdd100a4430 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.333+0000 7fdd2c212640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7fdd241027c0 0x7fdd24197a80 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.333+0000 7fdd2c212640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd24101e20 0x7fdd2419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.333+0000 7fdd2c212640 1 --2- 172.21.15.60:0/3109237465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd24100540 0x7fdd2419d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.333+0000 7fdd2c212640 1 -- 172.21.15.60:0/3109237465 >> 172.21.15.60:0/3109237465 conn(0x7fdd24077410 msgr2=0x7fdd24069d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:39.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.333+0000 7fdd2c212640 1 -- 172.21.15.60:0/3109237465 shutdown_connections 2024-03-04T12:36:39.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.333+0000 7fdd2c212640 1 -- 172.21.15.60:0/3109237465 wait complete. 2024-03-04T12:36:39.352 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-04T12:36:39.353 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-mon[36717]: pgmap v336: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:36:39.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2262392797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:36:39.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3985321725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:36:39.354 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2076250156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:36:39.379 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-mon[36851]: pgmap v336: 129 pgs: 129 active+clean; 80 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-04T12:36:39.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2262392797' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:36:39.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3985321725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-04T12:36:39.380 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2076250156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T12:36:39.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.456+0000 7f0a72f16640 1 -- 172.21.15.60:0/2769921294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a6c102540 msgr2=0x7f0a6c102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:39.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.456+0000 7f0a72f16640 1 --2- 172.21.15.60:0/2769921294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a6c102540 0x7f0a6c102920 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009930 tx=0x7f0a6802bc30 comp rx=0 tx=0).stop 2024-03-04T12:36:39.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.456+0000 7f0a72f16640 1 -- 172.21.15.60:0/2769921294 shutdown_connections 2024-03-04T12:36:39.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.456+0000 7f0a72f16640 1 --2- 172.21.15.60:0/2769921294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c107a30 0x7f0a6c10e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.456+0000 7f0a72f16640 1 --2- 172.21.15.60:0/2769921294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a6c102ef0 0x7f0a6c107320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.464 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.456+0000 7f0a72f16640 1 --2- 172.21.15.60:0/2769921294 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a6c102540 0x7f0a6c102920 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.456+0000 7f0a72f16640 1 -- 172.21.15.60:0/2769921294 >> 172.21.15.60:0/2769921294 conn(0x7f0a6c077410 msgr2=0x7f0a6c077820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:39.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.456+0000 7f0a72f16640 1 -- 172.21.15.60:0/2769921294 shutdown_connections 2024-03-04T12:36:39.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.456+0000 7f0a72f16640 1 -- 172.21.15.60:0/2769921294 wait complete. 2024-03-04T12:36:39.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a72f16640 1 Processor -- start 2024-03-04T12:36:39.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a72f16640 1 -- start start 2024-03-04T12:36:39.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a72f16640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a6c102540 0x7f0a6c19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:39.465 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a72f16640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a6c107a30 0x7f0a6c19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:39.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a72f16640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c197b20 0x7f0a6c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:39.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a72f16640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c111840 con 0x7f0a6c102540 2024-03-04T12:36:39.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a72f16640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c1116c0 con 0x7f0a6c107a30 2024-03-04T12:36:39.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a72f16640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c1119c0 con 0x7f0a6c197b20 2024-03-04T12:36:39.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a70cb2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a6c102540 0x7f0a6c19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:39.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a70cb2640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a6c102540 0x7f0a6c19da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59126/0 (socket says 172.21.15.60:59126) 2024-03-04T12:36:39.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a70cb2640 1 -- 172.21.15.60:0/588671285 learned_addr learned my addr 172.21.15.60:0/588671285 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:39.466 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a63fff640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a6c107a30 0x7f0a6c19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:39.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a714b3640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c197b20 0x7f0a6c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:39.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a70cb2640 1 -- 172.21.15.60:0/588671285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c197b20 msgr2=0x7f0a6c197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:39.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a70cb2640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c197b20 0x7f0a6c197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a70cb2640 1 -- 172.21.15.60:0/588671285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a6c107a30 msgr2=0x7f0a6c19dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:39.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a70cb2640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a6c107a30 0x7f0a6c19dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:39.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a70cb2640 1 -- 172.21.15.60:0/588671285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c198720 con 0x7f0a6c102540 2024-03-04T12:36:39.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.457+0000 7f0a714b3640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c197b20 0x7f0a6c197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:36:39.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.458+0000 7f0a70cb2640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a6c102540 0x7f0a6c19da60 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009230 tx=0x7f0a6802cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:39.467 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.458+0000 7f0a61ffb640 1 -- 172.21.15.60:0/588671285 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68041070 con 0x7f0a6c102540 2024-03-04T12:36:39.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.458+0000 7f0a72f16640 1 -- 172.21.15.60:0/588671285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c1989b0 con 0x7f0a6c102540 2024-03-04T12:36:39.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.458+0000 7f0a61ffb640 1 -- 172.21.15.60:0/588671285 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a68004450 con 0x7f0a6c102540 2024-03-04T12:36:39.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.458+0000 7f0a61ffb640 1 -- 172.21.15.60:0/588671285 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a68034440 con 0x7f0a6c102540 2024-03-04T12:36:39.468 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.458+0000 7f0a72f16640 1 -- 172.21.15.60:0/588671285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c1a4b40 con 0x7f0a6c102540 2024-03-04T12:36:39.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.459+0000 7f0a72f16640 1 -- 172.21.15.60:0/588671285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6c06a230 con 0x7f0a6c102540 2024-03-04T12:36:39.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.460+0000 7f0a61ffb640 1 -- 172.21.15.60:0/588671285 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f0a68044070 con 0x7f0a6c102540 2024-03-04T12:36:39.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.461+0000 7f0a61ffb640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0a540a1ef0 0x7f0a540a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:39.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.461+0000 7f0a63fff640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0a540a1ef0 0x7f0a540a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:39.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.461+0000 7f0a61ffb640 1 -- 172.21.15.60:0/588671285 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9881+0+0 (secure 0 0 0) 0x7f0a680f0070 con 0x7f0a6c102540 2024-03-04T12:36:39.469 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.461+0000 7f0a63fff640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0a540a1ef0 0x7f0a540a43b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c199300 tx=0x7f0a5c005f90 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:39.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.465+0000 7f0a61ffb640 1 -- 172.21.15.60:0/588671285 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0a680bccc0 con 0x7f0a6c102540 2024-03-04T12:36:39.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a[32741]: 2024-03-04T12:36:39.327+0000 7f423d882700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T12:36:39.678 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a[75233]: ignoring --setuser ceph since I am not root 2024-03-04T12:36:39.678 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a[75233]: ignoring --setgroup ceph since I am not root 2024-03-04T12:36:39.679 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-mds[75237]: -- 172.21.15.60:0/1719173049 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ce4688ba40 con 0x55ce4685ec00 2024-03-04T12:36:39.679 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a[75233]: starting mds.a at 2024-03-04T12:36:39.679 INFO:journalctl@ceph.mds.g.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-g[74874]: ignoring --setuser ceph since I am not root 2024-03-04T12:36:39.679 INFO:journalctl@ceph.mds.g.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-g[74874]: ignoring --setgroup ceph since I am not root 2024-03-04T12:36:39.679 INFO:journalctl@ceph.mds.g.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-mds[74878]: -- 172.21.15.60:0/365310755 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bf51211a40 con 0x55bf511e4800 2024-03-04T12:36:39.679 INFO:journalctl@ceph.mds.g.smithi060.stdout:Mar 04 12:36:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-g[74874]: starting mds.g at 2024-03-04T12:36:39.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:39.686+0000 7f0a72f16640 1 -- 172.21.15.60:0/588671285 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0a6c109a40 con 0x7f0a6c102540 2024-03-04T12:36:39.848 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[71488]: ignoring --setuser ceph since I am not root 2024-03-04T12:36:39.849 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[71488]: ignoring --setgroup ceph since I am not root 2024-03-04T12:36:39.849 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-mds[71492]: -- 172.21.15.86:0/2998613233 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558fe82c1a40 con 0x558fe8294c00 2024-03-04T12:36:39.849 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[71488]: starting mds.c at 2024-03-04T12:36:39.850 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-f[67540]: ignoring --setuser ceph since I am not root 2024-03-04T12:36:39.850 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-f[67540]: ignoring --setgroup ceph since I am not root 2024-03-04T12:36:39.850 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-mds[67544]: -- 172.21.15.86:0/3273349015 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5604d6eaba40 con 0x5604d6e7ec00 2024-03-04T12:36:39.850 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-f[67540]: starting mds.f at 2024-03-04T12:36:39.850 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-l[67216]: ignoring --setuser ceph since I am not root 2024-03-04T12:36:39.851 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-l[67216]: ignoring --setgroup ceph since I am not root 2024-03-04T12:36:39.851 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-l[67216]: starting mds.l at 2024-03-04T12:36:39.851 INFO:journalctl@ceph.mds.i.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-i[68150]: ignoring --setuser ceph since I am not root 2024-03-04T12:36:39.851 INFO:journalctl@ceph.mds.i.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-i[68150]: ignoring --setgroup ceph since I am not root 2024-03-04T12:36:39.851 INFO:journalctl@ceph.mds.i.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-mds[68154]: -- 172.21.15.86:0/3601715425 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55debf0b7a40 con 0x55debf08a800 2024-03-04T12:36:39.851 INFO:journalctl@ceph.mds.i.smithi086.stdout:Mar 04 12:36:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-i[68150]: starting mds.i at 2024-03-04T12:36:39.852 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: ignoring --setuser ceph since I am not root 2024-03-04T12:36:39.852 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: ignoring --setgroup ceph since I am not root 2024-03-04T12:36:39.852 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-mds[67103]: -- 172.21.15.63:0/3805406212 <== mon.0 v2:172.21.15.60:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563f76607a40 con 0x563f765da800 2024-03-04T12:36:39.852 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: starting mds.h at 2024-03-04T12:36:39.853 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b[67411]: ignoring --setuser ceph since I am not root 2024-03-04T12:36:39.853 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b[67411]: ignoring --setgroup ceph since I am not root 2024-03-04T12:36:39.853 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-mds[67415]: -- 172.21.15.63:0/1175024782 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56073e33fa40 con 0x56073e312800 2024-03-04T12:36:39.854 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b[67411]: starting mds.b at 2024-03-04T12:36:39.854 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e[66792]: ignoring --setuser ceph since I am not root 2024-03-04T12:36:39.854 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e[66792]: ignoring --setgroup ceph since I am not root 2024-03-04T12:36:39.854 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-mds[66796]: -- 172.21.15.63:0/3605464764 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fc1e579a40 con 0x55fc1e54cc00 2024-03-04T12:36:39.854 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e[66792]: starting mds.e at 2024-03-04T12:36:39.855 INFO:journalctl@ceph.mds.k.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-k[66470]: ignoring --setuser ceph since I am not root 2024-03-04T12:36:39.855 INFO:journalctl@ceph.mds.k.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-k[66470]: ignoring --setgroup ceph since I am not root 2024-03-04T12:36:39.855 INFO:journalctl@ceph.mds.k.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-mds[66474]: -- 172.21.15.63:0/1439577341 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5616a7ab9a40 con 0x5616a7a8d000 2024-03-04T12:36:39.855 INFO:journalctl@ceph.mds.k.smithi063.stdout:Mar 04 12:36:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-k[66470]: starting mds.k at 2024-03-04T12:36:40.340 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.334+0000 7f0a61ffb640 1 -- 172.21.15.60:0/588671285 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v42) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0a680bc5c0 con 0x7f0a6c102540 2024-03-04T12:36:40.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.337+0000 7f0a72f16640 1 -- 172.21.15.60:0/588671285 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0a540a1ef0 msgr2=0x7f0a540a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:40.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.337+0000 7f0a72f16640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0a540a1ef0 0x7f0a540a43b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c199300 tx=0x7f0a5c005f90 comp rx=0 tx=0).stop 2024-03-04T12:36:40.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.337+0000 7f0a72f16640 1 -- 172.21.15.60:0/588671285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a6c102540 msgr2=0x7f0a6c19da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:40.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.337+0000 7f0a72f16640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a6c102540 0x7f0a6c19da60 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009230 tx=0x7f0a6802cba0 comp rx=0 tx=0).stop 2024-03-04T12:36:40.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.337+0000 7f0a72f16640 1 -- 172.21.15.60:0/588671285 shutdown_connections 2024-03-04T12:36:40.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.337+0000 7f0a72f16640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6c197b20 0x7f0a6c197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:40.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.337+0000 7f0a72f16640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0a6c107a30 0x7f0a6c19dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:40.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.337+0000 7f0a72f16640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f0a540a1ef0 0x7f0a540a43b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:40.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.337+0000 7f0a72f16640 1 --2- 172.21.15.60:0/588671285 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f0a6c102540 0x7f0a6c19da60 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:40.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.337+0000 7f0a72f16640 1 -- 172.21.15.60:0/588671285 >> 172.21.15.60:0/588671285 conn(0x7f0a6c077410 msgr2=0x7f0a6c0ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:40.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.337+0000 7f0a72f16640 1 -- 172.21.15.60:0/588671285 shutdown_connections 2024-03-04T12:36:40.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.337+0000 7f0a72f16640 1 -- 172.21.15.60:0/588671285 wait complete. 2024-03-04T12:36:40.358 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:40 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3109237465' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-04T12:36:40.358 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:40 smithi060 ceph-mon[32745]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-04T12:36:40.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:40 smithi060 ceph-mon[32745]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T12:36:40.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:40 smithi060 ceph-mon[32745]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-03-04T12:36:40.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:40 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3109237465' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-04T12:36:40.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:40 smithi060 ceph-mon[32745]: osdmap e106: 12 total, 12 up, 12 in 2024-03-04T12:36:40.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:40 smithi060 ceph-mon[32745]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-04T12:36:40.359 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:40 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/588671285' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T12:36:40.360 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-04T12:36:40.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.466+0000 7f7311d59640 1 -- 172.21.15.60:0/2408322749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f730c1022d0 msgr2=0x7f730c1026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:40.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.466+0000 7f7311d59640 1 --2- 172.21.15.60:0/2408322749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f730c1022d0 0x7f730c1026b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009900 tx=0x7f72fc02bc30 comp rx=0 tx=0).stop 2024-03-04T12:36:40.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.466+0000 7f7311d59640 1 -- 172.21.15.60:0/2408322749 shutdown_connections 2024-03-04T12:36:40.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.466+0000 7f7311d59640 1 --2- 172.21.15.60:0/2408322749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730c1012b0 0x7f730c10d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:40.477 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.466+0000 7f7311d59640 1 --2- 172.21.15.60:0/2408322749 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c100910 0x7f730c100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:40.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.466+0000 7f7311d59640 1 --2- 172.21.15.60:0/2408322749 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f730c1022d0 0x7f730c1026b0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:40.478 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.466+0000 7f7311d59640 1 -- 172.21.15.60:0/2408322749 >> 172.21.15.60:0/2408322749 conn(0x7f730c076720 msgr2=0x7f730c0fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:40.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.466+0000 7f7311d59640 1 -- 172.21.15.60:0/2408322749 shutdown_connections 2024-03-04T12:36:40.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.466+0000 7f7311d59640 1 -- 172.21.15.60:0/2408322749 wait complete. 2024-03-04T12:36:40.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.466+0000 7f7311d59640 1 Processor -- start 2024-03-04T12:36:40.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f7311d59640 1 -- start start 2024-03-04T12:36:40.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f7311d59640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f730c100910 0x7f730c197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:40.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f7311d59640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730c1012b0 0x7f730c198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:40.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f7311d59640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c1022d0 0x7f730c19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:40.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f7311d59640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c111a50 con 0x7f730c100910 2024-03-04T12:36:40.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f7311d59640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c1118d0 con 0x7f730c1022d0 2024-03-04T12:36:40.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f7311d59640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c111bd0 con 0x7f730c1012b0 2024-03-04T12:36:40.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f730b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f730c100910 0x7f730c197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:40.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f730b7fe640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f730c100910 0x7f730c197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59142/0 (socket says 172.21.15.60:59142) 2024-03-04T12:36:40.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f730b7fe640 1 -- 172.21.15.60:0/3046282620 learned_addr learned my addr 172.21.15.60:0/3046282620 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:40.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f730b7fe640 1 -- 172.21.15.60:0/3046282620 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730c1012b0 msgr2=0x7f730c198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:40.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f730b7fe640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730c1012b0 0x7f730c198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:40.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f730bfff640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c1022d0 0x7f730c19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:40.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f730b7fe640 1 -- 172.21.15.60:0/3046282620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c1022d0 msgr2=0x7f730c19c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:40.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f730b7fe640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c1022d0 0x7f730c19c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:40.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f730b7fe640 1 -- 172.21.15.60:0/3046282620 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c19ce20 con 0x7f730c100910 2024-03-04T12:36:40.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f730bfff640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c1022d0 0x7f730c19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-04T12:36:40.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.467+0000 7f730b7fe640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f730c100910 0x7f730c197e30 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f72fc02efd0 tx=0x7f72fc004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:40.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.468+0000 7f7308ff9640 1 -- 172.21.15.60:0/3046282620 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc041070 con 0x7f730c100910 2024-03-04T12:36:40.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.468+0000 7f7311d59640 1 -- 172.21.15.60:0/3046282620 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c19d0b0 con 0x7f730c100910 2024-03-04T12:36:40.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.468+0000 7f7308ff9640 1 -- 172.21.15.60:0/3046282620 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72fc0054e0 con 0x7f730c100910 2024-03-04T12:36:40.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.468+0000 7f7308ff9640 1 -- 172.21.15.60:0/3046282620 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc003e30 con 0x7f730c100910 2024-03-04T12:36:40.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.468+0000 7f7311d59640 1 -- 172.21.15.60:0/3046282620 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c1a49a0 con 0x7f730c100910 2024-03-04T12:36:40.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.469+0000 7f7311d59640 1 -- 172.21.15.60:0/3046282620 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c101bc0 con 0x7f730c100910 2024-03-04T12:36:40.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.470+0000 7f7308ff9640 1 -- 172.21.15.60:0/3046282620 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f72fc044070 con 0x7f730c100910 2024-03-04T12:36:40.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.470+0000 7f7308ff9640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f72f40a1ef0 0x7f72f40a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:40.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.470+0000 7f730affd640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f72f40a1ef0 0x7f72f40a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:40.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.470+0000 7f7308ff9640 1 -- 172.21.15.60:0/3046282620 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9881+0+0 (secure 0 0 0) 0x7f72fc0f0300 con 0x7f730c100910 2024-03-04T12:36:40.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.471+0000 7f730affd640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f72f40a1ef0 0x7f72f40a43b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f730c0feb10 tx=0x7f7300005c70 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:40.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.475+0000 7f7308ff9640 1 -- 172.21.15.60:0/3046282620 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f72fc0bcfb0 con 0x7f730c100910 2024-03-04T12:36:40.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:40 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3109237465' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-04T12:36:40.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:40 smithi086 ceph-mon[36851]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-04T12:36:40.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:40 smithi086 ceph-mon[36851]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T12:36:40.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:40 smithi086 ceph-mon[36851]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-03-04T12:36:40.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:40 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3109237465' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-04T12:36:40.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:40 smithi086 ceph-mon[36851]: osdmap e106: 12 total, 12 up, 12 in 2024-03-04T12:36:40.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:40 smithi086 ceph-mon[36851]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-04T12:36:40.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:40 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/588671285' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T12:36:40.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:40 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3109237465' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-04T12:36:40.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:40 smithi063 ceph-mon[36717]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-04T12:36:40.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:40 smithi063 ceph-mon[36717]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-04T12:36:40.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:40 smithi063 ceph-mon[36717]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-03-04T12:36:40.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:40 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3109237465' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-04T12:36:40.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:40 smithi063 ceph-mon[36717]: osdmap e106: 12 total, 12 up, 12 in 2024-03-04T12:36:40.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:40 smithi063 ceph-mon[36717]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-04T12:36:40.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:40 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/588671285' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T12:36:40.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:40.658+0000 7f7311d59640 1 -- 172.21.15.60:0/3046282620 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f730c068600 con 0x7f730c100910 2024-03-04T12:36:41.341 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.335+0000 7f7308ff9640 1 -- 172.21.15.60:0/3046282620 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v107) v1 ==== 181+0+0 (secure 0 0 0) 0x7f72fc0bc8b0 con 0x7f730c100910 2024-03-04T12:36:41.341 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_metadata' removed 2024-03-04T12:36:41.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.339+0000 7f7311d59640 1 -- 172.21.15.60:0/3046282620 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f72f40a1ef0 msgr2=0x7f72f40a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:41.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.339+0000 7f7311d59640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f72f40a1ef0 0x7f72f40a43b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f730c0feb10 tx=0x7f7300005c70 comp rx=0 tx=0).stop 2024-03-04T12:36:41.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.339+0000 7f7311d59640 1 -- 172.21.15.60:0/3046282620 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f730c100910 msgr2=0x7f730c197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:41.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.339+0000 7f7311d59640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f730c100910 0x7f730c197e30 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f72fc02efd0 tx=0x7f72fc004430 comp rx=0 tx=0).stop 2024-03-04T12:36:41.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.339+0000 7f7311d59640 1 -- 172.21.15.60:0/3046282620 shutdown_connections 2024-03-04T12:36:41.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.339+0000 7f7311d59640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f730c1022d0 0x7f730c19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:41.345 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.339+0000 7f7311d59640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f730c1012b0 0x7f730c198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:41.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.339+0000 7f7311d59640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f72f40a1ef0 0x7f72f40a43b0 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f730c0feb10 tx=0x7f7300005c70 comp rx=0 tx=0).stop 2024-03-04T12:36:41.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.339+0000 7f7311d59640 1 --2- 172.21.15.60:0/3046282620 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f730c100910 0x7f730c197e30 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:41.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.339+0000 7f7311d59640 1 -- 172.21.15.60:0/3046282620 >> 172.21.15.60:0/3046282620 conn(0x7f730c076720 msgr2=0x7f730c105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:41.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.340+0000 7f7311d59640 1 -- 172.21.15.60:0/3046282620 shutdown_connections 2024-03-04T12:36:41.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.340+0000 7f7311d59640 1 -- 172.21.15.60:0/3046282620 wait complete. 2024-03-04T12:36:41.360 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: pgmap v338: 129 pgs: 129 active+clean; 80 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-03-04T12:36:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-04T12:36:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-04T12:36:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: Cluster is now healthy 2024-03-04T12:36:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/588671285' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-04T12:36:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.60:6839/476473380,v1:172.21.15.60:6841/476473380] up:boot 2024-03-04T12:36:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.86:6833/1825978487,v1:172.21.15.86:6835/1825978487] up:boot 2024-03-04T12:36:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.63:6833/2752509175,v1:172.21.15.63:6835/2752509175] up:boot 2024-03-04T12:36:41.361 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.60:6838/1053221568,v1:172.21.15.60:6840/1053221568] up:boot 2024-03-04T12:36:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.86:6837/4268995622,v1:172.21.15.86:6839/4268995622] up:boot 2024-03-04T12:36:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.86:6832/1619722325,v1:172.21.15.86:6834/1619722325] up:boot 2024-03-04T12:36:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.63:6832/1742802736,v1:172.21.15.63:6834/1742802736] up:boot 2024-03-04T12:36:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.63:6836/3431239869,v1:172.21.15.63:6837/3431239869] up:boot 2024-03-04T12:36:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.63:6838/1409133348,v1:172.21.15.63:6839/1409133348] up:boot 2024-03-04T12:36:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: mds.? [v2:172.21.15.86:6836/4013762235,v1:172.21.15.86:6838/4013762235] up:boot 2024-03-04T12:36:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: fsmap 12 up:standby 2024-03-04T12:36:41.362 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T12:36:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-04T12:36:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-04T12:36:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-04T12:36:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-04T12:36:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T12:36:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T12:36:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-04T12:36:41.363 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-04T12:36:41.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T12:36:41.364 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:41 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3046282620' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T12:36:41.365 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-04T12:36:41.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.471+0000 7f54c3309640 1 -- 172.21.15.60:0/3623287366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54bc0686a0 msgr2=0x7f54bc068a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:41.479 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.471+0000 7f54c3309640 1 --2- 172.21.15.60:0/3623287366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54bc0686a0 0x7f54bc068a80 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f54b4009900 tx=0x7f54b402bcd0 comp rx=0 tx=0).stop 2024-03-04T12:36:41.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 -- 172.21.15.60:0/3623287366 shutdown_connections 2024-03-04T12:36:41.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 --2- 172.21.15.60:0/3623287366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54bc10dcd0 0x7f54bc1100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:41.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 --2- 172.21.15.60:0/3623287366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54bc068fc0 0x7f54bc10d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:41.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 --2- 172.21.15.60:0/3623287366 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54bc0686a0 0x7f54bc068a80 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:41.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 -- 172.21.15.60:0/3623287366 >> 172.21.15.60:0/3623287366 conn(0x7f54bc077330 msgr2=0x7f54bc077740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:41.480 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 -- 172.21.15.60:0/3623287366 shutdown_connections 2024-03-04T12:36:41.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 -- 172.21.15.60:0/3623287366 wait complete. 2024-03-04T12:36:41.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 Processor -- start 2024-03-04T12:36:41.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 -- start start 2024-03-04T12:36:41.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54bc0686a0 0x7f54bc199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:41.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54bc068fc0 0x7f54bc19a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:41.481 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54bc10dcd0 0x7f54bc19e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:41.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc113860 con 0x7f54bc10dcd0 2024-03-04T12:36:41.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc1136e0 con 0x7f54bc068fc0 2024-03-04T12:36:41.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.472+0000 7f54c3309640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54bc1139e0 con 0x7f54bc0686a0 2024-03-04T12:36:41.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c18a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54bc10dcd0 0x7f54bc19e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:41.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c18a6640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54bc10dcd0 0x7f54bc19e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59168/0 (socket says 172.21.15.60:59168) 2024-03-04T12:36:41.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c18a6640 1 -- 172.21.15.60:0/1077903921 learned_addr learned my addr 172.21.15.60:0/1077903921 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:41.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c10a5640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54bc0686a0 0x7f54bc199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:41.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c18a6640 1 -- 172.21.15.60:0/1077903921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54bc0686a0 msgr2=0x7f54bc199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:41.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c18a6640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54bc0686a0 0x7f54bc199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:41.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c18a6640 1 -- 172.21.15.60:0/1077903921 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54bc068fc0 msgr2=0x7f54bc19a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:41.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c08a4640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54bc068fc0 0x7f54bc19a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:41.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c18a6640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54bc068fc0 0x7f54bc19a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:41.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c18a6640 1 -- 172.21.15.60:0/1077903921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc19ed20 con 0x7f54bc10dcd0 2024-03-04T12:36:41.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c18a6640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54bc10dcd0 0x7f54bc19e5f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f54b800cad0 tx=0x7f54b800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:41.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54b27fc640 1 -- 172.21.15.60:0/1077903921 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b800ae20 con 0x7f54bc10dcd0 2024-03-04T12:36:41.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54b27fc640 1 -- 172.21.15.60:0/1077903921 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54b8005020 con 0x7f54bc10dcd0 2024-03-04T12:36:41.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c3309640 1 -- 172.21.15.60:0/1077903921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54bc19f010 con 0x7f54bc10dcd0 2024-03-04T12:36:41.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54b27fc640 1 -- 172.21.15.60:0/1077903921 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b800f6f0 con 0x7f54bc10dcd0 2024-03-04T12:36:41.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.473+0000 7f54c3309640 1 -- 172.21.15.60:0/1077903921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54bc1a68a0 con 0x7f54bc10dcd0 2024-03-04T12:36:41.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.474+0000 7f54c3309640 1 -- 172.21.15.60:0/1077903921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54bc0681b0 con 0x7f54bc10dcd0 2024-03-04T12:36:41.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.475+0000 7f54b27fc640 1 -- 172.21.15.60:0/1077903921 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7f54b80051d0 con 0x7f54bc10dcd0 2024-03-04T12:36:41.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.476+0000 7f54b27fc640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f54a80a1f70 0x7f54a80a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:41.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.476+0000 7f54c10a5640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f54a80a1f70 0x7f54a80a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:41.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.476+0000 7f54b27fc640 1 -- 172.21.15.60:0/1077903921 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9385+0+0 (secure 0 0 0) 0x7f54b80ce600 con 0x7f54bc10dcd0 2024-03-04T12:36:41.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.476+0000 7f54c10a5640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f54a80a1f70 0x7f54a80a4430 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f54b4000c00 tx=0x7f54b402c830 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:41.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.480+0000 7f54b27fc640 1 -- 172.21.15.60:0/1077903921 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f54b8014030 con 0x7f54bc10dcd0 2024-03-04T12:36:41.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: pgmap v338: 129 pgs: 129 active+clean; 80 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-03-04T12:36:41.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-04T12:36:41.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-04T12:36:41.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: Cluster is now healthy 2024-03-04T12:36:41.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/588671285' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-04T12:36:41.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.60:6839/476473380,v1:172.21.15.60:6841/476473380] up:boot 2024-03-04T12:36:41.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.86:6833/1825978487,v1:172.21.15.86:6835/1825978487] up:boot 2024-03-04T12:36:41.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.63:6833/2752509175,v1:172.21.15.63:6835/2752509175] up:boot 2024-03-04T12:36:41.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.60:6838/1053221568,v1:172.21.15.60:6840/1053221568] up:boot 2024-03-04T12:36:41.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.86:6837/4268995622,v1:172.21.15.86:6839/4268995622] up:boot 2024-03-04T12:36:41.600 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.86:6832/1619722325,v1:172.21.15.86:6834/1619722325] up:boot 2024-03-04T12:36:41.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.63:6832/1742802736,v1:172.21.15.63:6834/1742802736] up:boot 2024-03-04T12:36:41.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.63:6836/3431239869,v1:172.21.15.63:6837/3431239869] up:boot 2024-03-04T12:36:41.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.63:6838/1409133348,v1:172.21.15.63:6839/1409133348] up:boot 2024-03-04T12:36:41.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: mds.? [v2:172.21.15.86:6836/4013762235,v1:172.21.15.86:6838/4013762235] up:boot 2024-03-04T12:36:41.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: fsmap 12 up:standby 2024-03-04T12:36:41.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T12:36:41.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-04T12:36:41.601 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-04T12:36:41.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-04T12:36:41.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-04T12:36:41.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T12:36:41.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T12:36:41.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-04T12:36:41.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-04T12:36:41.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T12:36:41.602 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:41 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3046282620' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T12:36:41.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: pgmap v338: 129 pgs: 129 active+clean; 80 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-03-04T12:36:41.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-04T12:36:41.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-04T12:36:41.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: Cluster is now healthy 2024-03-04T12:36:41.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/588671285' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-04T12:36:41.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.60:6839/476473380,v1:172.21.15.60:6841/476473380] up:boot 2024-03-04T12:36:41.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.86:6833/1825978487,v1:172.21.15.86:6835/1825978487] up:boot 2024-03-04T12:36:41.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.63:6833/2752509175,v1:172.21.15.63:6835/2752509175] up:boot 2024-03-04T12:36:41.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.60:6838/1053221568,v1:172.21.15.60:6840/1053221568] up:boot 2024-03-04T12:36:41.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.86:6837/4268995622,v1:172.21.15.86:6839/4268995622] up:boot 2024-03-04T12:36:41.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.86:6832/1619722325,v1:172.21.15.86:6834/1619722325] up:boot 2024-03-04T12:36:41.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.63:6832/1742802736,v1:172.21.15.63:6834/1742802736] up:boot 2024-03-04T12:36:41.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.63:6836/3431239869,v1:172.21.15.63:6837/3431239869] up:boot 2024-03-04T12:36:41.604 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.63:6838/1409133348,v1:172.21.15.63:6839/1409133348] up:boot 2024-03-04T12:36:41.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: mds.? [v2:172.21.15.86:6836/4013762235,v1:172.21.15.86:6838/4013762235] up:boot 2024-03-04T12:36:41.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: fsmap 12 up:standby 2024-03-04T12:36:41.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-04T12:36:41.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-04T12:36:41.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-04T12:36:41.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-04T12:36:41.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-04T12:36:41.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-04T12:36:41.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-04T12:36:41.605 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-04T12:36:41.606 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-04T12:36:41.606 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: from='mgr.15735 172.21.15.60:0/697811266' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-04T12:36:41.606 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:41 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3046282620' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T12:36:41.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:41.664+0000 7f54c3309640 1 -- 172.21.15.60:0/1077903921 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f54bc02ceb0 con 0x7f54bc10dcd0 2024-03-04T12:36:42.347 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.341+0000 7f54b27fc640 1 -- 172.21.15.60:0/1077903921 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v108) v1 ==== 169+0+0 (secure 0 0 0) 0x7f54b809b330 con 0x7f54bc10dcd0 2024-03-04T12:36:42.347 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data' removed 2024-03-04T12:36:42.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.345+0000 7f54c3309640 1 -- 172.21.15.60:0/1077903921 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f54a80a1f70 msgr2=0x7f54a80a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:42.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.345+0000 7f54c3309640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f54a80a1f70 0x7f54a80a4430 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f54b4000c00 tx=0x7f54b402c830 comp rx=0 tx=0).stop 2024-03-04T12:36:42.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.345+0000 7f54c3309640 1 -- 172.21.15.60:0/1077903921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54bc10dcd0 msgr2=0x7f54bc19e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:42.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.345+0000 7f54c3309640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54bc10dcd0 0x7f54bc19e5f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f54b800cad0 tx=0x7f54b800cf90 comp rx=0 tx=0).stop 2024-03-04T12:36:42.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.346+0000 7f54c3309640 1 -- 172.21.15.60:0/1077903921 shutdown_connections 2024-03-04T12:36:42.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.346+0000 7f54c3309640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7f54a80a1f70 0x7f54a80a4430 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:42.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.346+0000 7f54c3309640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7f54bc10dcd0 0x7f54bc19e5f0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:42.354 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.346+0000 7f54c3309640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54bc068fc0 0x7f54bc19a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:42.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.346+0000 7f54c3309640 1 --2- 172.21.15.60:0/1077903921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54bc0686a0 0x7f54bc199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:42.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.346+0000 7f54c3309640 1 -- 172.21.15.60:0/1077903921 >> 172.21.15.60:0/1077903921 conn(0x7f54bc077330 msgr2=0x7f54bc10f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:42.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.347+0000 7f54c3309640 1 -- 172.21.15.60:0/1077903921 shutdown_connections 2024-03-04T12:36:42.355 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.347+0000 7f54c3309640 1 -- 172.21.15.60:0/1077903921 wait complete. 2024-03-04T12:36:42.368 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-04T12:36:42.482 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.475+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2648130227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa281025a0 msgr2=0x7ffa28102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:42.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.475+0000 7ffa2d6e8640 1 --2- 172.21.15.60:0/2648130227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa281025a0 0x7ffa28102980 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009ca0 tx=0x7ffa1802c180 comp rx=0 tx=0).stop 2024-03-04T12:36:42.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.475+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2648130227 shutdown_connections 2024-03-04T12:36:42.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.475+0000 7ffa2d6e8640 1 --2- 172.21.15.60:0/2648130227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2810ba10 0x7ffa2810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:42.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.475+0000 7ffa2d6e8640 1 --2- 172.21.15.60:0/2648130227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa28102ec0 0x7ffa2810b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:42.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.475+0000 7ffa2d6e8640 1 --2- 172.21.15.60:0/2648130227 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa281025a0 0x7ffa28102980 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:42.483 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.475+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2648130227 >> 172.21.15.60:0/2648130227 conn(0x7ffa280773a0 msgr2=0x7ffa280777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:42.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.475+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2648130227 shutdown_connections 2024-03-04T12:36:42.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.475+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2648130227 wait complete. 2024-03-04T12:36:42.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.475+0000 7ffa2d6e8640 1 Processor -- start 2024-03-04T12:36:42.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa2d6e8640 1 -- start start 2024-03-04T12:36:42.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa2d6e8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa281025a0 0x7ffa2819d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:42.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa2d6e8640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa28102ec0 0x7ffa2819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:42.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa2d6e8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa2810ba10 0x7ffa28197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:42.484 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa2d6e8640 1 -- --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa281117c0 con 0x7ffa28102ec0 2024-03-04T12:36:42.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa2d6e8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa28111640 con 0x7ffa2810ba10 2024-03-04T12:36:42.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa2d6e8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa28111940 con 0x7ffa281025a0 2024-03-04T12:36:42.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa267fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa28102ec0 0x7ffa2819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:42.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa267fc640 1 --2- >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa28102ec0 0x7ffa2819de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.60:3300/0 says I am v2:172.21.15.60:59176/0 (socket says 172.21.15.60:59176) 2024-03-04T12:36:42.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa267fc640 1 -- 172.21.15.60:0/2405021724 learned_addr learned my addr 172.21.15.60:0/2405021724 (peer_addr_for_me v2:172.21.15.60:0/0) 2024-03-04T12:36:42.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa26ffd640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa281025a0 0x7ffa2819d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:42.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa277fe640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa2810ba10 0x7ffa28197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:42.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa267fc640 1 -- 172.21.15.60:0/2405021724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa281025a0 msgr2=0x7ffa2819d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:42.485 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa267fc640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa281025a0 0x7ffa2819d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:42.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa267fc640 1 -- 172.21.15.60:0/2405021724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa2810ba10 msgr2=0x7ffa28197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:42.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa267fc640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa2810ba10 0x7ffa28197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:42.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa267fc640 1 -- 172.21.15.60:0/2405021724 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa28198260 con 0x7ffa28102ec0 2024-03-04T12:36:42.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa277fe640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa2810ba10 0x7ffa28197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-04T12:36:42.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa267fc640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa28102ec0 0x7ffa2819de70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00d950 tx=0x7ffa1c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:42.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.476+0000 7ffa0ffff640 1 -- 172.21.15.60:0/2405021724 <== mon.0 v2:172.21.15.60:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c017070 con 0x7ffa28102ec0 2024-03-04T12:36:42.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.477+0000 7ffa0ffff640 1 -- 172.21.15.60:0/2405021724 <== mon.0 v2:172.21.15.60:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa1c009c30 con 0x7ffa28102ec0 2024-03-04T12:36:42.486 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.477+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2405021724 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa28198550 con 0x7ffa28102ec0 2024-03-04T12:36:42.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.477+0000 7ffa0ffff640 1 -- 172.21.15.60:0/2405021724 <== mon.0 v2:172.21.15.60:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c00f6b0 con 0x7ffa28102ec0 2024-03-04T12:36:42.487 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.477+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2405021724 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa281a47f0 con 0x7ffa28102ec0 2024-03-04T12:36:42.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.477+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2405021724 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2806a1f0 con 0x7ffa28102ec0 2024-03-04T12:36:42.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.479+0000 7ffa0ffff640 1 -- 172.21.15.60:0/2405021724 <== mon.0 v2:172.21.15.60:3300/0 4 ==== mgrmap(e 27) v1 ==== 136884+0+0 (secure 0 0 0) 0x7ffa1c0109d0 con 0x7ffa28102ec0 2024-03-04T12:36:42.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.480+0000 7ffa0ffff640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ffa100a1ef0 0x7ffa100a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-04T12:36:42.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.480+0000 7ffa26ffd640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ffa100a1ef0 0x7ffa100a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-04T12:36:42.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.480+0000 7ffa0ffff640 1 -- 172.21.15.60:0/2405021724 <== mon.0 v2:172.21.15.60:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8797+0+0 (secure 0 0 0) 0x7ffa1c0cdef0 con 0x7ffa28102ec0 2024-03-04T12:36:42.488 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.480+0000 7ffa26ffd640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ffa100a1ef0 0x7ffa100a43b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffa18004af0 tx=0x7ffa18005c50 comp rx=0 tx=0).ready entity=mgr.15735 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-04T12:36:42.489 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.483+0000 7ffa0ffff640 1 -- 172.21.15.60:0/2405021724 <== mon.0 v2:172.21.15.60:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffa1c009de0 con 0x7ffa28102ec0 2024-03-04T12:36:42.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:42 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/3046282620' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T12:36:42.598 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:42 smithi086 ceph-mon[36851]: osdmap e107: 12 total, 12 up, 12 in 2024-03-04T12:36:42.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:42 smithi086 ceph-mon[36851]: pgmap v340: 97 pgs: 97 active+clean; 26 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:36:42.599 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:42 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1077903921' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T12:36:42.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:42 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/3046282620' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T12:36:42.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:42 smithi063 ceph-mon[36717]: osdmap e107: 12 total, 12 up, 12 in 2024-03-04T12:36:42.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:42 smithi063 ceph-mon[36717]: pgmap v340: 97 pgs: 97 active+clean; 26 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:36:42.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:42 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1077903921' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T12:36:42.670 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:42.665+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2405021724 --> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffa28106a60 con 0x7ffa28102ec0 2024-03-04T12:36:42.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:42 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/3046282620' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T12:36:42.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:42 smithi060 ceph-mon[32745]: osdmap e107: 12 total, 12 up, 12 in 2024-03-04T12:36:42.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:42 smithi060 ceph-mon[32745]: pgmap v340: 97 pgs: 97 active+clean; 26 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-04T12:36:42.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:42 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1077903921' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T12:36:43.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.343+0000 7ffa0ffff640 1 -- 172.21.15.60:0/2405021724 <== mon.0 v2:172.21.15.60:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v109) v1 ==== 178+0+0 (secure 0 0 0) 0x7ffa1c09aed0 con 0x7ffa28102ec0 2024-03-04T12:36:43.349 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'cephfs_data_ec' removed 2024-03-04T12:36:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.346+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2405021724 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ffa100a1ef0 msgr2=0x7ffa100a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.346+0000 7ffa2d6e8640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ffa100a1ef0 0x7ffa100a43b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ffa18004af0 tx=0x7ffa18005c50 comp rx=0 tx=0).stop 2024-03-04T12:36:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.346+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2405021724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa28102ec0 msgr2=0x7ffa2819de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-04T12:36:43.352 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.346+0000 7ffa2d6e8640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa28102ec0 0x7ffa2819de70 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00d950 tx=0x7ffa1c00de10 comp rx=0 tx=0).stop 2024-03-04T12:36:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.346+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2405021724 shutdown_connections 2024-03-04T12:36:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.346+0000 7ffa2d6e8640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa2810ba10 0x7ffa28197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.346+0000 7ffa2d6e8640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.60:6800/248765372,v1:172.21.15.60:6801/248765372] conn(0x7ffa100a1ef0 0x7ffa100a43b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.346+0000 7ffa2d6e8640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] conn(0x7ffa28102ec0 0x7ffa2819de70 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.346+0000 7ffa2d6e8640 1 --2- 172.21.15.60:0/2405021724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa281025a0 0x7ffa2819d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-04T12:36:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.346+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2405021724 >> 172.21.15.60:0/2405021724 conn(0x7ffa280773a0 msgr2=0x7ffa280fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-04T12:36:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.347+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2405021724 shutdown_connections 2024-03-04T12:36:43.353 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-04T12:36:43.347+0000 7ffa2d6e8640 1 -- 172.21.15.60:0/2405021724 wait complete. 2024-03-04T12:36:43.372 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T12:36:43.405 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T12:36:43.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T12:36:43.478 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-04T12:36:43.479 DEBUG:teuthology.orchestra.run.smithi060:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T12:36:43.495 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T12:36:43.513 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T12:36:43.535 INFO:tasks.cephadm:Stopping all daemons... 2024-03-04T12:36:43.535 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-04T12:36:43.536 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.a 2024-03-04T12:36:43.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:43 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/1077903921' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T12:36:43.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:43 smithi060 ceph-mon[32745]: osdmap e108: 12 total, 12 up, 12 in 2024-03-04T12:36:43.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:43 smithi060 ceph-mon[32745]: from='client.? 172.21.15.60:0/2405021724' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T12:36:43.678 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:43 smithi060 systemd[1]: Stopping Ceph mon.a for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:36:43.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:43 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/1077903921' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T12:36:43.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:43 smithi086 ceph-mon[36851]: osdmap e108: 12 total, 12 up, 12 in 2024-03-04T12:36:43.848 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:43 smithi086 ceph-mon[36851]: from='client.? 172.21.15.60:0/2405021724' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T12:36:43.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:43 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/1077903921' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T12:36:43.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:43 smithi063 ceph-mon[36717]: osdmap e108: 12 total, 12 up, 12 in 2024-03-04T12:36:43.851 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:43 smithi063 ceph-mon[36717]: from='client.? 172.21.15.60:0/2405021724' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T12:36:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:43 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a[32741]: 2024-03-04T12:36:43.835+0000 7f424308d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:36:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:43 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a[32741]: 2024-03-04T12:36:43.835+0000 7f424308d700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-04T12:36:44.178 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:43 smithi060 podman[103503]: 2024-03-04 12:36:43.897241558 +0000 UTC m=+0.177780781 container died 421225a59508cb2b1a0566bfd30e7843d4ff6ce9a9846904fb394ecf477e8025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-04T12:36:44.627 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:44 smithi060 podman[103503]: 2024-03-04 12:36:44.287374143 +0000 UTC m=+0.567913365 container cleanup 421225a59508cb2b1a0566bfd30e7843d4ff6ce9a9846904fb394ecf477e8025 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222) 2024-03-04T12:36:44.628 INFO:journalctl@ceph.mon.a.smithi060.stdout:Mar 04 12:36:44 smithi060 bash[103503]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-a 2024-03-04T12:36:44.773 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.a.service' 2024-03-04T12:36:44.823 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:36:44.824 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-04T12:36:44.824 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-04T12:36:44.824 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.b 2024-03-04T12:36:45.166 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:44 smithi063 systemd[1]: Stopping Ceph mon.b for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:36:45.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:45 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-b[36713]: 2024-03-04T12:36:45.159+0000 7f3e2bfdb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:36:45.601 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:45 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-b[36713]: 2024-03-04T12:36:45.159+0000 7f3e2bfdb700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-04T12:36:45.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:45 smithi063 podman[72817]: 2024-03-04 12:36:45.221728478 +0000 UTC m=+0.183387233 container died 85077ac207da7629a49ca46f1aef7f2d63ec6f982f0d034cd17b1f03720c4e44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:36:46.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:45 smithi063 podman[72817]: 2024-03-04 12:36:45.827717927 +0000 UTC m=+0.789376680 container cleanup 85077ac207da7629a49ca46f1aef7f2d63ec6f982f0d034cd17b1f03720c4e44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-b, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5) 2024-03-04T12:36:46.101 INFO:journalctl@ceph.mon.b.smithi063.stdout:Mar 04 12:36:45 smithi063 bash[72817]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-b 2024-03-04T12:36:46.262 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.b.service' 2024-03-04T12:36:46.332 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:36:46.333 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-04T12:36:46.333 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-04T12:36:46.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.c 2024-03-04T12:36:46.684 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:46 smithi086 systemd[1]: Stopping Ceph mon.c for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:36:46.985 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:46 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-c[36835]: 2024-03-04T12:36:46.677+0000 7fa2a2be0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:36:46.985 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:46 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-c[36835]: 2024-03-04T12:36:46.678+0000 7fa2a2be0700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-04T12:36:46.985 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:46 smithi086 podman[73878]: 2024-03-04 12:36:46.760569181 +0000 UTC m=+0.241570633 container died 4bc60bafa8844131556585056afd940b1a1743599b3eb55cabf9c8aabe1869d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0) 2024-03-04T12:36:47.310 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:47 smithi086 podman[73878]: 2024-03-04 12:36:47.070275111 +0000 UTC m=+0.551276572 container cleanup 4bc60bafa8844131556585056afd940b1a1743599b3eb55cabf9c8aabe1869d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, ceph=True) 2024-03-04T12:36:47.310 INFO:journalctl@ceph.mon.c.smithi086.stdout:Mar 04 12:36:47 smithi086 bash[73878]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mon-c 2024-03-04T12:36:47.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mon.c.service' 2024-03-04T12:36:47.482 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:36:47.482 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-04T12:36:47.482 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-04T12:36:47.483 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.x 2024-03-04T12:36:47.806 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:36:47 smithi060 systemd[1]: Stopping Ceph mgr.x for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:36:48.094 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:36:47 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:36:47.800+0000 7f502e903700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:36:48.094 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:36:47 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x[32973]: 2024-03-04T12:36:47.800+0000 7f502e903700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-04T12:36:48.094 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:36:47 smithi060 podman[103627]: 2024-03-04 12:36:47.83245026 +0000 UTC m=+0.148810117 container died fbd5b7e7c4abcab1de820af83c91af70913d0633755ea8909216636527974333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-03-04T12:36:48.428 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:36:48 smithi060 podman[103627]: 2024-03-04 12:36:48.244320106 +0000 UTC m=+0.560679963 container cleanup fbd5b7e7c4abcab1de820af83c91af70913d0633755ea8909216636527974333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:717ce3cce8d0166e4e577faa8004e6f9cb4128c0, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_CLEAN=True) 2024-03-04T12:36:48.428 INFO:journalctl@ceph.mgr.x.smithi060.stdout:Mar 04 12:36:48 smithi060 bash[103627]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-x 2024-03-04T12:36:48.639 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.x.service' 2024-03-04T12:36:48.678 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:36:48.678 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-04T12:36:48.679 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-04T12:36:48.679 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.y 2024-03-04T12:36:49.034 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:36:48 smithi063 systemd[1]: Stopping Ceph mgr.y for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:36:49.351 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:36:49 smithi063 podman[72938]: 2024-03-04 12:36:49.028998947 +0000 UTC m=+0.150526465 container died 41d77cc52e14d3af7e577bb74731d0c3dbbc6bd2c61e32693da8f5b7c66aa6e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, io.buildah.version=1.33.5) 2024-03-04T12:36:49.699 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:36:49 smithi063 podman[72938]: 2024-03-04 12:36:49.467229339 +0000 UTC m=+0.588756822 container cleanup 41d77cc52e14d3af7e577bb74731d0c3dbbc6bd2c61e32693da8f5b7c66aa6e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-04T12:36:49.699 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:36:49 smithi063 bash[72938]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y 2024-03-04T12:36:49.943 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.y.service' 2024-03-04T12:36:49.971 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:36:49 smithi063 podman[72952]: 2024-03-04 12:36:49.693943892 +0000 UTC m=+0.657946620 container remove 41d77cc52e14d3af7e577bb74731d0c3dbbc6bd2c61e32693da8f5b7c66aa6e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-y, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222) 2024-03-04T12:36:49.971 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:36:49 smithi063 systemd[1]: ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-04T12:36:49.971 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:36:49 smithi063 systemd[1]: ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-04T12:36:49.971 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:36:49 smithi063 systemd[1]: Stopped Ceph mgr.y for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:36:49.971 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Mar 04 12:36:49 smithi063 systemd[1]: ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.y.service: Consumed 21.309s CPU time. 2024-03-04T12:36:49.985 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:36:49.985 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-04T12:36:49.985 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-04T12:36:49.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.z 2024-03-04T12:36:50.318 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:36:50 smithi086 systemd[1]: Stopping Ceph mgr.z for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:36:50.582 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:36:50 smithi086 podman[74000]: 2024-03-04 12:36:50.312708189 +0000 UTC m=+0.145741215 container died d4c9ad43a3c10f148e1e9ebb0e29e8d6a84fe07c6e5985c8068a55c2354d80c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-04T12:36:50.848 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:36:50 smithi086 podman[74000]: 2024-03-04 12:36:50.692197937 +0000 UTC m=+0.525230954 container cleanup d4c9ad43a3c10f148e1e9ebb0e29e8d6a84fe07c6e5985c8068a55c2354d80c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z, org.label-schema.build-date=20240222, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-04T12:36:50.848 INFO:journalctl@ceph.mgr.z.smithi086.stdout:Mar 04 12:36:50 smithi086 bash[74000]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mgr-z 2024-03-04T12:36:51.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mgr.z.service' 2024-03-04T12:36:51.189 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:36:51.190 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-04T12:36:51.190 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-04T12:36:51.190 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.0 2024-03-04T12:36:51.522 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:51 smithi060 systemd[1]: Stopping Ceph osd.0 for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:36:51.928 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:51 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0[47348]: 2024-03-04T12:36:51.515+0000 7f0cd2703700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:36:51.928 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:51 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0[47348]: 2024-03-04T12:36:51.515+0000 7f0cd2703700 -1 osd.0 109 *** Got signal Terminated *** 2024-03-04T12:36:51.928 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:51 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0[47348]: 2024-03-04T12:36:51.515+0000 7f0cd2703700 -1 osd.0 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T12:36:57.290 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:56 smithi060 podman[103760]: 2024-03-04 12:36:56.983363353 +0000 UTC m=+5.602641276 container died 861c1e5941d258998b5be1553d0509e8ff865605ff1b44c2a5db4301198288b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-04T12:36:57.640 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:57 smithi060 podman[103760]: 2024-03-04 12:36:57.367072268 +0000 UTC m=+5.986350197 container cleanup 861c1e5941d258998b5be1553d0509e8ff865605ff1b44c2a5db4301198288b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:36:57.640 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:57 smithi060 bash[103760]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0 2024-03-04T12:36:57.928 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:57 smithi060 podman[103777]: 2024-03-04 12:36:57.635458452 +0000 UTC m=+0.650823217 container remove 861c1e5941d258998b5be1553d0509e8ff865605ff1b44c2a5db4301198288b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5) 2024-03-04T12:36:58.227 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:57 smithi060 podman[103865]: 2024-03-04 12:36:57.877590788 +0000 UTC m=+0.018721044 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:36:58.228 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:58 smithi060 podman[103865]: 2024-03-04 12:36:58.020008867 +0000 UTC m=+0.161139122 container create f98cd8633165abfef25f8a73daefff3116825e5e1e813b226e2f9a0143a9c0c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-04T12:36:58.228 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:58 smithi060 podman[103865]: 2024-03-04 12:36:58.222796684 +0000 UTC m=+0.363926951 container init f98cd8633165abfef25f8a73daefff3116825e5e1e813b226e2f9a0143a9c0c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:36:58.551 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:58 smithi060 podman[103865]: 2024-03-04 12:36:58.225998055 +0000 UTC m=+0.367128318 container start f98cd8633165abfef25f8a73daefff3116825e5e1e813b226e2f9a0143a9c0c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222) 2024-03-04T12:36:58.551 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:58 smithi060 podman[103865]: 2024-03-04 12:36:58.331166744 +0000 UTC m=+0.472297020 container attach f98cd8633165abfef25f8a73daefff3116825e5e1e813b226e2f9a0143a9c0c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240222, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-04T12:36:58.857 INFO:journalctl@ceph.osd.0.smithi060.stdout:Mar 04 12:36:58 smithi060 podman[103865]: 2024-03-04 12:36:58.546363921 +0000 UTC m=+0.687494182 container died f98cd8633165abfef25f8a73daefff3116825e5e1e813b226e2f9a0143a9c0c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-04T12:36:59.183 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.0.service' 2024-03-04T12:36:59.224 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:36:59.224 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-04T12:36:59.224 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-04T12:36:59.225 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.1 2024-03-04T12:36:59.531 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:36:59 smithi063 systemd[1]: Stopping Ceph osd.1 for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:36:59.851 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:36:59 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1[41293]: 2024-03-04T12:36:59.554+0000 7f47f9354700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:36:59.852 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:36:59 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1[41293]: 2024-03-04T12:36:59.554+0000 7f47f9354700 -1 osd.1 109 *** Got signal Terminated *** 2024-03-04T12:36:59.855 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:36:59 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1[41293]: 2024-03-04T12:36:59.554+0000 7f47f9354700 -1 osd.1 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T12:37:05.130 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:37:04 smithi063 podman[73062]: 2024-03-04 12:37:04.86252938 +0000 UTC m=+5.442496451 container died 1af520c2fa6e239eb236c0291e42855991413fc1ec9b1e0571249513384f3180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-04T12:37:05.603 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:37:05 smithi063 podman[73062]: 2024-03-04 12:37:05.339158387 +0000 UTC m=+5.919125450 container cleanup 1af520c2fa6e239eb236c0291e42855991413fc1ec9b1e0571249513384f3180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:37:05.603 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:37:05 smithi063 bash[73062]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1 2024-03-04T12:37:05.974 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:37:05 smithi063 podman[73091]: 2024-03-04 12:37:05.641017505 +0000 UTC m=+0.773563379 container remove 1af520c2fa6e239eb236c0291e42855991413fc1ec9b1e0571249513384f3180 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:37:06.267 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:37:05 smithi063 podman[73183]: 2024-03-04 12:37:05.871922004 +0000 UTC m=+0.018333274 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:37:06.268 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:37:05 smithi063 podman[73183]: 2024-03-04 12:37:05.992140494 +0000 UTC m=+0.138551812 container create c8abe4bfc0f70662a7542357dc8dd7ff58a31b1341a824dd8266d02e7ecfd9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:37:06.268 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:37:06 smithi063 podman[73183]: 2024-03-04 12:37:06.262264642 +0000 UTC m=+0.408675903 container init c8abe4bfc0f70662a7542357dc8dd7ff58a31b1341a824dd8266d02e7ecfd9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:37:06.569 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:37:06 smithi063 podman[73183]: 2024-03-04 12:37:06.265912979 +0000 UTC m=+0.412324237 container start c8abe4bfc0f70662a7542357dc8dd7ff58a31b1341a824dd8266d02e7ecfd9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:37:06.569 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:37:06 smithi063 podman[73183]: 2024-03-04 12:37:06.395823509 +0000 UTC m=+0.542234782 container attach c8abe4bfc0f70662a7542357dc8dd7ff58a31b1341a824dd8266d02e7ecfd9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-04T12:37:06.852 INFO:journalctl@ceph.osd.1.smithi063.stdout:Mar 04 12:37:06 smithi063 podman[73183]: 2024-03-04 12:37:06.564411337 +0000 UTC m=+0.710822599 container died c8abe4bfc0f70662a7542357dc8dd7ff58a31b1341a824dd8266d02e7ecfd9af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:37:07.418 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.1.service' 2024-03-04T12:37:07.469 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:37:07.469 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-04T12:37:07.469 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-04T12:37:07.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.2 2024-03-04T12:37:07.763 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:07 smithi086 systemd[1]: Stopping Ceph osd.2 for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:37:08.098 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:07 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2[41969]: 2024-03-04T12:37:07.756+0000 7f9415060700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:37:08.098 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:07 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2[41969]: 2024-03-04T12:37:07.756+0000 7f9415060700 -1 osd.2 109 *** Got signal Terminated *** 2024-03-04T12:37:08.099 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:07 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2[41969]: 2024-03-04T12:37:07.756+0000 7f9415060700 -1 osd.2 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T12:37:13.479 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:13 smithi086 podman[74138]: 2024-03-04 12:37:13.189108947 +0000 UTC m=+5.538598375 container died 318ed0e9fe6c309daf81a9bd5370472bf86cf0ea452c34cba332d821a3bf9aa5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:37:13.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:13 smithi086 podman[74138]: 2024-03-04 12:37:13.573140827 +0000 UTC m=+5.922630258 container cleanup 318ed0e9fe6c309daf81a9bd5370472bf86cf0ea452c34cba332d821a3bf9aa5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-04T12:37:13.849 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:13 smithi086 bash[74138]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2 2024-03-04T12:37:14.333 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:13 smithi086 podman[74156]: 2024-03-04 12:37:13.933368687 +0000 UTC m=+0.737070888 container remove 318ed0e9fe6c309daf81a9bd5370472bf86cf0ea452c34cba332d821a3bf9aa5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True) 2024-03-04T12:37:14.584 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:14 smithi086 podman[74249]: 2024-03-04 12:37:14.231074655 +0000 UTC m=+0.017958088 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:37:14.585 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:14 smithi086 podman[74249]: 2024-03-04 12:37:14.334113984 +0000 UTC m=+0.120997403 container create 03895c6bb2e9cb734c3820aa4de7a401f7a69b0edd528157e48e7d14e03533e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, CEPH_POINT_RELEASE=) 2024-03-04T12:37:14.585 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:14 smithi086 podman[74249]: 2024-03-04 12:37:14.579448163 +0000 UTC m=+0.366331576 container init 03895c6bb2e9cb734c3820aa4de7a401f7a69b0edd528157e48e7d14e03533e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:37:14.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:14 smithi086 podman[74249]: 2024-03-04 12:37:14.582606856 +0000 UTC m=+0.369490283 container start 03895c6bb2e9cb734c3820aa4de7a401f7a69b0edd528157e48e7d14e03533e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_CLEAN=True) 2024-03-04T12:37:14.849 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:14 smithi086 podman[74249]: 2024-03-04 12:37:14.662851286 +0000 UTC m=+0.449734733 container attach 03895c6bb2e9cb734c3820aa4de7a401f7a69b0edd528157e48e7d14e03533e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-04T12:37:15.197 INFO:journalctl@ceph.osd.2.smithi086.stdout:Mar 04 12:37:14 smithi086 podman[74249]: 2024-03-04 12:37:14.875355315 +0000 UTC m=+0.662238739 container died 03895c6bb2e9cb734c3820aa4de7a401f7a69b0edd528157e48e7d14e03533e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:37:15.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.2.service' 2024-03-04T12:37:15.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:37:15.564 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-04T12:37:15.564 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-04T12:37:15.564 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.3 2024-03-04T12:37:15.868 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:15 smithi060 systemd[1]: Stopping Ceph osd.3 for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:37:16.177 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:15 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3[54334]: 2024-03-04T12:37:15.861+0000 7fd227101700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:37:16.178 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:15 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3[54334]: 2024-03-04T12:37:15.861+0000 7fd227101700 -1 osd.3 109 *** Got signal Terminated *** 2024-03-04T12:37:16.178 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:15 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3[54334]: 2024-03-04T12:37:15.861+0000 7fd227101700 -1 osd.3 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T12:37:19.348 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:18 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:18.877+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:20.178 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:19 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:19.829+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:20.249 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:19 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:19.893+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:21.178 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:20 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:20.792+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:21.178 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:20 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3[54334]: 2024-03-04T12:37:20.707+0000 7fd2236f5700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.268948+0000 front 2024-03-04T12:36:56.268972+0000 (oldest deadline 2024-03-04T12:37:20.369168+0000) 2024-03-04T12:37:21.348 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:20.908+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:21.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:21 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:21.020+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:21.464 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:21 smithi060 podman[103976]: 2024-03-04 12:37:21.226950354 +0000 UTC m=+5.467440424 container died 0249575dc6f23d79debbb018ef6f9a189a3ea8561af7e08f9d0e7b7738d47d5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5) 2024-03-04T12:37:21.812 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:21 smithi060 podman[103976]: 2024-03-04 12:37:21.622664357 +0000 UTC m=+5.863154426 container cleanup 0249575dc6f23d79debbb018ef6f9a189a3ea8561af7e08f9d0e7b7738d47d5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True) 2024-03-04T12:37:21.813 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:21 smithi060 bash[103976]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3 2024-03-04T12:37:22.098 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:21 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:21.916+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:22.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:21 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:21.670+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:22.178 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:21 smithi060 podman[103996]: 2024-03-04 12:37:21.807327895 +0000 UTC m=+0.580705670 container remove 0249575dc6f23d79debbb018ef6f9a189a3ea8561af7e08f9d0e7b7738d47d5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-04T12:37:22.178 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:21 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:21.831+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:22.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:22 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:22.016+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:22.524 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:22 smithi060 podman[104095]: 2024-03-04 12:37:22.163824156 +0000 UTC m=+0.017948363 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:37:22.525 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:22 smithi060 podman[104095]: 2024-03-04 12:37:22.275011736 +0000 UTC m=+0.129135949 container create e6dd6126c6b32715973a6abd63f33e12e35332c380a0d3ac5562c0874a8514dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:37:22.790 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:22 smithi060 podman[104095]: 2024-03-04 12:37:22.519914004 +0000 UTC m=+0.374038223 container init e6dd6126c6b32715973a6abd63f33e12e35332c380a0d3ac5562c0874a8514dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, ceph=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:37:22.791 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:22 smithi060 podman[104095]: 2024-03-04 12:37:22.523350722 +0000 UTC m=+0.377474928 container start e6dd6126c6b32715973a6abd63f33e12e35332c380a0d3ac5562c0874a8514dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0) 2024-03-04T12:37:22.791 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:22 smithi060 podman[104095]: 2024-03-04 12:37:22.63674431 +0000 UTC m=+0.490868529 container attach e6dd6126c6b32715973a6abd63f33e12e35332c380a0d3ac5562c0874a8514dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3-deactivate, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:37:23.098 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:22 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:22.896+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:23.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:22 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:22.683+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:23.137 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:22 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:22.784+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:23.138 INFO:journalctl@ceph.osd.3.smithi060.stdout:Mar 04 12:37:22 smithi060 podman[104095]: 2024-03-04 12:37:22.822630851 +0000 UTC m=+0.676755058 container died e6dd6126c6b32715973a6abd63f33e12e35332c380a0d3ac5562c0874a8514dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:37:23.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:23.035+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:23.467 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.3.service' 2024-03-04T12:37:23.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:37:23.512 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-04T12:37:23.512 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-04T12:37:23.513 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.4 2024-03-04T12:37:23.851 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:23 smithi063 systemd[1]: Stopping Ceph osd.4 for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:37:24.098 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:23 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:23.914+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:24.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:23 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:23.666+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:24.101 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:24.029+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:24.101 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4[48088]: 2024-03-04T12:37:23.844+0000 7f632af0c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:37:24.101 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4[48088]: 2024-03-04T12:37:23.844+0000 7f632af0c700 -1 osd.4 109 *** Got signal Terminated *** 2024-03-04T12:37:24.101 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:23 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4[48088]: 2024-03-04T12:37:23.844+0000 7f632af0c700 -1 osd.4 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T12:37:24.178 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:23 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:23.778+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:24.926 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:24 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:24.633+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:25.178 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:24 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:24.757+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:25.348 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:24 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:24.920+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:25.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:24 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:24.980+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:25.944 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:25 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:25.644+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:26.178 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:25 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:25.708+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:26.178 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:25 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:25.725+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:26.178 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:25 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:25.725+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:26.196 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:25 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:25.938+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:26.351 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4[48088]: 2024-03-04T12:37:25.930+0000 7f63234f8700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:36:59.146059+0000 front 2024-03-04T12:36:59.146163+0000 (oldest deadline 2024-03-04T12:37:25.046239+0000) 2024-03-04T12:37:26.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:25 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:25.952+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:26.983 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:26 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:26.614+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:27.178 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:26 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:26.710+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:27.178 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:26 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:26.685+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:27.178 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:26 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:26.685+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:27.348 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:26 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:26.977+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:27.351 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:26 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4[48088]: 2024-03-04T12:37:26.960+0000 7f63234f8700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:36:59.146059+0000 front 2024-03-04T12:36:59.146163+0000 (oldest deadline 2024-03-04T12:37:25.046239+0000) 2024-03-04T12:37:27.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:26 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:26.982+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:27.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:27 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:27.663+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:27.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:27 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:27.656+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:27.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:27 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:27.656+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:28.024 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:27 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:27.592+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:28.252 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:27 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4[48088]: 2024-03-04T12:37:27.996+0000 7f63234f8700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:36:59.146059+0000 front 2024-03-04T12:36:59.146163+0000 (oldest deadline 2024-03-04T12:37:25.046239+0000) 2024-03-04T12:37:28.252 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:28 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:28.006+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:28.351 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:28 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:28.066+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:28.351 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:28 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:28.018+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:28.351 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:28 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:28.018+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.666752+0000 front 2024-03-04T12:37:01.666742+0000 (oldest deadline 2024-03-04T12:37:27.566588+0000) 2024-03-04T12:37:28.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:28 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:28.646+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:28.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:28 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:28.624+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:28.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:28 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:28.624+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:29.009 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:28 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:28.625+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:29.039 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:28 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:28.715+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:29.348 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:29 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:29.091+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:29.349 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:29 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:29.003+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:29.349 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:29 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:29.003+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.666752+0000 front 2024-03-04T12:37:01.666742+0000 (oldest deadline 2024-03-04T12:37:27.566588+0000) 2024-03-04T12:37:29.351 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:29 smithi063 podman[73290]: 2024-03-04 12:37:29.19787095 +0000 UTC m=+5.484030055 container died 6bbacaaa55c9cb52865869f2305677b913daa6d75ebd276ccbc7a5580f255c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:37:29.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:29 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:29.033+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:29.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:29 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:29.624+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:29.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:29 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:29.593+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:29.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:29 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:29.593+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:29.930 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:29 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:29.762+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:29.930 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:29 smithi063 podman[73290]: 2024-03-04 12:37:29.714326143 +0000 UTC m=+6.000485230 container cleanup 6bbacaaa55c9cb52865869f2305677b913daa6d75ebd276ccbc7a5580f255c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:37:29.930 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:29 smithi063 bash[73290]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4 2024-03-04T12:37:29.963 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:29 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:29.594+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:29.963 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:29 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:29.594+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:30.342 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:29 smithi063 podman[73307]: 2024-03-04 12:37:29.925206971 +0000 UTC m=+0.723557371 container remove 6bbacaaa55c9cb52865869f2305677b913daa6d75ebd276ccbc7a5580f255c9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-04T12:37:30.343 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:30 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:30.080+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:30.343 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:30 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:30.080+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:30.348 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:30 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:30.136+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:30.348 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:29 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:29.957+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:30.348 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:29 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:29.957+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.666752+0000 front 2024-03-04T12:37:01.666742+0000 (oldest deadline 2024-03-04T12:37:27.566588+0000) 2024-03-04T12:37:30.601 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:30 smithi063 podman[73399]: 2024-03-04 12:37:30.240114757 +0000 UTC m=+0.018139122 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:37:30.602 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:30 smithi063 podman[73399]: 2024-03-04 12:37:30.359491825 +0000 UTC m=+0.137516198 container create fd287fb5ed7af8fb7c61e6f86b56748e1b909870f575f5d24bd5f1ba2287e808 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:37:30.602 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:30 smithi063 podman[73399]: 2024-03-04 12:37:30.578790379 +0000 UTC m=+0.356814729 container init fd287fb5ed7af8fb7c61e6f86b56748e1b909870f575f5d24bd5f1ba2287e808 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-04T12:37:30.602 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:30 smithi063 podman[73399]: 2024-03-04 12:37:30.58192121 +0000 UTC m=+0.359945555 container start fd287fb5ed7af8fb7c61e6f86b56748e1b909870f575f5d24bd5f1ba2287e808 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:37:30.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:30 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:30.552+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:30.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:30 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:30.552+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:30.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:30 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:30.590+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:30.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:30 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:30.636+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:30.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:30 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:30.636+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:31.092 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:30 smithi063 podman[73399]: 2024-03-04 12:37:30.678995662 +0000 UTC m=+0.457020035 container attach fd287fb5ed7af8fb7c61e6f86b56748e1b909870f575f5d24bd5f1ba2287e808 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0) 2024-03-04T12:37:31.092 INFO:journalctl@ceph.osd.4.smithi063.stdout:Mar 04 12:37:30 smithi063 podman[73399]: 2024-03-04 12:37:30.878507345 +0000 UTC m=+0.656531714 container died fd287fb5ed7af8fb7c61e6f86b56748e1b909870f575f5d24bd5f1ba2287e808 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:37:31.093 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:30 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:30.776+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:31.346 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:31 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:31.085+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:31.346 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:31 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:31.085+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:31.348 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:31 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:31.183+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:31.348 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:30 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:30.957+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:31.348 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:30 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:30.957+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.666752+0000 front 2024-03-04T12:37:01.666742+0000 (oldest deadline 2024-03-04T12:37:27.566588+0000) 2024-03-04T12:37:31.684 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.4.service' 2024-03-04T12:37:31.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:37:31.736 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-04T12:37:31.736 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-04T12:37:31.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.5 2024-03-04T12:37:31.848 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:31 smithi086 systemd[1]: Stopping Ceph osd.5 for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:37:31.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:31 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:31.589+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:31.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:31 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:31.589+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:31.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:31 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:31.600+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:31.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:31 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:31.652+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:31.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:31 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:31.652+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:32.101 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:31 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:31.769+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:32.220 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:31 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:31.926+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:32.221 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:31 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:31.926+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.666752+0000 front 2024-03-04T12:37:01.666742+0000 (oldest deadline 2024-03-04T12:37:27.566588+0000) 2024-03-04T12:37:32.221 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:32 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:32.027+0000 7f291d85c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:37:32.221 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:32 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:32.027+0000 7f291d85c700 -1 osd.5 109 *** Got signal Terminated *** 2024-03-04T12:37:32.221 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:32 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:32.027+0000 7f291d85c700 -1 osd.5 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T12:37:32.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:32 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:32.214+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:32.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:32 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:32.119+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:32.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:32 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:32.119+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:32.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:32 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:32.592+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:32.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:32 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:32.621+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:32.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:32 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:32.621+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:32.947 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:32 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:32.638+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:32.948 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:32 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:32.638+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:33.101 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:32 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:32.779+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:33.245 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:32 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:32.941+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:33.245 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:32 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:32.941+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.666752+0000 front 2024-03-04T12:37:01.666742+0000 (oldest deadline 2024-03-04T12:37:27.566588+0000) 2024-03-04T12:37:33.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:33 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:33.239+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:33.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:33 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:33.133+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:33.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:33 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:33.133+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:33.908 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:33 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:33.625+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:33.908 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:33 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:33.625+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:33.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:33 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:33.590+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:33.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:33 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:33.578+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:33.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:33 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:33.578+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:34.103 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:33 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:33.734+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:34.295 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:33 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:33.902+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:34.295 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:33 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:33.902+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.666752+0000 front 2024-03-04T12:37:01.666742+0000 (oldest deadline 2024-03-04T12:37:27.566588+0000) 2024-03-04T12:37:34.295 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:33 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:33.902+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.567961+0000 front 2024-03-04T12:37:11.567988+0000 (oldest deadline 2024-03-04T12:37:33.268122+0000) 2024-03-04T12:37:34.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:34 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:34.289+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:34.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:34 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:34.166+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:34.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:34 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:34.166+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:34.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:34 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:34.610+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:34.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:34 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:34.583+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:34.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:34 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:34.583+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:34.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:34 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:34.583+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:35.098 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:34 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:34.881+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:35.098 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:34 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:34.881+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.666752+0000 front 2024-03-04T12:37:01.666742+0000 (oldest deadline 2024-03-04T12:37:27.566588+0000) 2024-03-04T12:37:35.098 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:34 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:34.881+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.567961+0000 front 2024-03-04T12:37:11.567988+0000 (oldest deadline 2024-03-04T12:37:33.268122+0000) 2024-03-04T12:37:35.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:34 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:34.654+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:35.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:34 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:34.654+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:35.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:34 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:34.654+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:35.101 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:34 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:34.742+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:35.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:35 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:35.254+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:35.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:35 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:35.254+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:35.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:35 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:35.160+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:35.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:35 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:35.160+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:35.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:35 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:35.570+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:35.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:35 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:35.598+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:35.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:35 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:35.598+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:35.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:35 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:35.598+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:36.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:35 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:35.661+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:36.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:35 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:35.661+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:36.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:35 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:35.661+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:36.099 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:35 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:35.840+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:36.099 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:35 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:35.840+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.666752+0000 front 2024-03-04T12:37:01.666742+0000 (oldest deadline 2024-03-04T12:37:27.566588+0000) 2024-03-04T12:37:36.099 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:35 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:35.840+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.567961+0000 front 2024-03-04T12:37:11.567988+0000 (oldest deadline 2024-03-04T12:37:33.268122+0000) 2024-03-04T12:37:36.101 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:35 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:35.720+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:36.101 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:35 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:35.720+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:36.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:36 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:36.243+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:36.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:36 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:36.243+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:36.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:36 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:36.154+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:36.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:36 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:36.154+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:36.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:36 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:36.543+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:36.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:36 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:36.617+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:36.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:36 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:36.617+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:36.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:36 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:36.617+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:37.098 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:36 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:36.790+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.665565+0000 front 2024-03-04T12:36:54.665510+0000 (oldest deadline 2024-03-04T12:37:18.165711+0000) 2024-03-04T12:37:37.098 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:36 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:36.790+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.666752+0000 front 2024-03-04T12:37:01.666742+0000 (oldest deadline 2024-03-04T12:37:27.566588+0000) 2024-03-04T12:37:37.099 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:36 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5[48687]: 2024-03-04T12:37:36.790+0000 7f2915e48700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.567961+0000 front 2024-03-04T12:37:11.567988+0000 (oldest deadline 2024-03-04T12:37:33.268122+0000) 2024-03-04T12:37:37.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:36 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:36.635+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:37.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:36 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:36.635+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:37.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:36 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:36.635+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:37.101 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:36 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:36.714+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:37.101 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:36 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:36.714+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:37.598 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:37 smithi086 podman[74369]: 2024-03-04 12:37:37.448533343 +0000 UTC m=+5.525256750 container died fec9b645cfbf3afc484fee1127b66803802b79221cec6324ce9aca4f38cdc68b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:37:37.599 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:37 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:37.215+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:37.599 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:37 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:37.215+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:37.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:37 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:37.138+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:37.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:37 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:37.138+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:37.866 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:37 smithi086 podman[74369]: 2024-03-04 12:37:37.836766983 +0000 UTC m=+5.913490338 container cleanup fec9b645cfbf3afc484fee1127b66803802b79221cec6324ce9aca4f38cdc68b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-03-04T12:37:37.866 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:37 smithi086 bash[74369]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5 2024-03-04T12:37:37.867 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:37 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:37.602+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:37.867 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:37 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:37.602+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:37.867 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:37 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:37.602+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:37.927 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:37 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:37.560+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:37.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:37 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:37.560+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.534403+0000 front 2024-03-04T12:37:11.534200+0000 (oldest deadline 2024-03-04T12:37:36.834120+0000) 2024-03-04T12:37:37.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:37 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:37.581+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:37.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:37 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:37.581+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:37.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:37 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:37.581+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:38.101 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:37 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:37.676+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:38.101 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:37 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:37.676+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:38.412 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:38 smithi086 podman[74386]: 2024-03-04 12:37:38.09635738 +0000 UTC m=+0.640065832 container remove fec9b645cfbf3afc484fee1127b66803802b79221cec6324ce9aca4f38cdc68b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-04T12:37:38.413 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:38 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:38.220+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:38.413 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:38 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:38.220+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:38.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:38 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:38.126+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:38.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:38 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:38.126+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:38.681 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:38 smithi086 podman[74479]: 2024-03-04 12:37:38.309685678 +0000 UTC m=+0.017488093 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:37:38.681 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:38 smithi086 podman[74479]: 2024-03-04 12:37:38.438746749 +0000 UTC m=+0.146549160 container create e3c03cbcecae0c3740a4eee88956a5731955881e1b977d842271ed839adaedc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD) 2024-03-04T12:37:38.681 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:38 smithi086 podman[74479]: 2024-03-04 12:37:38.675299154 +0000 UTC m=+0.383101583 container init e3c03cbcecae0c3740a4eee88956a5731955881e1b977d842271ed839adaedc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) 2024-03-04T12:37:38.681 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:38 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:38.619+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:38.682 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:38 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:38.619+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:38.682 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:38 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:38.619+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:38.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:38 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:38.569+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:38.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:38 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:38.569+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.534403+0000 front 2024-03-04T12:37:11.534200+0000 (oldest deadline 2024-03-04T12:37:36.834120+0000) 2024-03-04T12:37:38.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:38 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:38.578+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:38.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:38 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:38.578+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:38.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:38 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:38.578+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:38.975 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:38 smithi086 podman[74479]: 2024-03-04 12:37:38.679369329 +0000 UTC m=+0.387171741 container start e3c03cbcecae0c3740a4eee88956a5731955881e1b977d842271ed839adaedc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-03-04T12:37:38.976 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:38 smithi086 podman[74479]: 2024-03-04 12:37:38.774881383 +0000 UTC m=+0.482683802 container attach e3c03cbcecae0c3740a4eee88956a5731955881e1b977d842271ed839adaedc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2) 2024-03-04T12:37:39.101 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:38 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:38.627+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:39.102 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:38 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:38.627+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:39.348 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 04 12:37:38 smithi086 podman[74479]: 2024-03-04 12:37:38.970363858 +0000 UTC m=+0.678166284 container died e3c03cbcecae0c3740a4eee88956a5731955881e1b977d842271ed839adaedc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux ) 2024-03-04T12:37:39.348 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:39.214+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:39.348 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:39.214+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:39.589 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:39.169+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:39.589 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:39.169+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:39.589 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:39.169+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:39.608 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:39.580+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:39.608 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:39.580+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:39.608 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:39 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:39.580+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:39.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.5.service' 2024-03-04T12:37:39.675 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:37:39.675 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-04T12:37:39.675 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-04T12:37:39.676 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.6 2024-03-04T12:37:39.852 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:39.582+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:39.852 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:39 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:39.582+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:39.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:39.553+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:39.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:39.553+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.534403+0000 front 2024-03-04T12:37:11.534200+0000 (oldest deadline 2024-03-04T12:37:36.834120+0000) 2024-03-04T12:37:39.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:39 smithi060 systemd[1]: Stopping Ceph osd.6 for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:37:39.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:39.541+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:39.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:39.541+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:39.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:39.541+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:40.428 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:39.985+0000 7ffaa3951700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:37:40.428 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:39.985+0000 7ffaa3951700 -1 osd.6 109 *** Got signal Terminated *** 2024-03-04T12:37:40.428 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:39 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:39.985+0000 7ffaa3951700 -1 osd.6 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T12:37:40.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:40 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:40.251+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:40.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:40 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:40.251+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:40.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:40 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:40.196+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:40.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:40 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:40.196+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:40.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:40 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:40.196+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:40.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:40.591+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:40.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:40.591+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.534403+0000 front 2024-03-04T12:37:11.534200+0000 (oldest deadline 2024-03-04T12:37:36.834120+0000) 2024-03-04T12:37:40.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:40.539+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:40.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:40.539+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:40.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:40.539+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:41.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:40 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:40.601+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:41.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:40 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:40.601+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:41.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:40 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:40.601+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:41.101 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:40 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:40.615+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:41.101 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:40 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:40.615+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:41.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:41 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:41.286+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:41.618 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:41 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:41.286+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:41.618 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:41 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:41.149+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:41.618 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:41 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:41.149+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:41.619 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:41 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:41.149+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:41.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:41 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:41.593+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:41.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:41 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:41.593+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:41.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:41 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:41.551+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:41.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:41 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:41.551+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.534403+0000 front 2024-03-04T12:37:11.534200+0000 (oldest deadline 2024-03-04T12:37:36.834120+0000) 2024-03-04T12:37:41.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:41 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:41.551+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:16.834630+0000 front 2024-03-04T12:37:16.834698+0000 (oldest deadline 2024-03-04T12:37:41.534796+0000) 2024-03-04T12:37:41.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:41 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:41.555+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:41.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:41 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:41.555+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:41.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:41 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:41.555+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:42.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:41 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:41.617+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:42.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:41 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:41.617+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:42.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:41 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:41.617+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:42.560 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:42 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:42.151+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:42.561 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:42 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:42.151+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:42.561 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:42 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:42.151+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:42.561 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:42 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:42.151+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:42.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:42 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:42.240+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:42.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:42 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:42.240+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:42.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:42 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:42.554+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:42.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:42 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:42.554+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:42.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:42.562+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:42.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:42.562+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.534403+0000 front 2024-03-04T12:37:11.534200+0000 (oldest deadline 2024-03-04T12:37:36.834120+0000) 2024-03-04T12:37:42.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:42.562+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:16.834630+0000 front 2024-03-04T12:37:16.834698+0000 (oldest deadline 2024-03-04T12:37:41.534796+0000) 2024-03-04T12:37:42.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:42.558+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:42.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:42.558+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:42.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:42 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:42.558+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:43.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:42 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:42.665+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:43.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:42 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:42.665+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:43.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:42 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:42.665+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:43.529 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:43 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:43.130+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:43.529 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:43 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:43.130+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:43.529 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:43 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:43.130+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:43.529 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:43 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:43.130+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:43.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:43 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:43.205+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:43.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:43 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:43.205+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:43.599 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:43 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:43.205+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:43.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:43 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:43.522+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:43.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:43 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:43.522+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:43.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:43 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:43.561+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:43.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:43 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:43.561+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.534403+0000 front 2024-03-04T12:37:11.534200+0000 (oldest deadline 2024-03-04T12:37:36.834120+0000) 2024-03-04T12:37:43.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:43 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:43.561+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:16.834630+0000 front 2024-03-04T12:37:16.834698+0000 (oldest deadline 2024-03-04T12:37:41.534796+0000) 2024-03-04T12:37:43.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:43 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:43.558+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:43.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:43 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:43.558+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:43.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:43 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:43.558+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:44.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:43 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:43.615+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:44.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:43 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:43.615+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:44.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:43 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:43.615+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:44.537 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:44 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:44.106+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:44.538 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:44 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:44.106+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:44.538 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:44 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:44.106+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:44.538 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:44 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:44.106+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:44.591 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:44 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:44.163+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:44.591 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:44 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:44.163+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:44.591 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:44 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:44.163+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:44.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:44 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:44.585+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:44.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:44 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:44.585+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:44.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:44 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:44.585+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:44.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:44 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:44.531+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:44.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:44 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:44.531+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:44.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:44 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:44.569+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.331953+0000 front 2024-03-04T12:37:02.331936+0000 (oldest deadline 2024-03-04T12:37:25.231975+0000) 2024-03-04T12:37:44.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:44 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:44.569+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.534403+0000 front 2024-03-04T12:37:11.534200+0000 (oldest deadline 2024-03-04T12:37:36.834120+0000) 2024-03-04T12:37:44.928 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:44 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6[61195]: 2024-03-04T12:37:44.569+0000 7ffa9ff45700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:16.834630+0000 front 2024-03-04T12:37:16.834698+0000 (oldest deadline 2024-03-04T12:37:41.534796+0000) 2024-03-04T12:37:44.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:44 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:44.538+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:44.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:44 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:44.538+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:44.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:44 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:44.538+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:45.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:45 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:45.095+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:45.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:45 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:45.095+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:45.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:45 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:45.095+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:45.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:45 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:45.095+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:45.591 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:45 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:45.173+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:45.591 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:45 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:45.173+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:45.591 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:45 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:45.173+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:45.783 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:45 smithi060 podman[104201]: 2024-03-04 12:37:45.476970759 +0000 UTC m=+5.611621858 container died 033afb30cd3c1974b39837bad72a6801025534bcd8e916be361d04be75c25e32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-04T12:37:45.783 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:45 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:45.587+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:45.783 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:45 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:45.587+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:45.784 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:45 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:45.587+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:45.784 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:45 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:45.587+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:45.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:45 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:45.585+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:45.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:45 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:45.585+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:45.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:45 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:45.585+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:45.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:45 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:45.521+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:45.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:45 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:45.521+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:46.150 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:45 smithi060 podman[104201]: 2024-03-04 12:37:45.877663632 +0000 UTC m=+6.012314730 container cleanup 033afb30cd3c1974b39837bad72a6801025534bcd8e916be361d04be75c25e32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:37:46.151 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:45 smithi060 bash[104201]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6 2024-03-04T12:37:46.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:46 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:46.063+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:46.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:46 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:46.063+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:46.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:46 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:46.063+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:46.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:46 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:46.063+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:46.428 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:46 smithi060 podman[104218]: 2024-03-04 12:37:46.145886093 +0000 UTC m=+0.668183601 container remove 033afb30cd3c1974b39837bad72a6801025534bcd8e916be361d04be75c25e32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-04T12:37:46.548 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:46 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:46.139+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:46.548 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:46 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:46.139+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:46.548 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:46 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:46.139+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:46.829 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:46 smithi060 podman[104309]: 2024-03-04 12:37:46.395564845 +0000 UTC m=+0.018474378 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:37:46.830 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:46 smithi060 podman[104309]: 2024-03-04 12:37:46.521883495 +0000 UTC m=+0.144793058 container create be15ae765252cb154331a695860b915148d7ac775ba51f557a10b91eee5f154d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:37:46.830 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:46 smithi060 podman[104309]: 2024-03-04 12:37:46.724368548 +0000 UTC m=+0.347278069 container init be15ae765252cb154331a695860b915148d7ac775ba51f557a10b91eee5f154d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-04T12:37:46.830 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:46 smithi060 podman[104309]: 2024-03-04 12:37:46.727531558 +0000 UTC m=+0.350441079 container start be15ae765252cb154331a695860b915148d7ac775ba51f557a10b91eee5f154d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:37:46.830 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:46 smithi060 podman[104309]: 2024-03-04 12:37:46.824531546 +0000 UTC m=+0.447441078 container attach be15ae765252cb154331a695860b915148d7ac775ba51f557a10b91eee5f154d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-04T12:37:46.830 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:46 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:46.583+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:46.830 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:46 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:46.583+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:46.831 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:46 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:46.583+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:46.831 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:46 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:46.583+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:46.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:46 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:46.542+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:46.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:46 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:46.542+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:46.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:46 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:46.542+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:46.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:46 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:46.518+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:46.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:46 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:46.518+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:46.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:46 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:46.518+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:15.228749+0000 front 2024-03-04T12:37:15.228788+0000 (oldest deadline 2024-03-04T12:37:46.429179+0000) 2024-03-04T12:37:47.178 INFO:journalctl@ceph.osd.6.smithi060.stdout:Mar 04 12:37:47 smithi060 podman[104309]: 2024-03-04 12:37:47.026243088 +0000 UTC m=+0.649152619 container died be15ae765252cb154331a695860b915148d7ac775ba51f557a10b91eee5f154d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-03-04T12:37:47.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:47 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:47.082+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:47.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:47 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:47.082+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:47.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:47 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:47.082+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:47.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:47 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:47.082+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:47.560 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:47 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:47.175+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:47.560 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:47 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:47.175+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:47.560 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:47 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:47.175+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:47.783 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:47 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:47.588+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:47.783 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:47 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:47.588+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:47.783 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:47 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:47.588+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:47.783 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:47 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:47.588+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:47.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.6.service' 2024-03-04T12:37:47.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:47 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:47.554+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:47.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:47 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:47.554+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:47.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:47 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:47.554+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:47.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:47 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:47.492+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:47.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:47 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:47.492+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:47.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:47 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:47.492+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:15.228749+0000 front 2024-03-04T12:37:15.228788+0000 (oldest deadline 2024-03-04T12:37:46.429179+0000) 2024-03-04T12:37:47.857 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:37:47.857 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-04T12:37:47.857 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-04T12:37:47.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.7 2024-03-04T12:37:48.209 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:47 smithi063 systemd[1]: Stopping Ceph osd.7 for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:37:48.209 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:48.113+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:48.210 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:48.113+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:48.210 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:48.113+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:48.210 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:48.113+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:48.468 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:48.202+0000 7f79d75b3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:37:48.469 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:48.202+0000 7f79d75b3700 -1 osd.7 109 *** Got signal Terminated *** 2024-03-04T12:37:48.469 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:48.202+0000 7f79d75b3700 -1 osd.7 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T12:37:48.469 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:48.462+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:48.470 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:48.462+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:48.470 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:48.462+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:15.228749+0000 front 2024-03-04T12:37:15.228788+0000 (oldest deadline 2024-03-04T12:37:46.429179+0000) 2024-03-04T12:37:48.553 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:48 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:48.150+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:48.554 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:48 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:48.150+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:48.554 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:48 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:48.150+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:48.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:48 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:48.547+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:48.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:48 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:48.547+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:48.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:48 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:48.547+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:48.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:48 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:48.617+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:48.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:48 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:48.617+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:48.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:48 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:48.617+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:48.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:48 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:48.617+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:49.432 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:49 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:49.100+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:49.432 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:49 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:49.100+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:49.432 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:49 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:49.100+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:49.432 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:49 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:49.100+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:49.583 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:49 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:49.133+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:49.583 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:49 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:49.133+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:49.583 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:49 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:49.133+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:49.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:49 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:49.577+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:49.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:49 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:49.577+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:49.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:49 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:49.577+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:49.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:49 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:49.426+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:49.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:49 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:49.426+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:49.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:49 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:49.426+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:15.228749+0000 front 2024-03-04T12:37:15.228788+0000 (oldest deadline 2024-03-04T12:37:46.429179+0000) 2024-03-04T12:37:49.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:49 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:49.622+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:49.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:49 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:49.622+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:49.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:49 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:49.622+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:49.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:49 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:49.622+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:50.465 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:50.126+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:50.465 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:50.126+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:50.465 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:50.126+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:50.466 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:50.126+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:50.466 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:50.126+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:37:50.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:50 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:50.154+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:50.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:50 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:50.154+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:50.599 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:50 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:50.154+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:50.599 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:50 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:50.154+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:27.637656+0000 front 2024-03-04T12:37:27.637701+0000 (oldest deadline 2024-03-04T12:37:49.937516+0000) 2024-03-04T12:37:50.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:50.459+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:50.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:50.459+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:50.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:50 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:50.459+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:15.228749+0000 front 2024-03-04T12:37:15.228788+0000 (oldest deadline 2024-03-04T12:37:46.429179+0000) 2024-03-04T12:37:50.868 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:50 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:50.617+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:50.868 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:50 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:50.617+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:50.869 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:50 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:50.617+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:50.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:50 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:50.665+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:50.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:50 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:50.665+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:50.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:50 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:50.665+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:50.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:50 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:50.665+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:51.442 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:51 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:51.113+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:51.442 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:51 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:51.113+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:51.442 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:51 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:51.114+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:51.442 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:51 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:51.114+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:51.442 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:51 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:51.114+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:37:51.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:51 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:51.139+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:51.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:51 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:51.139+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:51.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:51 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:51.139+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:51.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:51 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:51.139+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:27.637656+0000 front 2024-03-04T12:37:27.637701+0000 (oldest deadline 2024-03-04T12:37:49.937516+0000) 2024-03-04T12:37:51.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:51 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:51.591+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:51.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:51 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:51.591+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:51.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:51 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:51.591+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:51.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:51 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:51.591+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:37:51.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:51 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:51.435+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:51.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:51 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:51.435+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:51.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:51 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:51.435+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:15.228749+0000 front 2024-03-04T12:37:15.228788+0000 (oldest deadline 2024-03-04T12:37:46.429179+0000) 2024-03-04T12:37:51.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:51 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:51.435+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.429565+0000 front 2024-03-04T12:37:26.429597+0000 (oldest deadline 2024-03-04T12:37:50.529701+0000) 2024-03-04T12:37:51.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:51 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:51.669+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:51.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:51 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:51.669+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:51.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:51 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:51.669+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:51.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:51 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:51.669+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:52.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:52.072+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:52.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:52.072+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:52.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:52.072+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:52.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:52.072+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:52.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:52.072+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:37:52.582 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:52 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:52.165+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:52.582 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:52 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:52.165+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:52.582 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:52 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:52.165+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:52.582 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:52 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:52.165+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:27.637656+0000 front 2024-03-04T12:37:27.637701+0000 (oldest deadline 2024-03-04T12:37:49.937516+0000) 2024-03-04T12:37:52.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:52 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:52.576+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:52.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:52 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:52.576+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:52.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:52 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:52.576+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:52.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:52 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:52.576+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:37:52.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:52.392+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.825771+0000 front 2024-03-04T12:37:01.825808+0000 (oldest deadline 2024-03-04T12:37:27.725984+0000) 2024-03-04T12:37:52.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:52.392+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.328181+0000 front 2024-03-04T12:37:12.327939+0000 (oldest deadline 2024-03-04T12:37:35.227942+0000) 2024-03-04T12:37:52.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:52.392+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:15.228749+0000 front 2024-03-04T12:37:15.228788+0000 (oldest deadline 2024-03-04T12:37:46.429179+0000) 2024-03-04T12:37:52.851 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7[55084]: 2024-03-04T12:37:52.392+0000 7f79d3ba7700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.429565+0000 front 2024-03-04T12:37:26.429597+0000 (oldest deadline 2024-03-04T12:37:50.529701+0000) 2024-03-04T12:37:52.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:52 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:52.620+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:52.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:52 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:52.620+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:52.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:52 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:52.620+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:52.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:52 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:52.620+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:52.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:52 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:52.620+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:37:53.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:53 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:53.075+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:53.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:53 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:53.075+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:53.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:53 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:53.075+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:53.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:53 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:53.075+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:53.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:53 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:53.075+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:37:53.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:53 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:53.175+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:53.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:53 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:53.175+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:53.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:53 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:53.175+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:53.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:53 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:53.175+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:27.637656+0000 front 2024-03-04T12:37:27.637701+0000 (oldest deadline 2024-03-04T12:37:49.937516+0000) 2024-03-04T12:37:53.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:53.614+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:53.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:53.614+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:53.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:53.614+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:53.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:53.614+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:53.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:53 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:53.614+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:37:54.072 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:53 smithi063 podman[73505]: 2024-03-04 12:37:53.748934053 +0000 UTC m=+5.698090173 container died f3ad8b5be01eb91fed65a5ec2558d22415b0f3c298026653d9ceb69cc76ca619 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-04T12:37:54.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:53 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:53.605+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:54.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:53 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:53.605+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:54.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:53 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:53.605+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:54.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:53 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:53.605+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:37:54.351 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:54 smithi063 podman[73505]: 2024-03-04 12:37:54.229433595 +0000 UTC m=+6.178589713 container cleanup f3ad8b5be01eb91fed65a5ec2558d22415b0f3c298026653d9ceb69cc76ca619 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:37:54.351 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:54 smithi063 bash[73505]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7 2024-03-04T12:37:54.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:54 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:54.065+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:54.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:54 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:54.065+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:54.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:54 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:54.065+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:54.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:54 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:54.065+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:54.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:54 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:54.065+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:37:54.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:54 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:54.218+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:54.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:54 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:54.218+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:54.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:54 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:54.218+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:54.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:54 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:54.218+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:27.637656+0000 front 2024-03-04T12:37:27.637701+0000 (oldest deadline 2024-03-04T12:37:49.937516+0000) 2024-03-04T12:37:54.801 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:54 smithi063 podman[73522]: 2024-03-04 12:37:54.464367781 +0000 UTC m=+0.714645141 container remove f3ad8b5be01eb91fed65a5ec2558d22415b0f3c298026653d9ceb69cc76ca619 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5) 2024-03-04T12:37:54.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:54 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:54.567+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:54.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:54 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:54.567+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:54.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:54 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:54.567+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:54.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:54 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:54.567+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:54.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:54 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:54.567+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:37:55.064 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:54 smithi063 podman[73614]: 2024-03-04 12:37:54.699628922 +0000 UTC m=+0.018390099 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:37:55.064 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:54 smithi063 podman[73614]: 2024-03-04 12:37:54.932013112 +0000 UTC m=+0.250774268 container create 1ea74650448c03e8b2f55fc8accb5c144442e7567d7ff40db55d795e2fe92400 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7-deactivate, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:37:55.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:54 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:54.619+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:55.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:54 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:54.619+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:55.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:54 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:54.619+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:55.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:54 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:54.619+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:37:55.351 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:55 smithi063 podman[73614]: 2024-03-04 12:37:55.201264361 +0000 UTC m=+0.520025539 container init 1ea74650448c03e8b2f55fc8accb5c144442e7567d7ff40db55d795e2fe92400 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:37:55.351 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:55 smithi063 podman[73614]: 2024-03-04 12:37:55.204424254 +0000 UTC m=+0.523185418 container start 1ea74650448c03e8b2f55fc8accb5c144442e7567d7ff40db55d795e2fe92400 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-04T12:37:55.352 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:55 smithi063 podman[73614]: 2024-03-04 12:37:55.276442881 +0000 UTC m=+0.595204046 container attach 1ea74650448c03e8b2f55fc8accb5c144442e7567d7ff40db55d795e2fe92400 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:37:55.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:55 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:55.057+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:55.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:55 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:55.057+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:55.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:55 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:55.057+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:55.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:55 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:55.057+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:55.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:55 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:55.057+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:37:55.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:55 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:55.263+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:55.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:55 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:55.263+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:55.599 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:55 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:55.263+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:55.599 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:55 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:55.263+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:27.637656+0000 front 2024-03-04T12:37:27.637701+0000 (oldest deadline 2024-03-04T12:37:49.937516+0000) 2024-03-04T12:37:55.810 INFO:journalctl@ceph.osd.7.smithi063.stdout:Mar 04 12:37:55 smithi063 podman[73614]: 2024-03-04 12:37:55.509378301 +0000 UTC m=+0.828139466 container died 1ea74650448c03e8b2f55fc8accb5c144442e7567d7ff40db55d795e2fe92400 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-04T12:37:55.927 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:55 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:55.545+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:55.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:55 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:55.545+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:55.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:55 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:55.545+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:55.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:55 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:55.545+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:55.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:55 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:55.545+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:37:56.094 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.7.service' 2024-03-04T12:37:56.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:55 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:55.628+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:56.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:55 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:55.628+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:56.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:55 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:55.628+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:56.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:55 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:55.628+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:37:56.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:37:56.141 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-04T12:37:56.141 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-04T12:37:56.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.8 2024-03-04T12:37:56.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:56 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:56.070+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:56.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:56 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:56.070+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:56.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:56 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:56.070+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:56.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:56 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:56.070+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:56.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:56 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:56.070+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:37:56.461 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:56 smithi086 systemd[1]: Stopping Ceph osd.8 for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:37:56.461 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:56 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:56.214+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:56.462 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:56 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:56.214+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:56.462 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:56 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:56.214+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:56.462 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:56 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:56.214+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:27.637656+0000 front 2024-03-04T12:37:27.637701+0000 (oldest deadline 2024-03-04T12:37:49.937516+0000) 2024-03-04T12:37:56.848 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:56 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:56.455+0000 7fef6b4be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:37:56.848 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:56 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:56.455+0000 7fef6b4be700 -1 osd.8 109 *** Got signal Terminated *** 2024-03-04T12:37:56.848 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:56 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:56.455+0000 7fef6b4be700 -1 osd.8 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T12:37:56.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:56 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:56.641+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:56.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:56 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:56.641+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:56.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:56 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:56.641+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:56.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:56 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:56.641+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:37:56.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:56 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:56.534+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:56.993 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:56 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:56.534+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:56.993 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:56 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:56.534+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:56.994 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:56 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:56.534+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:56.994 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:56 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:56.534+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:37:57.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:57 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:57.210+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:57.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:57 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:57.210+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:57.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:57 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:57.211+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:57.599 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:57 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:57.211+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:27.637656+0000 front 2024-03-04T12:37:27.637701+0000 (oldest deadline 2024-03-04T12:37:49.937516+0000) 2024-03-04T12:37:57.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:57 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:57.119+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:57.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:57 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:57.119+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:57.602 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:57 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:57.119+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:57.602 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:57 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:57.119+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:57.602 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:57 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:57.119+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:37:57.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:57 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:57.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:57.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:57 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:57.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:57.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:57 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:57.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:57.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:57 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:57.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:57.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:57 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:57.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:37:58.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:57 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:57.680+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:58.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:57 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:57.680+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:58.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:57 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:57.680+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:58.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:57 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:57.680+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:37:58.503 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:58 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:58.134+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:58.503 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:58 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:58.134+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:58.504 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:58 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:58.134+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:58.504 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:58 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:58.134+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:58.504 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:58 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:58.134+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:37:58.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:58 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:58.242+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:58.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:58 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:58.242+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:58.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:58 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:58.242+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:58.599 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:58 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:58.242+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:27.637656+0000 front 2024-03-04T12:37:27.637701+0000 (oldest deadline 2024-03-04T12:37:49.937516+0000) 2024-03-04T12:37:58.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:58 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:58.519+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:58.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:58 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:58.519+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:58.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:58 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:58.519+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:58.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:58 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:58.519+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:58.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:58 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:58.519+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:37:59.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:58 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:58.682+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:37:59.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:58 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:58.682+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:37:59.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:58 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:58.682+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:37:59.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:58 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:58.682+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:37:59.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:59 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:59.193+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:37:59.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:59 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:59.193+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:37:59.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:59 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:59.193+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:37:59.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:37:59 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:37:59.193+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:27.637656+0000 front 2024-03-04T12:37:27.637701+0000 (oldest deadline 2024-03-04T12:37:49.937516+0000) 2024-03-04T12:37:59.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:59 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:59.170+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:37:59.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:59 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:59.170+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:37:59.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:59 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:59.170+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:37:59.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:59 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:59.170+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:37:59.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:37:59 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:37:59.170+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:37:59.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:59 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:59.553+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:37:59.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:59 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:59.553+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:37:59.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:59 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:59.553+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:37:59.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:59 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:59.553+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:37:59.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:59 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:59.553+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:37:59.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:37:59 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:37:59.553+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:35.758340+0000 front 2024-03-04T12:37:35.758448+0000 (oldest deadline 2024-03-04T12:37:59.258160+0000) 2024-03-04T12:38:00.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:59 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:59.641+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:00.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:59 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:59.641+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:00.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:59 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:59.641+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:00.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:59 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:59.641+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:00.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:37:59 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:37:59.641+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:00.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:00 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:38:00.229+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:38:00.598 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:00 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:38:00.229+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:38:00.599 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:00 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:38:00.229+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:38:00.599 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:00 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:38:00.229+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:27.637656+0000 front 2024-03-04T12:37:27.637701+0000 (oldest deadline 2024-03-04T12:37:49.937516+0000) 2024-03-04T12:38:00.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:00 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:00.161+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:00.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:00 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:00.161+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:00.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:00 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:00.161+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:00.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:00 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:00.161+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:00.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:00 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:00.161+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:00.602 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:00 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:00.161+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:00.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:00.581+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:38:00.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:00.581+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:38:00.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:00.581+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:38:00.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:00.581+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:38:00.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:00.581+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:38:00.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:00 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:00.581+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:35.758340+0000 front 2024-03-04T12:37:35.758448+0000 (oldest deadline 2024-03-04T12:37:59.258160+0000) 2024-03-04T12:38:01.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:00 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:00.627+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:01.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:00 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:00.627+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:01.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:00 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:00.627+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:01.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:00 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:00.627+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:01.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:00 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:00.627+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:01.590 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:01 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:38:01.256+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:01.234583+0000 front 2024-03-04T12:37:01.234619+0000 (oldest deadline 2024-03-04T12:37:27.134539+0000) 2024-03-04T12:38:01.591 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:01 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:38:01.256+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:12.435432+0000 front 2024-03-04T12:37:12.435616+0000 (oldest deadline 2024-03-04T12:37:34.735558+0000) 2024-03-04T12:38:01.591 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:01 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:38:01.256+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.436592+0000 front 2024-03-04T12:37:19.436446+0000 (oldest deadline 2024-03-04T12:37:42.936375+0000) 2024-03-04T12:38:01.591 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:01 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:38:01.256+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:27.637656+0000 front 2024-03-04T12:37:27.637701+0000 (oldest deadline 2024-03-04T12:37:49.937516+0000) 2024-03-04T12:38:01.591 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:01 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8[55696]: 2024-03-04T12:38:01.256+0000 7fef63aaa700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.538955+0000 front 2024-03-04T12:37:34.539028+0000 (oldest deadline 2024-03-04T12:38:00.439156+0000) 2024-03-04T12:38:01.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:01 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:01.130+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:01.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:01 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:01.130+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:01.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:01 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:01.130+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:01.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:01 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:01.130+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:01.602 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:01 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:01.130+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:01.602 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:01 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:01.130+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:01.848 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:01 smithi086 podman[74596]: 2024-03-04 12:38:01.776583308 +0000 UTC m=+5.439322150 container died a5f24ffd47af7ce7459f9e91bc02229f6dfd167952af26469bc342e0ee540110 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-04T12:38:01.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:01 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:01.584+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:01.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:01 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:01.584+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:01.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:01 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:01.584+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:01.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:01 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:01.584+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:01.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:01 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:01.584+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:01.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:01 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:01.590+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:38:01.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:01 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:01.590+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:38:01.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:01 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:01.590+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:38:01.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:01 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:01.590+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:38:01.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:01 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:01.590+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:38:01.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:01 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:01.590+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:35.758340+0000 front 2024-03-04T12:37:35.758448+0000 (oldest deadline 2024-03-04T12:37:59.258160+0000) 2024-03-04T12:38:02.584 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:02 smithi086 podman[74596]: 2024-03-04 12:38:02.164430861 +0000 UTC m=+5.827169678 container cleanup a5f24ffd47af7ce7459f9e91bc02229f6dfd167952af26469bc342e0ee540110 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222) 2024-03-04T12:38:02.585 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:02 smithi086 bash[74596]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8 2024-03-04T12:38:02.585 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:02 smithi086 podman[74614]: 2024-03-04 12:38:02.382513804 +0000 UTC m=+0.604887353 container remove a5f24ffd47af7ce7459f9e91bc02229f6dfd167952af26469bc342e0ee540110 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-04T12:38:02.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:02 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:02.111+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:02.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:02 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:02.111+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:02.602 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:02 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:02.111+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:02.602 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:02 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:02.111+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:02.602 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:02 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:02.111+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:02.602 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:02 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:02.111+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:02.848 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:02 smithi086 podman[74704]: 2024-03-04 12:38:02.719800151 +0000 UTC m=+0.017692163 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:38:02.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:02.578+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:02.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:02.578+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:02.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:02.578+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:02.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:02.578+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:02.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:02 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:02.578+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:02.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:02 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:02.594+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:38:02.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:02 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:02.594+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:38:02.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:02 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:02.594+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:38:02.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:02 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:02.594+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:38:02.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:02 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:02.594+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:38:02.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:02 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:02.594+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:35.758340+0000 front 2024-03-04T12:37:35.758448+0000 (oldest deadline 2024-03-04T12:37:59.258160+0000) 2024-03-04T12:38:03.135 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:02 smithi086 podman[74704]: 2024-03-04 12:38:02.878350275 +0000 UTC m=+0.176242266 container create 23c512e3ff1d451c0c6b8838ff8200c6acebc690221684d6c62318e401d73de6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-04T12:38:03.136 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:03 smithi086 podman[74704]: 2024-03-04 12:38:03.127344722 +0000 UTC m=+0.425236720 container init 23c512e3ff1d451c0c6b8838ff8200c6acebc690221684d6c62318e401d73de6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-04T12:38:03.136 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:03 smithi086 podman[74704]: 2024-03-04 12:38:03.130495602 +0000 UTC m=+0.428387610 container start 23c512e3ff1d451c0c6b8838ff8200c6acebc690221684d6c62318e401d73de6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:38:03.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:03 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:03.092+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:03.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:03 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:03.092+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:03.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:03 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:03.092+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:03.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:03 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:03.092+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:03.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:03 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:03.092+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:03.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:03 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:03.092+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:03.418 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:03 smithi086 podman[74704]: 2024-03-04 12:38:03.218993314 +0000 UTC m=+0.516885342 container attach 23c512e3ff1d451c0c6b8838ff8200c6acebc690221684d6c62318e401d73de6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-03-04T12:38:03.688 INFO:journalctl@ceph.osd.8.smithi086.stdout:Mar 04 12:38:03 smithi086 podman[74704]: 2024-03-04 12:38:03.413676869 +0000 UTC m=+0.711568867 container died 23c512e3ff1d451c0c6b8838ff8200c6acebc690221684d6c62318e401d73de6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:38:03.689 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:03 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:03.542+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:03.689 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:03 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:03.542+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:03.689 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:03 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:03.542+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:03.689 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:03 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:03.542+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:03.689 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:03 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:03.542+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:03.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:03 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:03.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:38:03.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:03 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:03.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:38:03.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:03 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:03.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:38:03.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:03 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:03.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:38:03.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:03 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:03.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:38:03.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:03 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:03.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:35.758340+0000 front 2024-03-04T12:37:35.758448+0000 (oldest deadline 2024-03-04T12:37:59.258160+0000) 2024-03-04T12:38:04.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.8.service' 2024-03-04T12:38:04.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:38:04.168 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-04T12:38:04.168 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-04T12:38:04.168 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.9 2024-03-04T12:38:04.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:04 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:04.063+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:04.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:04 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:04.063+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:04.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:04 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:04.063+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:04.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:04 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:04.063+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:04.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:04 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:04.063+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:04.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:04 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:04.063+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:04.486 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:04 smithi060 systemd[1]: Stopping Ceph osd.9 for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:38:04.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:04 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:04.576+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:04.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:04 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:04.576+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:04.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:04 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:04.576+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:04.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:04 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:04.576+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:04.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:04 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:04.576+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:04.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:04 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:04.480+0000 7fce4ced4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:38:04.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:04 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:04.480+0000 7fce4ced4700 -1 osd.9 109 *** Got signal Terminated *** 2024-03-04T12:38:04.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:04 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:04.480+0000 7fce4ced4700 -1 osd.9 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T12:38:04.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:04 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:04.551+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:38:04.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:04 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:04.551+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:38:04.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:04 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:04.551+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:38:04.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:04 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:04.551+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:38:04.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:04 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:04.551+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:38:04.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:04 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:04.551+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:35.758340+0000 front 2024-03-04T12:37:35.758448+0000 (oldest deadline 2024-03-04T12:37:59.258160+0000) 2024-03-04T12:38:05.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:05 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:05.033+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:05.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:05 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:05.033+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:05.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:05 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:05.033+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:05.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:05 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:05.033+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:05.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:05 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:05.033+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:05.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:05 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:05.033+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:05.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:05 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:05.574+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:05.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:05 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:05.574+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:05.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:05 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:05.574+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:05.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:05 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:05.574+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:05.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:05 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:05.574+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:05.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:05 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:05.505+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:38:05.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:05 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:05.505+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:38:05.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:05 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:05.505+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:38:05.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:05 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:05.505+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:38:05.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:05 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:05.505+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:38:05.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:05 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:05.505+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:35.758340+0000 front 2024-03-04T12:37:35.758448+0000 (oldest deadline 2024-03-04T12:37:59.258160+0000) 2024-03-04T12:38:06.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:05 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:05.989+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:06.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:05 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:05.989+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:06.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:05 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:05.989+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:06.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:05 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:05.989+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:06.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:05 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:05.989+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:06.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:05 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:05.989+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:06.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:06 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:06.546+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:38:06.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:06 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:06.546+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:38:06.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:06 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:06.546+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:38:06.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:06 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:06.546+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:38:06.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:06 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:06.546+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:38:06.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:06 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:06.546+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:35.758340+0000 front 2024-03-04T12:37:35.758448+0000 (oldest deadline 2024-03-04T12:37:59.258160+0000) 2024-03-04T12:38:07.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:06 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:06.610+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:07.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:06 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:06.610+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:07.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:06 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:06.610+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:07.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:06 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:06.610+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:07.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:06 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:06.610+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:07.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:06 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:06.610+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:07.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:07 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:07.025+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:07.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:07 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:07.025+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:07.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:07 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:07.025+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:07.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:07 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:07.025+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:07.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:07 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:07.025+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:07.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:07 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:07.025+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:07.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:07 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:07.560+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:38:07.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:07 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:07.560+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:38:07.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:07 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:07.560+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:38:07.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:07 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:07.560+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:38:07.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:07 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:07.560+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:38:07.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:07 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:07.560+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:35.758340+0000 front 2024-03-04T12:37:35.758448+0000 (oldest deadline 2024-03-04T12:37:59.258160+0000) 2024-03-04T12:38:08.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:07 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:07.600+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:08.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:07 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:07.600+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:08.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:07 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:07.600+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:08.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:07 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:07.600+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:08.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:07 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:07.600+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:08.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:07 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:07.600+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:08.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:08 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:08.060+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:08.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:08 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:08.060+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:08.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:08 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:08.060+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:08.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:08 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:08.060+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:08.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:08 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:08.060+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:08.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:08 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:08.060+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:08.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:08 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:08.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:54.649859+0000 front 2024-03-04T12:36:54.649741+0000 (oldest deadline 2024-03-04T12:37:19.349882+0000) 2024-03-04T12:38:08.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:08 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:08.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:02.251007+0000 front 2024-03-04T12:37:02.250900+0000 (oldest deadline 2024-03-04T12:37:25.151001+0000) 2024-03-04T12:38:08.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:08 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:08.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.152747+0000 front 2024-03-04T12:37:09.152650+0000 (oldest deadline 2024-03-04T12:37:33.852661+0000) 2024-03-04T12:38:08.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:08 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:08.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:20.254136+0000 front 2024-03-04T12:37:20.254160+0000 (oldest deadline 2024-03-04T12:37:44.954208+0000) 2024-03-04T12:38:08.928 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:08 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:08.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:28.355672+0000 front 2024-03-04T12:37:28.355759+0000 (oldest deadline 2024-03-04T12:37:51.855766+0000) 2024-03-04T12:38:08.929 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:08 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9[68236]: 2024-03-04T12:38:08.562+0000 7fce454c0700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:35.758340+0000 front 2024-03-04T12:37:35.758448+0000 (oldest deadline 2024-03-04T12:37:59.258160+0000) 2024-03-04T12:38:09.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:08 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:08.609+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:09.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:08 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:08.609+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:09.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:08 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:08.609+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:09.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:08 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:08.609+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:09.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:08 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:08.609+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:09.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:08 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:08.609+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:09.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:09 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:09.054+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:09.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:09 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:09.054+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:09.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:09 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:09.054+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:09.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:09 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:09.054+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:09.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:09 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:09.054+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:09.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:09 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:09.054+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:09.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:09 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:09.054+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.458388+0000 front 2024-03-04T12:37:42.458287+0000 (oldest deadline 2024-03-04T12:38:08.258558+0000) 2024-03-04T12:38:09.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:09 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:09.573+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:09.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:09 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:09.573+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:09.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:09 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:09.573+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:09.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:09 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:09.573+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:09.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:09 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:09.573+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:09.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:09 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:09.573+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:10.178 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:09 smithi060 podman[104427]: 2024-03-04 12:38:09.827033419 +0000 UTC m=+5.466210746 container died e2eed35f4b3113ee2916441c895a3a4f7357958279b247d5b12f86364fb918b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD) 2024-03-04T12:38:10.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.010+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:10.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.010+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:10.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.010+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:10.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.010+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:10.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.010+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:10.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.010+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:10.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.010+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.458388+0000 front 2024-03-04T12:37:42.458287+0000 (oldest deadline 2024-03-04T12:38:08.258558+0000) 2024-03-04T12:38:10.670 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:10 smithi060 podman[104427]: 2024-03-04 12:38:10.406001222 +0000 UTC m=+6.045178548 container cleanup e2eed35f4b3113ee2916441c895a3a4f7357958279b247d5b12f86364fb918b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=) 2024-03-04T12:38:10.670 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:10 smithi060 bash[104427]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9 2024-03-04T12:38:10.927 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:10 smithi060 podman[104444]: 2024-03-04 12:38:10.665582915 +0000 UTC m=+0.836704903 container remove e2eed35f4b3113ee2916441c895a3a4f7357958279b247d5b12f86364fb918b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux ) 2024-03-04T12:38:11.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:10 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:10.595+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:11.098 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:10 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:10.595+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:11.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:10 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:10.595+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:11.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:10 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:10.595+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:11.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:10 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:10.595+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:11.099 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:10 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:10.595+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:11.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.998+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:11.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.998+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:11.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.998+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:11.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.998+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:11.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.998+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:11.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.998+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:11.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:10 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:10.998+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.458388+0000 front 2024-03-04T12:37:42.458287+0000 (oldest deadline 2024-03-04T12:38:08.258558+0000) 2024-03-04T12:38:11.356 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:11 smithi060 podman[104536]: 2024-03-04 12:38:10.928362649 +0000 UTC m=+0.018232491 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:38:11.357 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:11 smithi060 podman[104536]: 2024-03-04 12:38:11.049677439 +0000 UTC m=+0.139547282 container create 31e7eae3c45bd77ce86b60704482ba3688598498dd6e97d447b1721bd87a836c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-03-04T12:38:11.357 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:11 smithi060 podman[104536]: 2024-03-04 12:38:11.268667342 +0000 UTC m=+0.358537198 container init 31e7eae3c45bd77ce86b60704482ba3688598498dd6e97d447b1721bd87a836c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9-deactivate, ceph=True, org.label-schema.build-date=20240222, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-04T12:38:11.357 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:11 smithi060 podman[104536]: 2024-03-04 12:38:11.271730196 +0000 UTC m=+0.361600038 container start 31e7eae3c45bd77ce86b60704482ba3688598498dd6e97d447b1721bd87a836c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:38:11.357 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:11 smithi060 podman[104536]: 2024-03-04 12:38:11.351867554 +0000 UTC m=+0.441737416 container attach 31e7eae3c45bd77ce86b60704482ba3688598498dd6e97d447b1721bd87a836c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-03-04T12:38:11.678 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:11 smithi060 podman[104536]: 2024-03-04 12:38:11.570507477 +0000 UTC m=+0.660377313 container died 31e7eae3c45bd77ce86b60704482ba3688598498dd6e97d447b1721bd87a836c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:38:11.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:11 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:11.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:11.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:11 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:11.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:11.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:11 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:11.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:11.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:11 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:11.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:11.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:11 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:11.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:11.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:11 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:11.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:12.196 INFO:journalctl@ceph.osd.9.smithi060.stdout:Mar 04 12:38:12 smithi060 podman[104561]: 2024-03-04 12:38:12.188986746 +0000 UTC m=+0.610153749 container remove 31e7eae3c45bd77ce86b60704482ba3688598498dd6e97d447b1721bd87a836c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-04T12:38:12.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.9.service' 2024-03-04T12:38:12.260 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:38:12.260 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-04T12:38:12.261 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-04T12:38:12.261 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.10 2024-03-04T12:38:12.314 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:12 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:12.043+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:12.366 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:12 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:12.043+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:12.366 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:12 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:12.043+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:12.366 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:12 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:12.043+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:12.366 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:12 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:12.043+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:12.367 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:12 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:12.043+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:12.367 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:12 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:12.043+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.458388+0000 front 2024-03-04T12:37:42.458287+0000 (oldest deadline 2024-03-04T12:38:08.258558+0000) 2024-03-04T12:38:12.367 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:12 smithi063 systemd[1]: Stopping Ceph osd.10 for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:38:12.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:12 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:12.530+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:12.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:12 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:12.530+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:12.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:12 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:12.530+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:12.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:12 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:12.530+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:12.850 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:12 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:12.530+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:12.850 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:12 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:12.530+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:12.851 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:12 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:12.583+0000 7f7297557700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:38:12.851 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:12 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:12.583+0000 7f7297557700 -1 osd.10 109 *** Got signal Terminated *** 2024-03-04T12:38:12.851 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:12 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:12.583+0000 7f7297557700 -1 osd.10 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T12:38:13.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:13 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:13.075+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:13.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:13 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:13.075+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:13.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:13 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:13.075+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:13.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:13 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:13.075+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:13.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:13 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:13.075+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:13.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:13 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:13.075+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:13.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:13 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:13.075+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.458388+0000 front 2024-03-04T12:37:42.458287+0000 (oldest deadline 2024-03-04T12:38:08.258558+0000) 2024-03-04T12:38:13.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:13 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:13.510+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:13.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:13 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:13.510+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:13.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:13 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:13.510+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:13.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:13 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:13.510+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:13.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:13 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:13.510+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:13.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:13 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:13.510+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:14.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:14 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:14.083+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:14.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:14 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:14.083+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:14.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:14 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:14.083+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:14.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:14 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:14.083+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:14.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:14 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:14.083+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:14.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:14 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:14.083+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:14.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:14 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:14.083+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.458388+0000 front 2024-03-04T12:37:42.458287+0000 (oldest deadline 2024-03-04T12:38:08.258558+0000) 2024-03-04T12:38:14.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:14 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:14.494+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:14.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:14 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:14.494+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:14.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:14 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:14.494+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:14.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:14 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:14.494+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:14.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:14 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:14.494+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:14.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:14 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:14.494+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:14.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:14 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:14.494+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-03-04T12:37:49.840349+0000 front 2024-03-04T12:37:49.840381+0000 (oldest deadline 2024-03-04T12:38:13.940267+0000) 2024-03-04T12:38:15.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:15 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:15.093+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:15.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:15 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:15.093+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:15.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:15 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:15.093+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:15.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:15 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:15.093+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:15.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:15 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:15.093+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:15.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:15 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:15.093+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:15.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:15 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:15.093+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.458388+0000 front 2024-03-04T12:37:42.458287+0000 (oldest deadline 2024-03-04T12:38:08.258558+0000) 2024-03-04T12:38:15.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:15.517+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:15.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:15.517+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:15.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:15.517+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:15.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:15.517+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:15.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:15.517+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:15.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:15.517+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:15.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:15 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:15.517+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-03-04T12:37:49.840349+0000 front 2024-03-04T12:37:49.840381+0000 (oldest deadline 2024-03-04T12:38:13.940267+0000) 2024-03-04T12:38:16.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:16 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:16.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:16.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:16 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:16.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:16.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:16 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:16.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:16.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:16 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:16.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:16.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:16 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:16.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:16.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:16 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:16.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:16.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:16 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:16.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.458388+0000 front 2024-03-04T12:37:42.458287+0000 (oldest deadline 2024-03-04T12:38:08.258558+0000) 2024-03-04T12:38:16.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:16 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:16.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:16.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:16 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:16.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:16.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:16 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:16.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:16.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:16 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:16.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:16.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:16 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:16.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:16.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:16 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:16.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:16.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:16 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:16.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-03-04T12:37:49.840349+0000 front 2024-03-04T12:37:49.840381+0000 (oldest deadline 2024-03-04T12:38:13.940267+0000) 2024-03-04T12:38:17.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:17 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:17.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:56.252220+0000 front 2024-03-04T12:36:56.252004+0000 (oldest deadline 2024-03-04T12:37:20.352090+0000) 2024-03-04T12:38:17.351 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:17 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:17.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.852943+0000 front 2024-03-04T12:37:03.853146+0000 (oldest deadline 2024-03-04T12:37:29.753098+0000) 2024-03-04T12:38:17.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:17 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:17.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:11.453860+0000 front 2024-03-04T12:37:11.453909+0000 (oldest deadline 2024-03-04T12:37:39.054478+0000) 2024-03-04T12:38:17.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:17 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:17.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6814 osd.3 since back 2024-03-04T12:37:19.055184+0000 front 2024-03-04T12:37:19.055062+0000 (oldest deadline 2024-03-04T12:37:41.354892+0000) 2024-03-04T12:38:17.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:17 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:17.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:26.056119+0000 front 2024-03-04T12:37:26.056089+0000 (oldest deadline 2024-03-04T12:37:49.556251+0000) 2024-03-04T12:38:17.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:17 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:17.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.857417+0000 front 2024-03-04T12:37:34.857273+0000 (oldest deadline 2024-03-04T12:37:59.557197+0000) 2024-03-04T12:38:17.352 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:17 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10[61705]: 2024-03-04T12:38:17.050+0000 7f7293b4b700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.458388+0000 front 2024-03-04T12:37:42.458287+0000 (oldest deadline 2024-03-04T12:38:08.258558+0000) 2024-03-04T12:38:17.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:17.541+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:17.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:17.541+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:17.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:17.541+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:17.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:17.541+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:17.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:17.541+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:17.850 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:17.541+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:17.850 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:17 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:17.541+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-03-04T12:37:49.840349+0000 front 2024-03-04T12:37:49.840381+0000 (oldest deadline 2024-03-04T12:38:13.940267+0000) 2024-03-04T12:38:18.323 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:17 smithi063 podman[73733]: 2024-03-04 12:38:17.925835348 +0000 UTC m=+5.465696867 container died 2fa9efedd41645e87ddf4ca5e1f0dfcaa0a9736cf5d2aa50c208893c95589da1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True) 2024-03-04T12:38:18.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:18 smithi063 podman[73733]: 2024-03-04 12:38:18.400262397 +0000 UTC m=+5.940123964 container cleanup 2fa9efedd41645e87ddf4ca5e1f0dfcaa0a9736cf5d2aa50c208893c95589da1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-04T12:38:18.601 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:18 smithi063 bash[73733]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10 2024-03-04T12:38:18.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:18 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:18.550+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:18.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:18 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:18.550+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:18.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:18 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:18.550+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:18.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:18 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:18.550+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:18.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:18 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:18.550+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:18.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:18 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:18.550+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:18.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:18 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:18.550+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-03-04T12:37:49.840349+0000 front 2024-03-04T12:37:49.840381+0000 (oldest deadline 2024-03-04T12:38:13.940267+0000) 2024-03-04T12:38:18.975 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:18 smithi063 podman[73750]: 2024-03-04 12:38:18.64330758 +0000 UTC m=+0.713583938 container remove 2fa9efedd41645e87ddf4ca5e1f0dfcaa0a9736cf5d2aa50c208893c95589da1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:38:19.227 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:18 smithi063 podman[73844]: 2024-03-04 12:38:18.8732937 +0000 UTC m=+0.018117408 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:38:19.227 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:18 smithi063 podman[73844]: 2024-03-04 12:38:18.98562129 +0000 UTC m=+0.130444998 container create 0e2f7b062c9542b413a504a55c56842361482a20f6f4d8e2c952de277a4c306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.build-date=20240222) 2024-03-04T12:38:19.227 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:19 smithi063 podman[73844]: 2024-03-04 12:38:19.221836298 +0000 UTC m=+0.366660018 container init 0e2f7b062c9542b413a504a55c56842361482a20f6f4d8e2c952de277a4c306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-04T12:38:19.527 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:19 smithi063 podman[73844]: 2024-03-04 12:38:19.225328373 +0000 UTC m=+0.370152075 container start 0e2f7b062c9542b413a504a55c56842361482a20f6f4d8e2c952de277a4c306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10-deactivate, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-04T12:38:19.527 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:19 smithi063 podman[73844]: 2024-03-04 12:38:19.305260022 +0000 UTC m=+0.450083743 container attach 0e2f7b062c9542b413a504a55c56842361482a20f6f4d8e2c952de277a4c306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-03-04T12:38:19.813 INFO:journalctl@ceph.osd.10.smithi063.stdout:Mar 04 12:38:19 smithi063 podman[73844]: 2024-03-04 12:38:19.522899492 +0000 UTC m=+0.667723201 container died 0e2f7b062c9542b413a504a55c56842361482a20f6f4d8e2c952de277a4c306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD) 2024-03-04T12:38:19.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:19 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:19.567+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:19.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:19 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:19.567+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:19.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:19 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:19.567+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:19.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:19 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:19.567+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:19.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:19 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:19.567+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:19.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:19 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:19.567+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:19.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:19 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:19.567+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-03-04T12:37:49.840349+0000 front 2024-03-04T12:37:49.840381+0000 (oldest deadline 2024-03-04T12:38:13.940267+0000) 2024-03-04T12:38:20.201 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.10.service' 2024-03-04T12:38:20.245 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:38:20.246 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-04T12:38:20.246 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-04T12:38:20.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.11 2024-03-04T12:38:20.556 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:20 smithi086 systemd[1]: Stopping Ceph osd.11 for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:38:20.815 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:20.549+0000 7fd37ead2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:38:20.815 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:20.549+0000 7fd37ead2700 -1 osd.11 109 *** Got signal Terminated *** 2024-03-04T12:38:20.815 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:20.549+0000 7fd37ead2700 -1 osd.11 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T12:38:20.815 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:20.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:20.816 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:20.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:20.816 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:20.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:20.816 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:20.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:20.816 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:20.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:20.816 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:20.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:20.816 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:20 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:20.561+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-03-04T12:37:49.840349+0000 front 2024-03-04T12:37:49.840381+0000 (oldest deadline 2024-03-04T12:38:13.940267+0000) 2024-03-04T12:38:21.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:21 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:21.534+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:21.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:21 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:21.534+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:21.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:21 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:21.534+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:21.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:21 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:21.534+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:21.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:21 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:21.534+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:21.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:21 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:21.534+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:21.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:21 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:21.534+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-03-04T12:37:49.840349+0000 front 2024-03-04T12:37:49.840381+0000 (oldest deadline 2024-03-04T12:38:13.940267+0000) 2024-03-04T12:38:22.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:22 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:22.524+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:22.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:22 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:22.524+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:22.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:22 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:22.524+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:22.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:22 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:22.524+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:22.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:22 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:22.524+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:22.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:22 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:22.524+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:22.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:22 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:22.524+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-03-04T12:37:49.840349+0000 front 2024-03-04T12:37:49.840381+0000 (oldest deadline 2024-03-04T12:38:13.940267+0000) 2024-03-04T12:38:23.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:23 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:23.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:23.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:23 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:23.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:23.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:23 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:23.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:23.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:23 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:23.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:23.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:23 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:23.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:23.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:23 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:23.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:23.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:23 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:23.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-03-04T12:37:49.840349+0000 front 2024-03-04T12:37:49.840381+0000 (oldest deadline 2024-03-04T12:38:13.940267+0000) 2024-03-04T12:38:23.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:23 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:23.497+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6820 osd.8 since back 2024-03-04T12:37:58.641235+0000 front 2024-03-04T12:37:58.641261+0000 (oldest deadline 2024-03-04T12:38:22.741231+0000) 2024-03-04T12:38:24.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:24 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:24.491+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:24.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:24 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:24.491+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:24.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:24 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:24.491+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:24.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:24 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:24.491+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:24.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:24 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:24.491+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:24.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:24 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:24.491+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:24.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:24 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:24.491+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-03-04T12:37:49.840349+0000 front 2024-03-04T12:37:49.840381+0000 (oldest deadline 2024-03-04T12:38:13.940267+0000) 2024-03-04T12:38:24.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:24 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:24.491+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6820 osd.8 since back 2024-03-04T12:37:58.641235+0000 front 2024-03-04T12:37:58.641261+0000 (oldest deadline 2024-03-04T12:38:22.741231+0000) 2024-03-04T12:38:25.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:25 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:25.458+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6806 osd.0 since back 2024-03-04T12:36:55.031902+0000 front 2024-03-04T12:36:55.031978+0000 (oldest deadline 2024-03-04T12:37:20.832562+0000) 2024-03-04T12:38:25.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:25 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:25.458+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-03-04T12:37:03.733578+0000 front 2024-03-04T12:37:03.733552+0000 (oldest deadline 2024-03-04T12:37:29.033639+0000) 2024-03-04T12:38:25.848 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:25 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:25.458+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6804 osd.2 since back 2024-03-04T12:37:09.034297+0000 front 2024-03-04T12:37:09.034031+0000 (oldest deadline 2024-03-04T12:37:34.234739+0000) 2024-03-04T12:38:25.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:25 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:25.458+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-03-04T12:37:25.337082+0000 front 2024-03-04T12:37:25.337248+0000 (oldest deadline 2024-03-04T12:37:51.236951+0000) 2024-03-04T12:38:25.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:25 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:25.458+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6812 osd.5 since back 2024-03-04T12:37:34.038317+0000 front 2024-03-04T12:37:34.038531+0000 (oldest deadline 2024-03-04T12:37:59.338422+0000) 2024-03-04T12:38:25.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:25 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:25.458+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.60:6822 osd.6 since back 2024-03-04T12:37:42.839468+0000 front 2024-03-04T12:37:42.839528+0000 (oldest deadline 2024-03-04T12:38:05.739368+0000) 2024-03-04T12:38:25.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:25 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:25.458+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-03-04T12:37:49.840349+0000 front 2024-03-04T12:37:49.840381+0000 (oldest deadline 2024-03-04T12:38:13.940267+0000) 2024-03-04T12:38:25.849 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:25 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11[62594]: 2024-03-04T12:38:25.458+0000 7fd3770be700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.86:6820 osd.8 since back 2024-03-04T12:37:58.641235+0000 front 2024-03-04T12:37:58.641261+0000 (oldest deadline 2024-03-04T12:38:22.741231+0000) 2024-03-04T12:38:26.213 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:25 smithi086 podman[74822]: 2024-03-04 12:38:25.916468764 +0000 UTC m=+5.479215360 container died 06a14f562bcabf691a87f0459b24af7913913bfef51447e9e7b783ff0b2c134a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-03-04T12:38:26.214 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:26 smithi086 podman[74822]: 2024-03-04 12:38:26.208311711 +0000 UTC m=+5.771058326 container cleanup 06a14f562bcabf691a87f0459b24af7913913bfef51447e9e7b783ff0b2c134a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11, GIT_CLEAN=True, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-04T12:38:26.598 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:26 smithi086 bash[74822]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11 2024-03-04T12:38:26.598 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:26 smithi086 podman[74852]: 2024-03-04 12:38:26.425880287 +0000 UTC m=+0.509950691 container remove 06a14f562bcabf691a87f0459b24af7913913bfef51447e9e7b783ff0b2c134a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, ceph=True) 2024-03-04T12:38:27.068 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:26 smithi086 podman[74944]: 2024-03-04 12:38:26.743564735 +0000 UTC m=+0.108851321 container create adca57a66503b49dbc7974edda71523ca703e8804af783473d63d63bf047da0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11-deactivate, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:38:27.068 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:26 smithi086 podman[74944]: 2024-03-04 12:38:26.653148419 +0000 UTC m=+0.018435023 image pull d71b839abdf892550525dad59fdf20a6b925b73dbcc7526f0d04da8380b9fabf quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377 2024-03-04T12:38:27.068 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:26 smithi086 podman[74944]: 2024-03-04 12:38:26.98758211 +0000 UTC m=+0.352868701 container init adca57a66503b49dbc7974edda71523ca703e8804af783473d63d63bf047da0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c) 2024-03-04T12:38:27.068 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:26 smithi086 podman[74944]: 2024-03-04 12:38:26.990547787 +0000 UTC m=+0.355834375 container start adca57a66503b49dbc7974edda71523ca703e8804af783473d63d63bf047da0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-03-04T12:38:27.348 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:27 smithi086 podman[74944]: 2024-03-04 12:38:27.062539761 +0000 UTC m=+0.427826354 container attach adca57a66503b49dbc7974edda71523ca703e8804af783473d63d63bf047da0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11-deactivate, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222) 2024-03-04T12:38:27.349 INFO:journalctl@ceph.osd.11.smithi086.stdout:Mar 04 12:38:27 smithi086 podman[74944]: 2024-03-04 12:38:27.274502526 +0000 UTC m=+0.639789121 container died adca57a66503b49dbc7974edda71523ca703e8804af783473d63d63bf047da0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-osd-11-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-04T12:38:27.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@osd.11.service' 2024-03-04T12:38:27.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:38:27.805 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-04T12:38:27.805 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-04T12:38:27.806 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.a 2024-03-04T12:38:28.097 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:38:27 smithi060 systemd[1]: Stopping Ceph mds.a for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:38:28.427 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:38:28 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a[75233]: 2024-03-04T12:38:28.091+0000 7ff248a57700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:38:28.428 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:38:28 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a[75233]: 2024-03-04T12:38:28.091+0000 7ff248a57700 -1 mds.a *** got signal Terminated *** 2024-03-04T12:38:31.679 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:38:31 smithi060 podman[104652]: 2024-03-04 12:38:31.395860692 +0000 UTC m=+3.401568757 container died 5b43f6f536a382f17cc9d07216325579b2c6d63e06057afda41b7c1f7d798996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5) 2024-03-04T12:38:32.046 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:38:31 smithi060 podman[104652]: 2024-03-04 12:38:31.832437727 +0000 UTC m=+3.838145794 container cleanup 5b43f6f536a382f17cc9d07216325579b2c6d63e06057afda41b7c1f7d798996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-04T12:38:32.047 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:38:31 smithi060 bash[104652]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a 2024-03-04T12:38:32.272 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.a.service' 2024-03-04T12:38:32.297 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:38:32 smithi060 podman[104666]: 2024-03-04 12:38:32.04187372 +0000 UTC m=+0.649688785 container remove 5b43f6f536a382f17cc9d07216325579b2c6d63e06057afda41b7c1f7d798996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:38:32.297 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:38:32 smithi060 systemd[1]: ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.a.service: Deactivated successfully. 2024-03-04T12:38:32.298 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:38:32 smithi060 systemd[1]: Stopped Ceph mds.a for a1c49686-da20-11ee-95c6-87774f69a715. 2024-03-04T12:38:32.298 INFO:journalctl@ceph.mds.a.smithi060.stdout:Mar 04 12:38:32 smithi060 systemd[1]: ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.a.service: Consumed 2.227s CPU time. 2024-03-04T12:38:32.308 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:38:32.309 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-04T12:38:32.309 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-04T12:38:32.309 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.d 2024-03-04T12:38:32.616 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:38:32 smithi060 systemd[1]: Stopping Ceph mds.d for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:38:32.928 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:38:32 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-d[74281]: 2024-03-04T12:38:32.610+0000 7f46d06f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:38:32.928 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:38:32 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-d[74281]: 2024-03-04T12:38:32.610+0000 7f46d06f5700 -1 mds.d *** got signal Terminated *** 2024-03-04T12:38:35.826 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:38:35 smithi060 podman[104773]: 2024-03-04 12:38:35.441340053 +0000 UTC m=+2.918183583 container died 06f28728ec761d925e616cd61bf4e7edb9fc60de13eaf6b753f33df1edc55575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-d, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-04T12:38:36.178 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:38:35 smithi060 podman[104773]: 2024-03-04 12:38:35.894722651 +0000 UTC m=+3.371566192 container cleanup 06f28728ec761d925e616cd61bf4e7edb9fc60de13eaf6b753f33df1edc55575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-04T12:38:36.178 INFO:journalctl@ceph.mds.d.smithi060.stdout:Mar 04 12:38:35 smithi060 bash[104773]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-d 2024-03-04T12:38:36.330 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.d.service' 2024-03-04T12:38:36.366 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:38:36.366 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-04T12:38:36.367 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-04T12:38:36.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.g 2024-03-04T12:38:36.729 INFO:journalctl@ceph.mds.g.smithi060.stdout:Mar 04 12:38:36 smithi060 systemd[1]: Stopping Ceph mds.g for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:38:37.178 INFO:journalctl@ceph.mds.g.smithi060.stdout:Mar 04 12:38:36 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-g[74874]: 2024-03-04T12:38:36.723+0000 7fba4bcf0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:38:37.178 INFO:journalctl@ceph.mds.g.smithi060.stdout:Mar 04 12:38:36 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-g[74874]: 2024-03-04T12:38:36.723+0000 7fba4bcf0700 -1 mds.g *** got signal Terminated *** 2024-03-04T12:38:39.662 INFO:journalctl@ceph.mds.g.smithi060.stdout:Mar 04 12:38:39 smithi060 podman[104892]: 2024-03-04 12:38:39.394004081 +0000 UTC m=+2.804877019 container died fb7bf3da2f1e3e9a2ba5af41c0fb900ea8f964e19109b548421ed1c78727e558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:38:39.927 INFO:journalctl@ceph.mds.g.smithi060.stdout:Mar 04 12:38:39 smithi060 podman[104892]: 2024-03-04 12:38:39.748527664 +0000 UTC m=+3.159400610 container cleanup fb7bf3da2f1e3e9a2ba5af41c0fb900ea8f964e19109b548421ed1c78727e558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-g, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-04T12:38:39.928 INFO:journalctl@ceph.mds.g.smithi060.stdout:Mar 04 12:38:39 smithi060 bash[104892]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-g 2024-03-04T12:38:40.183 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.g.service' 2024-03-04T12:38:40.216 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:38:40.217 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-04T12:38:40.217 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-04T12:38:40.217 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.j 2024-03-04T12:38:40.680 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:38:40 smithi060 systemd[1]: Stopping Ceph mds.j for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:38:40.680 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:38:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-j[73921]: 2024-03-04T12:38:40.508+0000 7f3a506e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:38:40.680 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:38:40 smithi060 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-j[73921]: 2024-03-04T12:38:40.508+0000 7f3a506e7700 -1 mds.j *** got signal Terminated *** 2024-03-04T12:38:44.815 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:38:44 smithi060 podman[105014]: 2024-03-04 12:38:44.520727512 +0000 UTC m=+4.093583574 container died 1364bf2fcc86c1c266fd2cbedd871185afb7dd66e2d7fedef659b017029734af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-j, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-03-04T12:38:44.815 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:38:44 smithi060 podman[105014]: 2024-03-04 12:38:44.810434896 +0000 UTC m=+4.383290970 container cleanup 1364bf2fcc86c1c266fd2cbedd871185afb7dd66e2d7fedef659b017029734af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-j, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-03-04T12:38:45.158 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:38:44 smithi060 bash[105014]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-j 2024-03-04T12:38:45.159 INFO:journalctl@ceph.mds.j.smithi060.stdout:Mar 04 12:38:45 smithi060 podman[105028]: 2024-03-04 12:38:45.044849735 +0000 UTC m=+0.518588517 container remove 1364bf2fcc86c1c266fd2cbedd871185afb7dd66e2d7fedef659b017029734af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-j, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-04T12:38:45.177 DEBUG:teuthology.orchestra.run.smithi060:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.j.service' 2024-03-04T12:38:45.210 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:38:45.210 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-04T12:38:45.211 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-04T12:38:45.211 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.b 2024-03-04T12:38:45.532 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:38:45 smithi063 systemd[1]: Stopping Ceph mds.b for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:38:45.803 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:38:45 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b[67411]: 2024-03-04T12:38:45.526+0000 7f7fe9d83700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:38:45.803 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:38:45 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b[67411]: 2024-03-04T12:38:45.526+0000 7f7fe9d83700 -1 mds.b *** got signal Terminated *** 2024-03-04T12:38:47.707 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:38:47 smithi063 podman[73950]: 2024-03-04 12:38:47.409614304 +0000 UTC m=+1.999384541 container died e613f71e49fc41ffb30cd24caca856c9c99ba4317b9c4e653a0594960cf29a76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-04T12:38:48.101 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:38:47 smithi063 podman[73950]: 2024-03-04 12:38:47.792945968 +0000 UTC m=+2.382716174 container cleanup e613f71e49fc41ffb30cd24caca856c9c99ba4317b9c4e653a0594960cf29a76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-04T12:38:48.102 INFO:journalctl@ceph.mds.b.smithi063.stdout:Mar 04 12:38:47 smithi063 bash[73950]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-b 2024-03-04T12:38:48.380 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.b.service' 2024-03-04T12:38:48.422 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:38:48.422 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-04T12:38:48.422 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-04T12:38:48.423 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.e 2024-03-04T12:38:48.769 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:38:48 smithi063 systemd[1]: Stopping Ceph mds.e for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:38:49.101 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:38:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e[66792]: 2024-03-04T12:38:48.763+0000 7fd0b725d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:38:49.101 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:38:48 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e[66792]: 2024-03-04T12:38:48.763+0000 7fd0b725d700 -1 mds.e *** got signal Terminated *** 2024-03-04T12:38:51.743 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:38:51 smithi063 podman[74084]: 2024-03-04 12:38:51.411552677 +0000 UTC m=+2.768562844 container died 9856d0e13ea092a944c00453def0834b7b998bd25d7afce79118de00be69163c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True) 2024-03-04T12:38:52.101 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:38:51 smithi063 podman[74084]: 2024-03-04 12:38:51.738642286 +0000 UTC m=+3.095652457 container cleanup 9856d0e13ea092a944c00453def0834b7b998bd25d7afce79118de00be69163c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T12:38:52.101 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:38:51 smithi063 bash[74084]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e 2024-03-04T12:38:52.101 INFO:journalctl@ceph.mds.e.smithi063.stdout:Mar 04 12:38:51 smithi063 podman[74098]: 2024-03-04 12:38:51.973194997 +0000 UTC m=+0.555202942 container remove 9856d0e13ea092a944c00453def0834b7b998bd25d7afce79118de00be69163c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-e, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-04T12:38:52.224 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.e.service' 2024-03-04T12:38:52.264 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:38:52.264 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-04T12:38:52.264 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-04T12:38:52.265 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.h 2024-03-04T12:38:52.601 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:38:52 smithi063 systemd[1]: Stopping Ceph mds.h for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:38:53.101 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:38:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:38:52.600+0000 7f71c502d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:38:53.101 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:38:52 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h[67099]: 2024-03-04T12:38:52.600+0000 7f71c502d700 -1 mds.h *** got signal Terminated *** 2024-03-04T12:38:55.782 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:38:55 smithi063 podman[74206]: 2024-03-04 12:38:55.395206458 +0000 UTC m=+2.906891550 container died 6f810dfaa3daab6877bd52272a98aa0fc0f5dea907ed555c455ae0dd4851f0e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T12:38:56.101 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:38:55 smithi063 podman[74206]: 2024-03-04 12:38:55.86739731 +0000 UTC m=+3.379082392 container cleanup 6f810dfaa3daab6877bd52272a98aa0fc0f5dea907ed555c455ae0dd4851f0e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-04T12:38:56.101 INFO:journalctl@ceph.mds.h.smithi063.stdout:Mar 04 12:38:55 smithi063 bash[74206]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-h 2024-03-04T12:38:56.306 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.h.service' 2024-03-04T12:38:56.345 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:38:56.345 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-04T12:38:56.345 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-04T12:38:56.346 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.k 2024-03-04T12:38:56.693 INFO:journalctl@ceph.mds.k.smithi063.stdout:Mar 04 12:38:56 smithi063 systemd[1]: Stopping Ceph mds.k for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:38:57.101 INFO:journalctl@ceph.mds.k.smithi063.stdout:Mar 04 12:38:56 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-k[66470]: 2024-03-04T12:38:56.686+0000 7f384f9c4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:38:57.101 INFO:journalctl@ceph.mds.k.smithi063.stdout:Mar 04 12:38:56 smithi063 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-k[66470]: 2024-03-04T12:38:56.686+0000 7f384f9c4700 -1 mds.k *** got signal Terminated *** 2024-03-04T12:38:59.676 INFO:journalctl@ceph.mds.k.smithi063.stdout:Mar 04 12:38:59 smithi063 podman[74325]: 2024-03-04 12:38:59.396382791 +0000 UTC m=+2.824779615 container died ee27880ef103cc2cdd9686b64e24b0d83eff79a798d5b281cb15f807722b2cbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-k, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:39:00.043 INFO:journalctl@ceph.mds.k.smithi063.stdout:Mar 04 12:38:59 smithi063 podman[74325]: 2024-03-04 12:38:59.771000165 +0000 UTC m=+3.199396989 container cleanup ee27880ef103cc2cdd9686b64e24b0d83eff79a798d5b281cb15f807722b2cbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-k, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-03-04T12:39:00.044 INFO:journalctl@ceph.mds.k.smithi063.stdout:Mar 04 12:38:59 smithi063 bash[74325]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-k 2024-03-04T12:39:00.207 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.k.service' 2024-03-04T12:39:00.245 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:39:00.245 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-04T12:39:00.245 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-04T12:39:00.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.c 2024-03-04T12:39:00.529 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:39:00 smithi086 systemd[1]: Stopping Ceph mds.c for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:39:00.788 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:39:00 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[71488]: 2024-03-04T12:39:00.523+0000 7fb075cf2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:39:00.789 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:39:00 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c[71488]: 2024-03-04T12:39:00.523+0000 7fb075cf2700 -1 mds.c *** got signal Terminated *** 2024-03-04T12:39:03.688 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:39:03 smithi086 podman[75049]: 2024-03-04 12:39:03.391326587 +0000 UTC m=+2.961818974 container died 5bb45a38feef9804a6b74f7cb5a8bf4421818c7d63e8291be68a3129c32eb859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T12:39:04.013 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:39:03 smithi086 podman[75049]: 2024-03-04 12:39:03.774126517 +0000 UTC m=+3.344618924 container cleanup 5bb45a38feef9804a6b74f7cb5a8bf4421818c7d63e8291be68a3129c32eb859 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD) 2024-03-04T12:39:04.014 INFO:journalctl@ceph.mds.c.smithi086.stdout:Mar 04 12:39:03 smithi086 bash[75049]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-c 2024-03-04T12:39:04.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.c.service' 2024-03-04T12:39:04.157 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:39:04.158 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-04T12:39:04.158 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-04T12:39:04.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.f 2024-03-04T12:39:04.348 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:39:04 smithi086 systemd[1]: Stopping Ceph mds.f for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:39:04.848 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:39:04 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-f[67540]: 2024-03-04T12:39:04.460+0000 7f9dbfcd2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:39:04.848 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:39:04 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-f[67540]: 2024-03-04T12:39:04.460+0000 7f9dbfcd2700 -1 mds.f *** got signal Terminated *** 2024-03-04T12:39:07.666 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:39:07 smithi086 podman[75182]: 2024-03-04 12:39:07.400475344 +0000 UTC m=+3.036448907 container died 089cf647a6e0676c0c25bc7ccc1f9c6ff059ab88c4ec204ea4829ef42ffca9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux ) 2024-03-04T12:39:08.008 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:39:07 smithi086 podman[75182]: 2024-03-04 12:39:07.752244223 +0000 UTC m=+3.388217786 container cleanup 089cf647a6e0676c0c25bc7ccc1f9c6ff059ab88c4ec204ea4829ef42ffca9ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-f, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-04T12:39:08.008 INFO:journalctl@ceph.mds.f.smithi086.stdout:Mar 04 12:39:07 smithi086 bash[75182]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-f 2024-03-04T12:39:08.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.f.service' 2024-03-04T12:39:08.209 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:39:08.209 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-04T12:39:08.210 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-04T12:39:08.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.i 2024-03-04T12:39:08.598 INFO:journalctl@ceph.mds.i.smithi086.stdout:Mar 04 12:39:08 smithi086 systemd[1]: Stopping Ceph mds.i for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:39:08.598 INFO:journalctl@ceph.mds.i.smithi086.stdout:Mar 04 12:39:08 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-i[68150]: 2024-03-04T12:39:08.513+0000 7fa1b82eb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:39:08.599 INFO:journalctl@ceph.mds.i.smithi086.stdout:Mar 04 12:39:08 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-i[68150]: 2024-03-04T12:39:08.513+0000 7fa1b82eb700 -1 mds.i *** got signal Terminated *** 2024-03-04T12:39:11.598 INFO:journalctl@ceph.mds.i.smithi086.stdout:Mar 04 12:39:11 smithi086 podman[75302]: 2024-03-04 12:39:11.393680461 +0000 UTC m=+2.978963648 container died 2f83800fdca46c76a89166f2dd76c72f54c2f515d4d179c8a3dfad3a88cfd4e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-i, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-04T12:39:12.012 INFO:journalctl@ceph.mds.i.smithi086.stdout:Mar 04 12:39:11 smithi086 podman[75302]: 2024-03-04 12:39:11.789083026 +0000 UTC m=+3.374366197 container cleanup 2f83800fdca46c76a89166f2dd76c72f54c2f515d4d179c8a3dfad3a88cfd4e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-i, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-04T12:39:12.012 INFO:journalctl@ceph.mds.i.smithi086.stdout:Mar 04 12:39:11 smithi086 bash[75302]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-i 2024-03-04T12:39:12.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.i.service' 2024-03-04T12:39:12.182 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:39:12.183 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-04T12:39:12.183 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-04T12:39:12.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.l 2024-03-04T12:39:12.348 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:39:12 smithi086 systemd[1]: Stopping Ceph mds.l for a1c49686-da20-11ee-95c6-87774f69a715... 2024-03-04T12:39:12.848 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:39:12 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-l[67216]: 2024-03-04T12:39:12.492+0000 7f827317c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T12:39:12.848 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:39:12 smithi086 ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-l[67216]: 2024-03-04T12:39:12.492+0000 7f827317c700 -1 mds.l *** got signal Terminated *** 2024-03-04T12:39:15.690 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:39:15 smithi086 podman[75423]: 2024-03-04 12:39:15.401614154 +0000 UTC m=+3.012264367 container died 20368d8c0c29ca80f03bf1065d2fbb7864a9b627b3c0c2ddc807b3c42eadd0cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-l, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, ceph=True) 2024-03-04T12:39:16.023 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:39:15 smithi086 podman[75423]: 2024-03-04 12:39:15.750539943 +0000 UTC m=+3.361190153 container cleanup 20368d8c0c29ca80f03bf1065d2fbb7864a9b627b3c0c2ddc807b3c42eadd0cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d881e7a4d830e0159c9b14ceabcad19438c4299e6d54a0d2518f6aa597d0b377, name=ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240304.042522-717ce3c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-04T12:39:16.023 INFO:journalctl@ceph.mds.l.smithi086.stdout:Mar 04 12:39:15 smithi086 bash[75423]: ceph-a1c49686-da20-11ee-95c6-87774f69a715-mds-l 2024-03-04T12:39:16.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a1c49686-da20-11ee-95c6-87774f69a715@mds.l.service' 2024-03-04T12:39:16.202 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T12:39:16.202 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-04T12:39:16.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1c49686-da20-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T12:39:16.400 INFO:teuthology.orchestra.run.smithi060.stdout:Deleting cluster with fsid: a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:39:18.833 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1c49686-da20-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T12:39:19.035 INFO:teuthology.orchestra.run.smithi063.stdout:Deleting cluster with fsid: a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:39:21.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1c49686-da20-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T12:39:21.919 INFO:teuthology.orchestra.run.smithi086.stdout:Deleting cluster with fsid: a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:39:24.327 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T12:39:24.362 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T12:39:24.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T12:39:24.431 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-04T12:39:24.433 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054/remote/smithi060/crash 2024-03-04T12:39:24.434 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/crash -- . 2024-03-04T12:39:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:tar: /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T12:39:24.501 INFO:teuthology.orchestra.run.smithi060.stderr:tar: Error is not recoverable: exiting now 2024-03-04T12:39:24.503 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054/remote/smithi063/crash 2024-03-04T12:39:24.504 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/crash -- . 2024-03-04T12:39:24.558 INFO:teuthology.orchestra.run.smithi063.stderr:tar: /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T12:39:24.558 INFO:teuthology.orchestra.run.smithi063.stderr:tar: Error is not recoverable: exiting now 2024-03-04T12:39:24.560 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054/remote/smithi086/crash 2024-03-04T12:39:24.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/crash -- . 2024-03-04T12:39:24.627 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /var/lib/ceph/a1c49686-da20-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T12:39:24.628 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2024-03-04T12:39:24.629 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-04T12:39:24.629 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-04T12:39:24.668 INFO:tasks.cephadm:Compressing logs... 2024-03-04T12:39:24.668 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T12:39:24.711 DEBUG:teuthology.orchestra.run.smithi063:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T12:39:24.713 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T12:39:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T12:39:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:: No such file or directory 2024-03-04T12:39:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.6.log 2024-03-04T12:39:24.739 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mon.a.log 2024-03-04T12:39:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.6.log: -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T12:39:24.740 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.g.log 2024-03-04T12:39:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.j.log 2024-03-04T12:39:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.audit.log: /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.3.log 2024-03-04T12:39:24.745 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mgr.x.log 2024-03-04T12:39:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.0.log 2024-03-04T12:39:24.746 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mgr.x.log: 90.1% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T12:39:24.746 INFO:teuthology.orchestra.run.smithi060.stderr: 94.6% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.j.log.gz 2024-03-04T12:39:24.747 INFO:teuthology.orchestra.run.smithi060.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T12:39:24.755 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T12:39:24.755 INFO:teuthology.orchestra.run.smithi086.stderr:: No such file or directory 2024-03-04T12:39:24.755 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.l.log 2024-03-04T12:39:24.755 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T12:39:24.755 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.l.log: --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.i.log 2024-03-04T12:39:24.755 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.11.log 2024-03-04T12:39:24.756 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mgr.z.log 2024-03-04T12:39:24.756 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.i.log: /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.8.log 2024-03-04T12:39:24.756 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mgr.z.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T12:39:24.756 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.5.log 2024-03-04T12:39:24.756 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.f.log 2024-03-04T12:39:24.756 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T12:39:24.756 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.f.log: 90.2% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T12:39:24.756 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.log 2024-03-04T12:39:24.757 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T12:39:24.757 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.2.log 2024-03-04T12:39:24.757 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.log: 88.8% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T12:39:24.757 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.c.log 2024-03-04T12:39:24.757 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.2.log: 91.9% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mgr.z.log.gz 2024-03-04T12:39:24.757 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T12:39:24.757 INFO:teuthology.orchestra.run.smithi063.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-04T12:39:24.757 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T12:39:24.758 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.10.log 2024-03-04T12:39:24.758 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose/var/log/ceph/cephadm.log: -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T12:39:24.758 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.10.log: --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.b.log 2024-03-04T12:39:24.758 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.4.log 2024-03-04T12:39:24.758 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.1.log 2024-03-04T12:39:24.758 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.b.log: 92.1%/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.4.log: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T12:39:24.758 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.k.log 2024-03-04T12:39:24.758 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.h.log 2024-03-04T12:39:24.758 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.7.log 2024-03-04T12:39:24.759 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.e.log 2024-03-04T12:39:24.759 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mgr.y.log 2024-03-04T12:39:24.759 INFO:teuthology.orchestra.run.smithi063.stderr: 90.2% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T12:39:24.759 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T12:39:24.762 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T12:39:24.762 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.log 2024-03-04T12:39:24.762 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T12:39:24.762 INFO:teuthology.orchestra.run.smithi060.stderr: 84.1% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T12:39:24.763 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.log: 88.8% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T12:39:24.763 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.9.log 2024-03-04T12:39:24.763 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.d.log 2024-03-04T12:39:24.765 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mgr.y.log: 91.8% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mgr.y.log.gz 2024-03-04T12:39:24.765 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.log 2024-03-04T12:39:24.765 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.cephadm.log: /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.7.log: 83.7% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T12:39:24.765 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T12:39:24.767 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mon.c.log 2024-03-04T12:39:24.768 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.log: 88.9% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T12:39:24.768 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mon.b.log 2024-03-04T12:39:24.772 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.a.log 2024-03-04T12:39:24.775 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.d.log: 91.0% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T12:39:24.777 INFO:teuthology.orchestra.run.smithi060.stderr: 94.6% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.d.log.gz 2024-03-04T12:39:24.778 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-04T12:39:24.783 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-volume.log: /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T12:39:24.790 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-04T12:39:24.794 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-volume.log: /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T12:39:24.801 INFO:teuthology.orchestra.run.smithi060.stderr: 94.5% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.g.log.gz 2024-03-04T12:39:24.806 INFO:teuthology.orchestra.run.smithi086.stderr: 93.8% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.l.log.gz 2024-03-04T12:39:24.832 INFO:teuthology.orchestra.run.smithi060.stderr: 94.5% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.a.log.gz 2024-03-04T12:39:24.878 INFO:teuthology.orchestra.run.smithi063.stderr: 93.7% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.k.log.gz 2024-03-04T12:39:25.011 INFO:teuthology.orchestra.run.smithi086.stderr: 94.6% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.c.log.gz 2024-03-04T12:39:25.088 INFO:teuthology.orchestra.run.smithi060.stderr: 89.9% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mgr.x.log.gz 2024-03-04T12:39:25.242 INFO:teuthology.orchestra.run.smithi086.stderr: 92.0% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mon.c.log.gz 2024-03-04T12:39:25.244 INFO:teuthology.orchestra.run.smithi063.stderr: 92.1% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mon.b.log.gz 2024-03-04T12:39:25.478 INFO:teuthology.orchestra.run.smithi063.stderr: 95.3% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.e.log.gz 2024-03-04T12:39:25.634 INFO:teuthology.orchestra.run.smithi086.stderr: 95.3% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.f.log.gz 2024-03-04T12:39:25.707 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mon.a.log.gz 2024-03-04T12:39:26.842 INFO:teuthology.orchestra.run.smithi063.stderr: 93.4% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.b.log.gz 2024-03-04T12:39:27.087 INFO:teuthology.orchestra.run.smithi063.stderr: 92.1% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.10.log.gz 2024-03-04T12:39:27.200 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.6.log.gz 2024-03-04T12:39:27.355 INFO:teuthology.orchestra.run.smithi060.stderr: 92.0% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.3.log.gz 2024-03-04T12:39:27.433 INFO:teuthology.orchestra.run.smithi063.stderr: 92.5% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.1.log.gz 2024-03-04T12:39:27.713 INFO:teuthology.orchestra.run.smithi060.stderr: 92.5% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.0.log.gz 2024-03-04T12:39:27.781 INFO:teuthology.orchestra.run.smithi086.stderr: 92.2% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.2.log.gz 2024-03-04T12:39:27.836 INFO:teuthology.orchestra.run.smithi063.stderr: 91.9% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.7.log.gz 2024-03-04T12:39:28.069 INFO:teuthology.orchestra.run.smithi086.stderr: 92.6% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.11.log.gz 2024-03-04T12:39:28.221 INFO:teuthology.orchestra.run.smithi086.stderr: 92.4% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.8.log.gz 2024-03-04T12:39:28.339 INFO:teuthology.orchestra.run.smithi086.stderr: 92.2% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.5.log.gz 2024-03-04T12:39:28.487 INFO:teuthology.orchestra.run.smithi060.stderr: 92.1% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.9.log.gz 2024-03-04T12:39:28.489 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-04T12:39:28.489 INFO:teuthology.orchestra.run.smithi060.stderr:real 0m3.764s 2024-03-04T12:39:28.489 INFO:teuthology.orchestra.run.smithi060.stderr:user 0m12.840s 2024-03-04T12:39:28.489 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m0.398s 2024-03-04T12:39:28.879 INFO:teuthology.orchestra.run.smithi063.stderr: 92.2% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-osd.4.log.gz 2024-03-04T12:39:30.354 INFO:teuthology.orchestra.run.smithi063.stderr: 93.3% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.h.log.gz 2024-03-04T12:39:30.355 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-03-04T12:39:30.356 INFO:teuthology.orchestra.run.smithi063.stderr:real 0m5.627s 2024-03-04T12:39:30.356 INFO:teuthology.orchestra.run.smithi063.stderr:user 0m20.457s 2024-03-04T12:39:30.356 INFO:teuthology.orchestra.run.smithi063.stderr:sys 0m0.595s 2024-03-04T12:39:32.996 INFO:teuthology.orchestra.run.smithi086.stderr: 93.4% -- replaced with /var/log/ceph/a1c49686-da20-11ee-95c6-87774f69a715/ceph-mds.i.log.gz 2024-03-04T12:39:32.998 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-04T12:39:32.998 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m8.271s 2024-03-04T12:39:32.998 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m22.655s 2024-03-04T12:39:32.998 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m0.695s 2024-03-04T12:39:32.999 INFO:tasks.cephadm:Archiving logs... 2024-03-04T12:39:32.999 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054/remote/smithi060/log 2024-03-04T12:39:33.000 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T12:39:34.940 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054/remote/smithi063/log 2024-03-04T12:39:34.941 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T12:39:37.539 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054/remote/smithi086/log 2024-03-04T12:39:37.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T12:39:40.804 INFO:tasks.cephadm:Removing cluster... 2024-03-04T12:39:40.805 DEBUG:teuthology.orchestra.run.smithi060:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1c49686-da20-11ee-95c6-87774f69a715 --force 2024-03-04T12:39:41.002 INFO:teuthology.orchestra.run.smithi060.stdout:Deleting cluster with fsid: a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:39:41.267 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1c49686-da20-11ee-95c6-87774f69a715 --force 2024-03-04T12:39:41.471 INFO:teuthology.orchestra.run.smithi063.stdout:Deleting cluster with fsid: a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:39:41.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a1c49686-da20-11ee-95c6-87774f69a715 --force 2024-03-04T12:39:41.949 INFO:teuthology.orchestra.run.smithi086.stdout:Deleting cluster with fsid: a1c49686-da20-11ee-95c6-87774f69a715 2024-03-04T12:39:42.222 INFO:tasks.cephadm:Removing cephadm ... 2024-03-04T12:39:42.222 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T12:39:42.240 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T12:39:42.256 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T12:39:42.272 INFO:tasks.cephadm:Teardown complete 2024-03-04T12:39:42.272 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-04T12:39:42.285 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-04T12:39:42.285 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T12:39:42.288 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T12:39:42.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T12:39:42.418 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-04T12:39:42.429 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-04T12:39:42.429 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T12:39:42.432 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T12:39:42.434 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T12:39:42.446 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-03-04T12:39:42.447 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-03-04T12:39:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-03-04T12:39:42.451 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T12:39:42.451 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-03-04T12:39:42.451 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 6 +5995us[+5995us] +/- 121ms 2024-03-04T12:39:42.451 INFO:teuthology.orchestra.run.smithi063.stdout:^* hv02.front.sepia.ceph.com 3 6 377 64 -3904us[-3844us] +/- 65ms 2024-03-04T12:39:42.452 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +964us[ +964us] +/- 92ms 2024-03-04T12:39:42.452 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:39:42.453 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T12:39:42.453 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-03-04T12:39:42.454 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 0 +6026us[+6026us] +/- 121ms 2024-03-04T12:39:42.454 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -3948us[-3747us] +/- 66ms 2024-03-04T12:39:42.454 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +996us[+1197us] +/- 92ms 2024-03-04T12:39:42.454 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:39:42.454 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T12:39:42.454 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-03-04T12:39:42.454 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 +6137us[+6137us] +/- 121ms 2024-03-04T12:39:42.454 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -3825us[-3498us] +/- 66ms 2024-03-04T12:39:42.455 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +1116us[+1443us] +/- 92ms 2024-03-04T12:39:42.455 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T12:39:42.455 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-04T12:39:42.466 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-04T12:39:42.466 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-04T12:39:42.496 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T12:39:42.585 DEBUG:teuthology.orchestra.run.smithi063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T12:39:42.667 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T12:39:42.737 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-04T12:39:42.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:39:42.770 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-04T12:39:42.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:39:42.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-04T12:39:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:39:42.835 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-04T12:39:42.847 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-04T12:39:42.908 INFO:teuthology.task.internal:Duration was 1992.529825 seconds 2024-03-04T12:39:42.908 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-04T12:39:42.933 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-04T12:39:42.933 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T12:39:42.935 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T12:39:42.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T12:39:42.980 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T12:39:42.981 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T12:39:42.988 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T12:39:43.383 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-04T12:39:43.383 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-03-04T12:39:43.383 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T12:39:43.405 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-03-04T12:39:43.405 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T12:39:43.434 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-03-04T12:39:43.435 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T12:39:43.464 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-04T12:39:43.464 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T12:39:43.467 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T12:39:43.477 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T12:39:43.532 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-04T12:39:43.532 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T12:39:43.537 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T12:39:43.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T12:39:43.946 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-04T12:39:43.957 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-04T12:39:43.958 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T12:39:43.984 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T12:39:44.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T12:39:44.043 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-04T12:39:44.055 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T12:39:44.057 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T12:39:44.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T12:39:44.080 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-03-04T12:39:44.086 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-03-04T12:39:44.108 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-03-04T12:39:44.146 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T12:39:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:39:44.160 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T12:39:44.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:39:44.175 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T12:39:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T12:39:44.200 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-04T12:39:44.212 INFO:teuthology.task.internal:Transferring archived files... 2024-03-04T12:39:44.212 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054/remote/smithi060 2024-03-04T12:39:44.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T12:39:44.256 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054/remote/smithi063 2024-03-04T12:39:44.256 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T12:39:44.298 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054/remote/smithi086 2024-03-04T12:39:44.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T12:39:44.341 INFO:teuthology.task.internal:Removing archive directory... 2024-03-04T12:39:44.342 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T12:39:44.344 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T12:39:44.347 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T12:39:44.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-04T12:39:44.398 INFO:teuthology.task.internal:Not uploading archives. 2024-03-04T12:39:44.399 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-04T12:39:44.420 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-04T12:39:44.420 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T12:39:44.422 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T12:39:44.425 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T12:39:44.434 INFO:teuthology.orchestra.run.smithi060.stdout: 266038 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 12:39 /home/ubuntu/cephtest 2024-03-04T12:39:44.437 INFO:teuthology.orchestra.run.smithi063.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 12:39 /home/ubuntu/cephtest 2024-03-04T12:39:44.440 INFO:teuthology.orchestra.run.smithi086.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 12:39 /home/ubuntu/cephtest 2024-03-04T12:39:44.442 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-04T12:39:44.472 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-04T12:39:44.512 INFO:teuthology.nuke:Checking targets against current locks 2024-03-04T12:39:44.543 DEBUG:teuthology.nuke:shortname: smithi060 2024-03-04T12:39:44.543 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T12:39:44.567 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 11:57:57.402365', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH2C7eOZ0wrUqj5MViV6ezJadMYDbaK5FmjU4W+QOUL367quJYKBuHlEaC5U6HQZJ+fZ/Gj0GUd4exvLCC9qG7E='} 2024-03-04T12:39:44.572 DEBUG:teuthology.nuke:shortname: smithi063 2024-03-04T12:39:44.572 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T12:39:44.598 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 11:57:57.398791', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBATIsvh1BLe9YhwLFtXJ2Sv7ZNO0g8aFNI679y+Jdt0fEx0C/mOgcroWE3IVVkjd42gfla3/dEWnOFCu0RX/bhM='} 2024-03-04T12:39:44.599 DEBUG:teuthology.nuke:shortname: smithi086 2024-03-04T12:39:44.599 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T12:39:44.631 INFO:teuthology.orchestra.console.smithi060:Power off 2024-03-04T12:39:44.632 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T12:39:44.641 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-03-04_08:26:39-fs-wip-vshankar-testing-20240304.042522-testing-default-smithi/7581054', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 11:57:57.400798', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMYWj+J7HylnDPtk4An/coA6jD0uZFBqa9szme/tpHgu4LbUo5z/0AdWU8KJ3J+4bDaZWZCT4b+egmwW6qApGt4='} 2024-03-04T12:39:44.661 DEBUG:teuthology.orchestra.console.smithi060:power off output: Chassis Power Control: Down/Off 2024-03-04T12:39:44.661 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T12:39:44.680 INFO:teuthology.orchestra.console.smithi063:Power off 2024-03-04T12:39:44.681 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T12:39:44.696 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-03-04T12:39:44.708 DEBUG:teuthology.orchestra.console.smithi063:power off output: Chassis Power Control: Down/Off 2024-03-04T12:39:44.708 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T12:39:44.724 INFO:teuthology.orchestra.console.smithi086:Power off 2024-03-04T12:39:44.725 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T12:39:44.736 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-04T12:39:44.751 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2024-03-04T12:39:44.751 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T12:39:44.776 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-04T12:39:48.699 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T12:39:48.738 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T12:39:48.777 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T12:39:48.812 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-03-04T12:39:48.849 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is on 2024-03-04T12:39:48.888 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-04T12:39:52.816 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T12:39:52.850 DEBUG:teuthology.orchestra.console.smithi063:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T12:39:52.889 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T12:39:52.926 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is off 2024-03-04T12:39:52.960 DEBUG:teuthology.orchestra.console.smithi063:check power output: Chassis Power is off 2024-03-04T12:39:52.999 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-04T12:39:53.028 INFO:teuthology.orchestra.console.smithi060:Power off completed 2024-03-04T12:39:53.061 INFO:teuthology.orchestra.console.smithi063:Power off completed 2024-03-04T12:39:57.004 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T12:39:57.113 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2024-03-04T12:39:57.214 INFO:teuthology.orchestra.console.smithi086:Power off completed 2024-03-04T12:39:57.320 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} duration: 1992.5298249721527 failure_reason: 'Command failed (workunit test suites/fsx.sh) on smithi060 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=717ce3cce8d0166e4e577faa8004e6f9cb4128c0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh''' flavor: default owner: scheduled_vshankar@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=72dcf58a6f1547c4bb919cde95d5548b status: fail success: false 2024-03-04T12:39:57.321 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T12:39:57.407 INFO:teuthology.run:FAIL